What are multimodal models?

Summary slides

Why combining modalities is a genuinely different problem than handling each one separately

Building a model that understands text well and, separately, building a model that understands images well doesn’t automatically produce a model that can meaningfully connect the two together, and this is the core technical challenge multimodal models actually solve. A system that runs a text model and an image model side by side, with no shared understanding between them, can describe an image in isolation or answer a text question in isolation, but it struggles to do the kind of integrated reasoning a genuinely multimodal model handles naturally, answering a nuanced question that depends on details in an image combined with context provided in text, or generating an image that correctly reflects a complex, multi-part text description.

What makes a model genuinely multimodal, rather than just several separate models loosely stitched together, is that it’s trained to build a shared, unified representation across these different kinds of data from the start, learning how concepts in one modality relate to concepts in another during training itself, rather than trying to bridge that gap after the fact through some separate coordination layer bolted on top of otherwise independent systems.

How multimodal training produces this shared understanding

Training a multimodal model typically involves exposing it to large amounts of paired data across modalities, images with their corresponding text descriptions, video with its accompanying audio and captions, letting the model learn the underlying relationships between what’s expressed in one modality and what’s expressed in another. Through this process, the model builds an internal representation where related concepts across different modalities end up positioned close together, a photo of a dog and the word “dog” come to share meaningfully overlapping internal representation, even though one is a text token and the other is a grid of pixel values, because the training process has exposed the model to enough examples connecting the two.

This shared representation is what lets a multimodal model reason across modalities in ways a collection of separate, unimodal models fundamentally can’t, since the connections between modalities aren’t something added afterward through external coordination, they’re baked directly into how the model represents information internally from the earliest stages of training onward.

The different directions multimodal capability flows

Multimodal capability isn’t a single, uniform thing, it spans several distinct capabilities that don’t all show up together in every multimodal model. Understanding multiple modalities as input, taking an image and text together and reasoning about both, is one capability. Generating output in a modality different from the input, producing an image from a text description, or producing text describing an image, is a related but distinct capability. Some multimodal models handle understanding well without strong generation capability in every modality, while others are built specifically around generation, and a model’s actual usefulness for a given application depends on which specific combination of these capabilities it has, not just whether it’s broadly labeled multimodal.

This distinction matters directly for evaluating whether a specific multimodal model fits a specific task, a model strong at understanding images and answering questions about them may have comparatively weak image generation capability, and a team evaluating multimodal models for a particular application needs to look at the specific capability that application needs, rather than assuming a model labeled multimodal automatically excels across every possible combination of input and output modality.

Why multimodal capability unlocks tasks a text-only model simply cannot handle

A considerable range of real-world tasks are fundamentally impossible for a text-only model to handle at all, regardless of how capable that model’s text understanding is, analyzing a photograph, understanding a diagram or chart, processing a screenshot of an application’s interface, transcribing and reasoning about spoken audio. These tasks require access to information that simply doesn’t exist in text form, and a text-only model has no way to bridge that gap no matter how it’s prompted, since the information it needs was never available to it in the first place.

This is why multimodal capability has become increasingly central to a growing range of practical applications, an AI coding agent that can look at a screenshot of a broken user interface and reason about what’s visually wrong, a document processing system that can read both the text and the embedded charts and images in a report, a customer support system that can accept a photo of a damaged product alongside a written description of the problem, all of these depend on genuine multimodal capability that no amount of clever text-only prompting could ever substitute for.

What evaluating a multimodal model’s quality actually requires

Evaluating a multimodal model well requires testing its actual cross-modal reasoning specifically, not just its performance on each modality in isolation, connecting to the broader discussion of AI native testing covered elsewhere in this collection. A model can perform reasonably well when asked to describe an image alone, and separately perform well when answering a pure text question, while still performing poorly on a task that genuinely requires connecting information across both together, a gap that testing each modality separately would never surface.

This means a meaningful evaluation of a multimodal model needs test cases built specifically around genuine cross-modal reasoning, a question whose correct answer depends on details present only in an image combined with instructions given only in text, rather than test cases that could technically be answered using either modality alone. This distinction between genuine multimodal reasoning and merely handling multiple modalities separately is exactly the kind of nuance a superficial evaluation is most likely to miss.

The added cost and complexity multimodal capability introduces

Processing additional modalities beyond text carries real computational cost, an image or a segment of audio typically requires considerably more processing to represent internally than an equivalent amount of text does, which means a multimodal request often costs more and takes longer to process than an equivalent text-only request would. This connects directly to the broader discussion of model selection covered elsewhere in this collection, a task that doesn’t actually need multimodal capability pays an unnecessary cost when handled by a multimodal model unnecessarily, and the same deliberate matching of model capability to actual task requirements applies here just as it does for choosing between models of different sizes.

This cost consideration is part of why a well-designed system doesn’t route every request through a multimodal model by default, reserving that capability specifically for requests that involve non-text input or output, while handling pure text requests through a more efficient, text-only path when the task doesn’t actually call for anything beyond that.

Common mistakes teams make around multimodal models

1. Assuming a model labeled multimodal excels equally across every combination of input and output modality, rather than checking which specific capabilities the model actually has.

2. Evaluating a multimodal model’s quality using each modality in isolation, missing whether it can reliably reason across modalities together in the way a specific application requires.

3. Routing every request through a multimodal model by default, incurring unnecessary cost and latency on requests that never involved anything beyond plain text in the first place.

4. Underestimating how much a task genuinely depends on non-text information, attempting to solve a fundamentally visual or audio-based problem through text-only prompting instead.

5. Assuming a system combining separate, independently trained models for each modality delivers the same cross-modal reasoning a genuinely unified multimodal model provides.

What connects these mistakes is underestimating how much genuine multimodal reasoning depends on a model actually being trained to connect modalities together from the start, rather than simply having access to more than one kind of data, evaluating and applying multimodal models well requires understanding this distinction clearly rather than assuming the label alone guarantees the capability a specific task actually needs.

The deeper point about multimodal models is that they extend AI capability into the considerable range of real-world information that was never expressible in text alone, images, sound, video, and the growing importance of this capability reflects how much of the world’s actual information genuinely lives outside plain text, making multimodal capability not a niche add-on but an increasingly central requirement for AI systems meant to engage meaningfully with how information actually exists outside of language.