Multimodal Models: Modalities In One Vector Space

Multimodal models refer to large-scale neural network models capable of simultaneously understanding, fusing, and generating information across multiple modalities such as text, image, audio, and video. Their core idea is to achieve cross-modal semantic alignment and collaborative reasoning through a unified vector space and Transformer architecture.

What Is Modality

Modality is the form in which information exists and is expressed. Humans perceive the world in a multimodal way: we obtain visual information through our eyes, auditory information through our ears, express ideas through language, and interact with the environment through actions. The most common modalities include text, image, speech, and video. The more modalities a model supports, the closer its capabilities are to human ones—which is why text-only models are evolving into multimodal models.

Multimodal Examples：

Modality	Input Examples	Output Examples	Typical Applications
Text	Novels, chat logs	Answers, articles	Q&A, writing, translation
Image	Photos, screenshots	Image understanding, image generation	OCR, visual question answering
Audio	Speech, music	Transcription, speech synthesis	Meeting summarization, voice assistants
Video	Video streams	Video analysis, timelines	Lecture summarization, content retrieval
Action	Mouse clicks, keyboard input	Execute actions	GUI agents, automation

Recommended Related Reading from AI Robots Eidos

The “Action” modality mentioned above has already touched upon the execution level. If we want the model to operate a computer like a human, multimodal models alone are not enough—this requires the involvement of a Visual‑Language‑Action Model (VLA).

I have detailed VLA in another article. Interested readers are welcome to read that piece

Vision Language Action Models

---

What Is a Multimodal Model

A multimodal model is a model that can simultaneously process these different modalities of information. It can convert text, images, sound, video, and other data into computable representations and establish connections between different modalities. This means that multimodal large models do not simply call several independent models; instead, through modality encoding, cross-modal alignment, connection modules, or unified tokenization, they enable information from different modalities to enter the same task system, where it can be jointly understood, associated, reasoned about, and generated.

For example, Google’s Gemini model incorporates different types of data—text, video, audio, etc.—during training, which is why it can understand information and generate content in a multimodal manner. Multimodal large models have gradually moved from “image-text matching” to “cross-modal understanding, reasoning, and generation.”

---

How Do Multimodal Models Work?

The basic idea of a multimodal model can be summarized as: converting data from different modalities into representations that the model can process, then through alignment, fusion, and generation, enabling the model to establish connections across multiple types of information.

Encoding:

Raw data from different modalities vary greatly in form. Text is a sequence of symbols, images are pixel matrices, speech is acoustic wave signals, and video is a time-varying sequence of images and sound. The model cannot directly “understand” these raw data; it must first convert them into computable vector representations. For example, text needs to be tokenized and then converted into text embeddings; images need to be passed through a visual encoder to obtain visual features. The role of encoding is to transform information from different forms into mathematical representations that the model can process.

Modality	Encoding Pipeline
Text	tokenization → look up embedding table → 4096‑dim vector → Transformer → semantic vector
Image	patchification → linear projection → 4096‑dim vector → Transformer → semantic vector
Video	frame patchification + temporal encoding → 4096‑dim vector → Transformer → semantic vector
Audio	spectrogram slicing → encoding → 4096‑dim vector → Transformer → semantic vector

Core idea: everything becomes a vector of the same dimensionality and enters the same space.

Multimodal Alignment And Multimodal Fusion:

If an image contains a dog and the text caption is “a dog sitting in the living room,” the model needs to know that the image and the sentence describe the same thing. This is called cross-modal alignment—making semantically similar content across different modalities (image, text, audio, etc.) lie close together in the representation space.

After alignment, the model also needs to fuse information from multiple modalities. For example, when analyzing a video, the model must combine visuals, sound, subtitles, and temporal order; when reading a document, it must integrate text content, layout structure, tables, and images.

Fusion can occur at different levels: some models encode different modalities separately and then concatenate the features into the language model; others perform cross‑modal attention interactions at intermediate layers; still others adopt a more unified tokenization approach, organizing text, images, and audio into sequences that can be fed into the same Transformer.

Input	What the model focuses on
Text‑only	Semantic relationships between sentences
Image + text	Correspondence between image regions and text descriptions
Video + text	Association between timeline segments and the question

Generation Output:

Multimodal models can not only understand inputs but also generate outputs. The output can be textual answers, image captions, images, speech, video, structured results, or even tool calls or action instructions. Therefore, multimodal models are becoming an important technical foundation that connects perception, language, generation, and action.

---

Training of Multimodal Models

Multimodal models are capable of processing data from different modalities (e.g., text, image, audio) simultaneously and mining the latent connections between them, thereby enabling more intelligent and comprehensive information processing. However, training multimodal models is not trivial—it involves multiple complex steps, including data construction and preprocessing, model design and construction, and model training and optimization.

–Data Construction and Preprocessing

Multimodal models require large amounts of rich multimodal data to support learning and training. The goal of preprocessing is to align modalities, extract features, and improve data quality and usability.

Modal alignment is a key step in multimodal data preprocessing: data from different modalities must be accurately aligned in time or semantics so that the model can better learn their associations.

Feature extraction is the core step of preprocessing. For different modalities, appropriate techniques are used: for images, convolutional neural networks (CNNs) can extract feature vectors; for text, word embeddings and language models extract semantic features; for audio, spectrogram‑based feature extraction algorithms are used.

–Model Design and Construction

At the model building stage, the architecture must be designed and selected according to the specific task requirements and data characteristics, while also considering model complexity, computational resource needs, and other application constraints.

Modality fusion design is a critical part of model design. Fusion combines feature representations from different modalities into a shared representation space, enabling cross‑modal interaction and sharing.

–Model Training and Optimization

During training, appropriate loss functions and optimizers must be chosen to update model parameters. The loss function should account for the relationships and consistency between different modalities. For example, for an image‑text multimodal large model, a joint loss function can be designed, consisting of weighted sums of image and text losses to achieve multimodal information fusion and interaction. The choice of optimisation algorithm depends on the model’s large‑scale parameters and complex structure, and should be made case by case.

---

Multimodal Model Architecture (Using a Multimodal Large Language Model as an Example)

–Modality Encoder:

The modality encoder primarily encodes non‑text modal information into features that are easy for the LLM to use. The size of the encoded features affects the amount of information passed to the LLM. For example, for a visual encoder, increasing the input image resolution is an effective way to improve performance. One approach is to directly raise the resolution, which requires fine‑tuning the visual encoder to adapt to the higher resolution. Another approach is to split a high‑resolution image into multiple sub‑images, each fed into the visual encoder at a lower resolution, indirectly increasing the effective input resolution.

–LLM:

Since text information must be handled, an LLM is required. In terms of performance, increasing the LLM parameter size can bring significant gains. For example, LLaVA‑NeXT experiments on 7B/13B/34B LLMs show that larger LLMs yield substantial improvements across various benchmarks. Besides parameter scaling, the MoE (Mixture of Experts) architecture offers a more efficient approach by using sparse computation to increase the total number of model parameters without increasing actual computation.

–Connector / Projection Layer:

The connector (or projection layer) further processes non‑text features (e.g., visual features) into token formats suitable for the LLM’s input. Compared with the encoder and LLM, the connector is relatively less critical—its importance is not as high as the number of visual tokens (which determines the available visual information for the LLM) or the image resolution (which determines the visual encoder’s input information).

–Generator:

The generator is generally optional; its main role is to generate non‑text multimodal content from the LLM’s output features, compensating for the LLM’s limited multimodal generation capability. Examples include:

Image generation: generating images from text descriptions using LLM output features.

Video generation: generating video content from text descriptions, or generating video sequences from image sequences.

---

Common Multimodal Models

Model	Core Idea	Model Architecture	Loss Design	Training Data
CLIP	Align text and images in feature space.	Text encoder (BERT), image encoder (ViT); compute cosine distance between features from the two encoders.	InfoNCE loss (contrastive learning loss used in MoCo).	400 million image‑text pairs.
BLIP	A multimodal model that handles both image‑text understanding and generation, jointly pre‑trained on three vision‑language objectives: image‑text contrastive learning, image‑text matching, and image‑conditioned language modeling.	Multimodal mixture of Encoder‑Decoder with four components: • Unimodal image patch encoder (ViT) • Unimodal text encoder (BERT) • Multimodal image‑grounded text encoder (adds a cross‑attention layer between self‑attention and FFN) • Multimodal image‑grounded text decoder (replaces bidirectional self‑attention with causal self‑attention for next‑token prediction).	ITC loss (understanding‑oriented, optimizes unimodal encoders): image‑text contrastive learning to align feature spaces. ITM loss (understanding‑oriented, optimizes multimodal encoder): binary image‑text matching classification to model cross‑modal relevance. LM loss (generation‑oriented, optimizes multimodal decoder): cross‑entropy loss for autoregressive generation of coherent descriptions.	Pre‑training datasets: COCO, Visual Genome, web data (Conceptual Captions 3M, Conceptual 12M [noisy], SBU Captions). Also experimented with an additional noisy web dataset LAION (115M images).
BLIP‑2	Leverage pre‑trained visual encoders and LLMs to improve multimodal performance and reduce training cost.	Pre‑trained Image Encoder, pre‑trained Large Language Model, and a learnable Q‑Former (Image Transformer + Text Transformer).	ITC loss (understanding): image‑text contrastive learning to align feature spaces. ITG loss (generation): generate text description given input image, forcing queries to extract visual features relevant to text. ITM loss (understanding): binary image‑text matching for fine‑grained alignment.	BLIP’s data (129M); additionally used CapFit to synthesize text descriptions from web images (taking top‑2 candidates).
LLaVA	Use GPT‑4 to generate multimodal language‑image instruction‑following data.	Visual encoder + LLM + projection layer.	Autoregressive language modeling loss (maximizing likelihood of current answer given previous Q&A rounds).	158K instruction‑following samples: 58K conversations, 23K detailed descriptions, 77K complex reasoning.
mini‑GPT‑4	Connect visual encoder with a more powerful LLM using a single projection layer.	Visual encoder (ViT‑G / F4 + Q‑Former) + LLM (Vicuna) + one projection layer.	Similar to BLIP‑2, mainly language modeling loss.	Only train the projection layer: 5M aligned image‑text pairs + 3,500 high‑quality image‑text pairs.
Instruct BLIP	Instruction fine‑tuning for a general‑purpose vision‑language model.	Similar to BLIP‑2, but adds instruction input to both Q‑Former and LLM.	Similar to BLIP‑2, mainly language modeling loss.	26 datasets (in instruction‑tuning format): 13 used for instruction fine‑tuning and 13 for zero‑shot evaluation.

---

Challenges Faced by Multimodal Models

–Modality conflict and semantic gap: The low‑level statistical characteristics of different modalities differ greatly (e.g., text is discrete, images are continuous), leading to gradient conflicts during training. In complex scenarios (e.g., “the emotion of a person in a video contradicts the text description”), processing accuracy still falls below 80%.

–Inference cost: A 224×224 image contributes 196 tokens; a high‑definition image can contribute 784 tokens. The more tokens, the larger the KV cache footprint and the quadratically increasing attention computation. The hundreds of tokens brought by a single image make inference far more expensive than for a short text—this is the core reason why multimodal model API calls are typically more costly than pure text models.

–Alignment difficulty: Making image vectors and text vectors “speak the same language” in the same space sounds simple but is in fact highly challenging. The quality of the projection layer directly determines the model’s ability to “understand” images. If the projection layer is not well trained, the model may “see” the image but fail to comprehend it.

–Ethical and safety risks: Native multimodal models can generate realistic deepfake videos, posing fraud risks. Detection difficulty is 50% higher than for single‑modal content. Also, if training data contains algorithmic biases—such as gender stereotypes associating men with engineers and women with teachers—the model may amplify these biases in generated content, exacerbating social prejudice.

---

Applications of Multimodal Models

–Office Work: Multimodal models have rich applications in office settings. For example, in AI meeting summarization, the model can simultaneously understand meeting recordings (including video and audio) and shared PPT screenshots, automatically generating timestamped meeting minutes. In document analysis, a user can upload a PDF containing images and tables, ask questions, and receive precise answers.

–Learning: In tutoring scenarios, when a student encounters a difficult problem, they can take a photo of a geometry question; the model recognises the problem and provides step‑by‑step solutions or detailed explanations (text/voice output).

–Content Creation: Mixed text‑and‑image generation is becoming more popular: users can input an article outline, and the model directly generates a well‑formatted article with matching images, tables, and layout. In video editing and understanding, given a 1‑hour lecture video, the user can ask about specific topics and the model returns the corresponding timestamps.

–Daily Life Services: While shopping, you can discuss desired products with the model while showing it product photos, and the model gives purchasing advice. For travel, if you encounter a foreign‑language menu, the model can translate it directly from a photo of the menu.

---

Future Directions for Multimodal Models

Autonomy

Current multimodal models are largely passive executors that require explicit human instructions. Future development aims to endow models with higher‑level autonomy, making them active planners and decision‑makers.

–Active learning and exploration: Future multimodal models will proactively explore unknown environments, identify knowledge gaps, and ask valuable questions. For example, a home service robot encountering an unfamiliar object might actively ask: “What is this? How should I handle it?” This active learning ability will enable continuous, lifelong knowledge acquisition.

–Long‑term task planning: Models will understand and plan complex tasks that take hours, days, or even longer to complete. For instance, a user could give a high‑level instruction like “Help me plan a one‑week trip to Paris,” and the model autonomously performs information retrieval, itinerary planning, hotel booking, and writing a travel guide.

Fusion Innovation

–Integration with reinforcement learning (RL): RL provides a powerful framework for learning optimal policies through trial and error. Using the general knowledge and reasoning capabilities of LLMs as “prior knowledge” for RL agents can greatly improve learning efficiency and generalisation—this has already shown great promise in game AI and robot control.

–Integration with knowledge graphs: Knowledge graphs store and represent world knowledge in a structured way. Combining multimodal models with knowledge graphs can, on one hand, provide more accurate and interpretable knowledge sources, alleviating the “hallucination” problem; on the other hand, the model can automatically extract and update knowledge from unstructured multimodal data, building more comprehensive and up‑to‑date knowledge graphs.

Cost Reduction

–Data efficiency and reuse: Through cross‑modal data alignment, soft labels, one‑time feature embedding generation, etc., data utilisation is improved, reducing data acquisition and cleaning costs.

–Efficient architecture design: Using Mixture of Experts (MoE) for dynamic sparse activation—invoking only relevant modules—reduces inference computation while maintaining performance; lightweight visual encoders and efficient projectors reduce the number of visual tokens and computational overhead.

–Synthetic data: Synthetic data can replace part of real data, cutting costs by 50%. It is projected that by 2026, synthetic data will account for over 40% of multimodal training data.