Table of Contents
Vision Language Action Models, also known as VLA models, aim to build a universal robot brain that “sees, hears, and acts,” enabling robots to perform complex physical operations based on visual and linguistic instructions, just like humans. This is becoming a key technical direction in the field of embodied intelligence.
What Is A Vision Language Action Model?
A vision language action model represents a cutting-edge technical paradigm in multimodal artificial intelligence and embodies the core development direction of embodied intelligence. By deeply integrating visual perception, natural language understanding, and action strategy generation, these models aim to build intelligent agents that can perceive physical environments, understand human intentions, and execute complex manipulation tasks. VLA models are not only an extension of perceptual intelligence toward cognitive intelligence but also a critical bridge from the “digital world” to the “physical world.” Their core logic can be summarized as:
Perception (Vision) + Cognition (Language) → Decision Making & Execution (Action).
Composition of Vision Language Action Models
The power of Vision Language Action Models stems from the deep integration and synergy of three core modules: vision, language, and action.
–Vision Encoder: Acts as the “eyes” of VLA models, processing image and video data from sensors such as cameras, extracting key information like objects, spatial relationships, and dynamic changes in the scene. For example, using models like ViT or CNN, it extracts spatial details from camera feeds and LiDAR point clouds, distinguishing whether a distant object is stationary or moving.
–Language Encoder: Integrates large language models such as LLaMA and Qwen, enabling VLA to understand natural language instructions, analyze, and break them down. Advanced models can even perform chain-of-thought reasoning, decomposing complex tasks into clear steps. For example, when you say “I’m feeling carsick,” the model not only understands the meaning but also infers from the semantics that “smoother acceleration and deceleration are needed.”
—Action Decoder: The action decoder of VLA uses diffusion models or Transformers to output fine-grained operation sequences. This is the core feature that distinguishes VLA from ordinary vision-language models. Instead of outputting coarse commands like “turn left,” it acts like an experienced driver producing continuous, smooth responses such as “turn the steering wheel 15 degrees left, return to center after 2 seconds, slightly release the accelerator, and then gently press it again.”
Process of Vision Language Action Models
The inference process of VLA models is a closed-loop perception-decision system:
–Input Acquisition: Real-time capture of image frames and user instructions (e.g., “put the bowl into the dishwasher”).
–Feature Extraction: The vision encoder extracts scene features, while the language encoder parses instruction semantics.
–Semantic Alignment: The fusion layer uses attention mechanisms to bind objects mentioned in the instruction (e.g., “bowl”) with corresponding objects in the visual input.
–Strategy Generation: After understanding the task and goal, the model enters a “thinking” phase, decomposing the task and planning a feasible action plan. Some models also generate intermediate reasoning steps to make the decision process more transparent.
–Execution and Feedback: The model decodes the planned action plan into precise parameter commands executable by the robot and drives hardware such as robotic arms and mobile bases to begin working, then enters the next inference loop based on environmental changes.
Advantages of Vision Language Action Models
–Multimodal Fusion: VLA models deeply integrate vision, language, and action. Using cross-attention mechanisms, they tightly combine road conditions, traffic signs, and other visual information with language instructions and semantic map knowledge to make optimal decisions. The accuracy and rationality of these decisions surpass traditional single-modality or simple multimodal models.
–Complex Scene Handling: Built on advanced architectures like Transformers, VLA models possess powerful global context understanding. In complex scenes, they can simultaneously analyze visual and linguistic information from the surrounding environment, constructing a more complete environmental representation and scene awareness to support strategy generation—a capability difficult for traditional models to achieve.
–Explainability: A key breakthrough of VLA models is the introduction of chain-of-thought reasoning, enabling explainability of environment understanding and decision reasoning through language models. The built-in language encoder supports natural language interaction; users can issue complex instructions via voice or text, and the model can also provide decision rationale in natural language—for example, “Due to congestion ahead, changing lanes for faster passage”—making the VLA model’s decision process interpretable.
Challenges Facing Vision Language Action Models
–Data Acquisition: Obtaining high-quality data is a challenge. Training a general-purpose VLA model requires massive, diverse real-world data. However, real data collection is costly and inefficient, while synthetic simulation data has gaps compared to the real world, limiting the versatility and reliability of VLA models.
–Generalization: This is a key measure of how “smart” a VLA model is, requiring stable performance in both seen and unseen novel scenarios. Currently, insufficient generalization remains the biggest pain point; many models fail easily when the environment changes.
–Architecture and Compute: To achieve smooth real-time interaction, VLA models must balance strong generalization with fast response speed. Large model parameters improve understanding but introduce high computational load and latency.
Different Types and Comparison of Vision Language Action Models
Types of Vision Language Action Models
–RT-2: The core of RT-2 is jointly fine-tuning a pre-trained vision-language model (VLM) with robot action data, transferring VLM knowledge to robot control. It pioneeringly discretizes robot actions into tokens, unifying them within a language model. However, this architecture generates actions serially, which is inefficient, and due to the technology at the time, action smoothness is only moderate.
–π-series (π0, π0.5): π0, released by Physical Intelligence, uses a sparse Mixture-of-Experts (MoE) architecture comprising a VLM expert (3B parameters) and an action expert (0.3B parameters), employing flow matching to generate smooth, continuous actions. The subsequent π0.5, through “knowledge insulation” fine-tuning, further improves generalization while maintaining control precision for high-frequency fine movements. Its zero-shot generalization is among the best currently, but it has high compute requirements, and its technical details are not fully public.
–OpenVLA: Built on Llama 2 and fine-tuned on 970K trajectories from the Open X-Embodiment dataset, OpenVLA is one of the most popular open-source VLA foundation models. Its open-source nature and strong generalization have greatly contributed to the spread of VLA technology, but inference efficiency remains a bottleneck due to its autoregressive architecture.
–SmolVLA: Released by Hugging Face, SmolVLA has only 450M parameters and can run on a MacBook or ordinary GPUs. Based on the SmolVLM-2 vision-language model, it uses flow matching for action generation and achieves efficient operation through techniques such as reducing visual tokens, layer skipping, and asynchronous inference. It pushes computational efficiency and open-source friendliness to the extreme, making it an excellent starting point for resource-constrained research.
–X-VLA: Proposed by Tsinghua University and Shanghai AI Lab, X-VLA has only 0.9B parameters but demonstrates strong zero-shot transfer across different robot morphologies using flow matching + soft prompting. It shows remarkable data efficiency, learning complex tasks (e.g., folding clothes) with little data and transferring to new robots, though the limits of its lightweight design are still being explored.
–GOVLA: From SiPuxin (Zhi Square), GOVLA adopts a “fast-slow dual-system” architecture. It is the first to overcome the limitation of conventional VLAs, which only output arm actions, by simultaneously outputting whole-body control and mobile trajectories to coordinate mobile bases and robotic arms. It is better suited for complex scenarios like industrial manufacturing (industrial VLA) that require cross-area movement and multi-robot collaboration. Still, it poses extremely high challenges for low-level hardware control and system integration.
Recommended In-depth Reading from AI Robots Eidos
GOVLA is an advanced visual-language-action (VLA) large model that possesses strong environmental perception, semantic understanding, and autonomous decision-making capabilities. By integrating multimodal information such as vision and language, it provides AGVs/AMRs with smarter navigation, obstacle avoidance, and task execution abilities. For example, GOVLA’s “dual system” architecture balances real-time response with complex task planning, allowing AGVs/AMRs to flexibly navigate obstacles, adjust paths, and even understand natural language instructions in dynamic environments, resulting in more efficient human-machine collaboration.
If readers wish to delve deeper into the knowledge of AGVs/AMRs, please read the articles on AGVs and AMRs.
–HoloBrain-0: From Horizon Robotics, HoloBrain-0 is the first to explicitly inject robot embodiment information (e.g., camera parameters, kinematic structure) into the architecture, giving the model stronger 3-Dimensional spatial perception. It comes in 0.2B and 1.1B versions, with the lightweight version proving the feasibility of deploying complex VLAs on edge chips. Its precise understanding of 3-Dimensional spatial relationships is unmatched by other models, but constrained by a lightweight design, its complex reasoning ability may be weaker than that of very large models.
–WALL-A: From Auto-Robot, WALL-A pioneered a deeply integrated joint framework of “world model + VLA.” Using the world model for spatiotemporal state prediction and causal reasoning, it helps the model internalize physical common sense. WALL-A is known for its largest parameter scale and strong zero-shot generalization, capable of completing minute-long long-horizon tasks in over 95% of scenarios. However, the deployment cost of such a large model is substantial.
–Goal-VLA: Proposed by the National University of Singapore, Goal-VLA completely decouples planning from control. It uses an image-generative VLM as an “object-centric world model” to generate semantic goal states, which are then executed by a training-free low-level policy. It does not require paired “instruction-vision-action” data and achieves strong zero-shot generalization, though its generalization in complex real-time tasks still needs verification.
–EfficientVLA & FlashVLA: These are not standalone models but plug-and-play acceleration frameworks. EfficientVLA accelerates CogACT by 1.93x, reducing computation to 28.9%; FlashVLA reduces computation by 55.7% through an “action reuse” strategy. They efficiently accelerate existing VLA models, but their adaptability to specific architectures requires validation.
Comparison of Vision Language Action Models
Features of Different Vision Language Action Models
| Model Name | Affiliated Institution | Core Technical Route | Key Highlights | Applicable Scenarios & Main Limitations |
| OpenVLA | Stanford, etc. | Autoregressive (VLM as Backbone) | Open-source benchmark, based on Llama 2, fine-tuned on 970k trajectories. | Research/developer community, strong generalization. |
| Limitation: Training on consumer-grade hardware still has barriers. | ||||
| π-series (π0, π0.5) | Physical Intelligence | Flow matching | Industry performance benchmark, smooth action generation, strong generalization. | Complex, long-horizon, dexterous manipulation requiring fine force control. |
| Limitation: Model details not fully disclosed. | ||||
| SmolVLA | Hugging Face | Flow matching + Lightweighting | Extremely lightweight (450M parameters), runs on MacBook, open-source and easy to use. | Resource-constrained environments, rapid prototyping, community-driven development. |
| Limitation: Performance ceiling lower than that of very large models. | ||||
| X-VLA | Tsinghua University & Shanghai AI Lab | Flow matching + Soft prompting | Strong cross-embodiment generality (0.9B), high data efficiency, zero-shot transfer. | Tasks requiring deployment across different robot platforms. |
| Limitation: Exploring the limits of lightweight design. | ||||
| WALL-A | Auto-Robot | End-to-end unified + World model | Largest parameter scale, pioneering deep fusion of “world model + VLA”, extremely strong zero-shot generalization. | Pursuing ultimate zero-shot generality, covering scenarios with diverse complex manipulations. |
| Limitation: Large parameter count, high real-world deployment cost. | ||||
| GOVLA | SiPuxin (Zhi Square) | Whole-body global + Dual-system | Whole-body control, pioneering output of mobile trajectories + arm actions, suitable for complex long-horizon tasks. | Complex tasks requiring coordination of mobile base and robotic arm, e.g., industrial manufacturing, commercial service. |
| Limitation: High technical barriers, high deployment cost. | ||||
| HoloBrain-0 | Horizon Robotics | Autoregressive + Embodied priors | Strong 3D spatial understanding, pioneering injection of robot embodiment information; lightweight version has only 0.2B parameters. | Tasks requiring high spatial understanding, edge deployment, cross-embodiment generalization. |
| Limitation: Constrained by lightweight design, complex reasoning ability may be limited. | ||||
| RT-2 | Google DeepMind | Autoregressive (VLM as Backbone) | Founder of the VLA concept, pioneering transfer of internet knowledge to robot control. | Basic research, evaluating the effect of internet knowledge on robot control. |
| Limitation: Architecture no longer optimal, low action generation efficiency. | ||||
| Goal-VLA | National University of Singapore | World model (generative VLM) | New paradigm for zero-shot manipulation, decoupling planning from control, no need for paired action data. | General manipulation scenarios requiring open-vocabulary, zero-shot generalization. |
| Limitation: Currently academic exploration; real-time performance and generalization on complex tasks need verification. | ||||
| EfficientVLA | Shanghai Jiao Tong University | Inference acceleration (training‑free) | Plug‑and‑play, accelerates CogACT inference by 1.93×, reduces computation to 28.9%. | Fast, lossless acceleration for existing VLA models. |
| Limitation: An acceleration framework, not a standalone model. | ||||
| FlashVLA | Fudan University | Inference acceleration (training‑free) | First plug‑and‑play acceleration framework supporting “action reuse”, reduces computation by 55.7%. | Efficient inference acceleration for existing VLA models. |
| Limitation: An acceleration framework, not a standalone model. |
Selection of Different Vision Language Action Models
| Dimension | Recommended Models |
| Generalization Ability | WALL-A, π-series, Goal-VLA, X-VLA |
| Computational Efficiency | SmolVLA, EfficientVLA, FlashVLA, HoloBrain-0 |
| Action Smoothness | π-series, SmolVLA, X-VLA |
| 3D Spatial Understanding | HoloBrain-0 |
| Whole-body / Mobile Control | GOVLA |
| Force Control Friendly | π-series, X-VLA, SmolVLA |
| Zero-shot Capability | Goal-VLA, WALL-A |
| Open-source Ecosystem & Ease of Use | OpenVLA, SmolVLA, HoloBrain-0 |
Training of Vision Language Action Models
To make VLA models perform effectively, a carefully designed training process is essential.
Pre-training. In this stage, the model learns from massive amounts of data: images, videos, and text. The VLA model learns general visual and linguistic knowledge, acquiring common sense about the world.
Post-training (Fine-tuning). In this stage, the pre-trained VLA model is placed into real application environments and fine-tuned with actual operational data. The model learns how experienced practitioners in the field handle various situations, strengthening its capabilities through a large volume of real-world data. Here, the model transitions from book knowledge to practical knowledge.
Reinforcement training (with a world model). This is a critical step. The VLA model interacts with a world model (an AI-simulated “sandbox” of the real physical world), repeatedly trying various actions, verifying its decisions, and adjusting based on “how well it performed.” Just as pilots practice in flight simulators before flying actual aircraft, the world model allows the VLA to accumulate extensive experience at very low cost.
Applications of Vision Language Action Models
–Transportation: Integrating multi-view vision and language instructions to achieve dynamic path planning, e.g., “avoid construction zones” or “take a scenic route.” In urban autonomous driving tests, decision accuracy is significantly better than traditional models.
–Agriculture: For example, in precision harvesting, using multispectral vision to identify “Grade A apples” and coordinating a robotic arm for non-destructive picking, improving efficiency and reducing damage rates.
–Service Industry: Supporting natural language interaction for more intuitive human-robot collaboration in home and office environments. For instance, following natural language commands like “hand me the water cup” or “tidy up the room,” enabling whole-body motion control that coordinates arms, torso, and fingers to complete specified tasks.
Investment Analysis for Vision Language Action Models
Investment Opportunities of Vision Language Action Models
The explosion of VLA models creates multiple layers of investment opportunities:
–Compute infrastructure: The training compute demand for VLA models is 3–5 times that of large language models, further driving demand for AI training chips and high-bandwidth memory (HBM).
–Model development and deployment platforms: Companies that provide VLA model training frameworks, deployment toolchains, and inference optimization solutions will reap excess returns.
–VLA-enabled industry endpoints: Including humanoid robots, autonomous driving, industrial vision, and other use cases. These fields will experience product capability leaps and accelerated penetration rates driven by VLA.
Investment Risks of Vision Language Action Models
Here, AI Robot Eidos reminds readers that the commercialization of VLA models still faces several challenges, as detailed in the earlier “Challenges” section. A brief summary follows for potential investors to consider carefully.
–Data issues: High-quality multimodal aligned datasets are extremely scarce and costly to construct.
–Safety issues: Safety constraints for action-oriented AI are far more complex than for conversational AI; a single wrong action could cause physical-world harm.
–Computational cost: The inference computation of VLA models is far greater than that of pure language models, and real-time deployment on edge devices still requires breakthroughs.
Insight from AI Robots Eidos about Vision Language Action Models
—The next breakthrough for Vision Language Action models will not be in accumulating larger scales of robotic action data, but rather in introducing explicit causal world models. Such models will enable robots to perform “mental simulations” before execution—similar to how humans pre-rehearse the consequences of different actions in their minds—thereby achieving true long-range task generalization and zero-shot adaptation. The future universal robotic brain will feature a dual-layer architecture with a “world model as the core and VLA model as the execution frontend.”
—In the future, Vision Language Action models will commonly adopt a modular expert separation: a frozen universal Vision-Language Model (VLM) for scene understanding and instruction decomposition, and an independent, lightly trainable action expert for high-frequency/force control responses. This “general brain + specialized cerebellum” model will enable robots to possess common sense while being capable of delicate operations akin to those of a craftsman.
—As lightweight technologies (such as visual token reduction, layer skipping, and asynchronous reasoning) mature, Vision Language Action models will penetrate consumer-grade robots. This will give rise to a new ecosystem: edge-embodied intelligence—robots capable of understanding and acting in real-time without needing a network connection, with data privacy, low latency, and offline reliability becoming core competitive advantages. In the future, service robots in every household may be equipped with a tiny yet efficient VLA model.
Image Credits: Medium & Stackademic & AI & Semanticscholar & Themoonlight