--- title: "Language-Conditioned World Models: Pandora, 3D-VLA, RT-2." canonical: "https://world-models.io/en/research/language-conditioned-world-models/" description: "Explore language-conditioned world models that connect text, action, and simulation across Pandora, 3D-VLA, RT-2, and hybrid embodied AI systems." author: "Tyler D." generated: "2026-10-30" --- New: the Timeline is live. Track world model releases, papers, and benchmark updates in real time. The Knowledge Hub for AI World Models # Language-Conditioned World Models How language-conditioned world models use text prompts or natural-language actions to control simulation, planning, and embodied behavior across Pandora, 3D-VLA, RT-2, and hybrid systems. robotics model-based-rl simulation embodied-ai ## What is a world model? A world model is an AI system that learns a predictive representation of how an environment changes over time. It can use observations, actions, or language to estimate future states and possible outcomes before an agent acts. World models support planning in model-based reinforcement learning, robotics, embodied AI, autonomous driving, and interactive simulation. This site catalogs these systems with model records, research papers, benchmark context, and source links. Start with the [world models database](https://world-models.io/en/world-models-database/), compare systems on the [leaderboard](https://world-models.io/en/leaderboard/), or read the [Performance Index methodology](https://world-models.io/en/methodology/performance-index/). ## Research Snapshot Static research summary generated from local editorial content. | Attribute | Value | | --- | --- | | Topic | Language-Conditioned World Models | | Summary | How language-conditioned world models use text prompts or natural-language actions to control simulation, planning, and embodied behavior across Pandora, 3D-VLA, RT-2, and hybrid systems. | | Related Models | 6 | | Citations | 3 | ## What Are Language-Conditioned World Models? Editorial body section preserved directly in static HTML. Language-conditioned world models accept text prompts, instructions, or natural-language actions as part of the state transition process. They aim to connect high-level symbolic intent with grounded world prediction, allowing users or agents to steer imagined futures through language. ## Why This Research Direction Matters Editorial body section preserved directly in static HTML. Language gives world models a flexible control interface. Instead of requiring low-level action specifications only, researchers can express goals, constraints, or hypothetical interventions in natural language. This is especially important for robotics, generalist agents, and multimodal planning stacks. ## Key Systems and Architectures Editorial body section preserved directly in static HTML. Pandora connects pretrained language and video models for text-driven world simulation. 3D-VLA and RT-2 point toward action systems where language, perception, and embodiment are integrated. More broadly, hybrid stacks increasingly pair LLM reasoning with video or latent world models. ## What Language Adds and What It Still Doesn't Solve Editorial body section preserved directly in static HTML. Language improves instruction following, abstraction, and interaction, but it does not automatically provide grounded physical understanding. A model may follow textual intent while still failing on contact dynamics, long-horizon consistency, or fine-grained motor consequences. ## Open Problems in Language-Grounded Simulation Editorial body section preserved directly in static HTML. The main open problems are grounding, controllability, evaluation, and alignment between text semantics and physical state transitions. The hard question is not whether text can steer generation, but whether language-conditioned systems can remain causally faithful under real interaction. ## Related Models | Model | Lab | Category | Index v1.1 | | --- | --- | --- | --- | | [Pandora](https://world-models.io/en/models/pandora/) | Tsinghua University / ByteDance | Generative World Model | 52/100 | | [3D-VLA](https://world-models.io/en/models/3d-vla/) | MIT / Tsinghua | Foundation World Model | 47/100 | | [RT-2](https://world-models.io/en/models/rt-2/) | Google DeepMind | Foundation World Model | 72/100 | | [Sora](https://world-models.io/en/models/sora/) | OpenAI | Generative World Model | 63/100 | | [Genie 3](https://world-models.io/en/models/genie-3/) | Google DeepMind | Generative World Model | 89/100 | | [AMI World Model](https://world-models.io/en/models/ami-world-model/) | AMI Labs | Foundation World Model | N/A | ## Frequently Asked Questions FAQ answers rendered directly into static HTML for extractable responses. ### Can natural language replace action conditioning? Not entirely. Natural language is powerful for high-level intent, but many tasks still need low-level continuous or discrete control signals. ### Why is Pandora interesting? Pandora is interesting because it explicitly tries to simulate video world states under free-text actions, making the language-to-world transition a central research object rather than an afterthought. ## Quick Answer Short extractable summary preserved directly in static HTML. - Language-Conditioned World Models explains the core definition, methods, and systems involved in this research area. - This topic highlights the main trade-offs, open challenges, and practical implications for world models. - Related models and references connect the concept to concrete systems and primary sources. ## Editorial Trust Signals Editorial provenance and refresh policy preserved directly in static HTML. **Published by** world-models.io editorial board. **Lead editor** Bernard Grenat. This research page curates topic explanations, linked models, and citations grounded in primary research sources. Each editorial page is assembled from primary sources, normalized into extractable summaries, checked for factual drift, and reviewed before publication or major refreshes. Last reviewed: 2026-09-27. Pages are refreshed when a new paper, benchmark, release, architecture update, or stronger primary source materially changes the answer a reader or AI system should retrieve. Each page links back to relevant primary sources and keeps a stable canonical URL so readers can verify claims, trace context, and reference the most up-to-date version. [See the editorial policy.](https://world-models.io/en/editorial-policy/) Primary sources only Last reviewed date visible Methodology documented Source links included Correction channel published Review date visible ## External Sources Primary research citations embedded in static HTML. - [Language-Conditioned World Models reference](https://arxiv.org/abs/2406.09455) Xiang et al., 2024. Pandora: Towards General World Model with Natural Language Actions and Video States. - [Language-Conditioned World Models reference](https://arxiv.org/abs/2403.09631) Zhen et al., 2024. 3D-VLA: A 3D Vision-Language-Action Generative World Model. - [Language-Conditioned World Models reference](https://arxiv.org/abs/2307.15818) Brohan et al., 2023. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. ## References 1. [[1] Xiang et al., 2024. Pandora: Towards General World Model with Natural Language Actions and Video States.](https://arxiv.org/abs/2406.09455) 2. [[2] Zhen et al., 2024. 3D-VLA: A 3D Vision-Language-Action Generative World Model.](https://arxiv.org/abs/2403.09631) 3. [[3] Brohan et al., 2023. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.](https://arxiv.org/abs/2307.15818)