← Research library

Find your way
through world models.

Definitions, reading paths, and the questions behind the papers.
Read the field from its foundations to its open problems.

3 reading tracks37 glossary terms11 evaluation dimensions12 open problems

15 chapters · From the curated README

01 / Start here

Start Here

Source ↗
If you are interested in...Start with
The definition and scope of world modelsDefinition and Scope, Taxonomic Overview
Classic foundations and cognitive originsMind World Models, Latent Dynamics Models
Video, games, and interactive simulationGame & Interactive World Simulation, General Video World Models, Persistent Narrative & Multi-Shot Video
Autonomous driving world modelsAutonomous Driving — Generative, Occupancy & BEV Representations, Closed-Loop Simulation & Evaluation
Robotics, VLA, and World Action ModelsEmbodied AI & Robotics, VLA & WAM, World-Model-Guided Planning
Benchmarks, datasets, and open toolkitsEvaluation Dimensions, Benchmarks & Evaluation, Community Resources & Open Repositories
A guided reading orderReading Roadmap, Historical Timeline, Architecture Cheat Sheet
Terminology, labs, and open problemsGlossary, Labs, Companies & Open Stacks, Open Problems, FAQ
02 / Start here

Definition and Scope

Source ↗

A short working definition

A world model is an internal predictive model of an environment that helps an agent answer:

What will happen if I act, wait, intervene, or imagine an alternative future?

That definition is intentionally broader than model-based RL, but narrower than "any model that understands the world".

World model vs. nearby concepts

ConceptCore questionTypical output
world modelwhat happens next under state, action, or intervention?future observations, latent states, occupancy, trajectories, or executable rollouts
simulatorcan the environment be replayed or executed?environment transitions, often hand-built or learned
planner / policywhat should the agent do?actions, plans, control sequences
perception modelwhat is in the scene now?labels, detections, depth, segmentation

A practical boundary

A paper is strongest as a world-model entry when it does at least two of the following:

  1. models state,
  2. predicts state evolution under action or intervention,
  3. supports imagination, planning, evaluation, or controllable simulation.

Inclusion heuristics

  • Prefer primary sources: arXiv, conference/journal pages, official project pages, and official repositories.
  • Prefer papers with an explicit dynamics / future / intervention component over static perception-only work.
  • Include adjacent work only when it materially improves the understanding of world models: latent planning, JEPA-style predictive representation learning, embodied evaluation, or physically grounded simulation.
  • When a paper is domain-specific, place it by its main technical role first and by domain second.

Deliberate non-goals

  • This is not a generic list of video generation, VLA, or autonomous driving papers.
  • Pure perception, segmentation, or forecasting papers without a genuine world-modeling role are deprioritized.
  • Blog posts and secondary commentary are included selectively and are kept separate from the paper taxonomy.
03 / Start here

How to Use This List

Source ↗

Navigate by paradigm, then by domain

The taxonomy has one deliberate spine:

  1. Decide what kind of model you care about. Synthesizing plausible futures → 1 · Generative. Learning structured internal state without pixel decoding → 2 · Representational. Coupling a world model to acting, planning, and evaluation → 3 · Agentic. Cognitive and biological grounding → 0 · Mind World Models.
  2. Then narrow by domain or mechanism inside that paradigm (e.g. 1.2 driving, 2.2 JEPA, 3.3 closed-loop evaluation).
  3. When a paper is domain-specific, it is filed by its main technical role first and domain second. For example, JEPLO and GLAM live under JEPA (§2.2) because they predict latent representations, with robotics domains recorded on each entry.

The Start Here table maps common intents directly to sections. The Taxonomic Overview shows the full tree at a glance.

What the badges mean

BadgeMeaning
arXivThe arXiv paper; the badge label carries the full arXiv ID, so you can search the page for an ID you already know.
GitHubOfficial code or the official project repository. Unofficial reimplementations are not badged.
ProjectOfficial project page.
HuggingFaceOfficial model, dataset, space, or leaderboard.
BlogTechnical blog post or official research-lab write-up.
PaperNon-arXiv primary source: DOI, OpenReview, or proceedings page.

Tables vs. bullets

Two entry formats coexist by design:

  • Tables are used where a family is mature enough to compare on fixed columns: the RSSM / Dreamer family, MBRL, Surveys, Benchmarks, and Community Resources.
  • Bullets with a one-line rationale are used in fast-moving areas where entries do not yet share a comparable schema. The indented > line under each bullet states, factually, why the entry is in scope.

Finding a paper you already know

Search the rendered page for the arXiv ID. Every arXiv badge label carries the full ID (2408.14837, 2608.14530, …), so an ID search is unambiguous even when titles or acronyms collide.

How a curation pass works

The header date is the last full pass, not the last commit. A pass means: (1) new in-scope papers since the previous date were considered against Definition and Scope; (2) arXiv IDs and metadata were checked; (3) duplicate and badge checks were run; (4) List Statistics were recomputed with node scripts/check-list-stats.mjs --update README.md; (5) search coverage and decisions were recorded under curation/. Structural changes are logged in News.

Finding code and reproducible stacks

  • Skim any section for GitHub badges — they always mean official code.
  • For end-to-end stacks (training recipes, checkpoints, serving), go straight to Open Toolkits & Platforms, which collects Cosmos, minWM, Matrix-Game, Genie Envisioner, DreamerV3, TD-MPC2, V-JEPA 2, OpenDWM, and others.
  • Leaderboards and datasets have their own subsections under Community Resources.

What this list is not

This is not a video-generation dump. A video model appears only when it explicitly targets control, causality, memory, or world-model conversion — that boundary is stated at the top of §1.6 General Video World Models & Rollout Backbones and enforced even more tightly in §1.7 Persistent Narrative & Multi-Shot Video, which requires documented state, memory, or structured planning that crosses a shot boundary. Length, resolution, visual quality, and identity consistency alone never qualify a paper. If you want a broad video-generation list, several are linked in Curated Lists & Awesome Repos.

04 / Start here

Taxonomic Overview

Source ↗
World Models
│
├── 0. Mind World Models (Cognitive / Biological Grounding)
│   ├── 0.1 Cognitive & neuroscientific origins (Craik → Tolman → predictive coding → JEPA)
│   └── 0.2 Formative computational papers (Schmidhuber 1990 → Dyna → DreamerV3)
│
├── 1. Generative World Models  [focus: synthesizing plausible futures]
│   │
│   ├── By Domain / Task
│   │   ├── 1.1 Game & Interactive Simulation
│   │   │     1.1.1 Pixel-space engines (GAN / CNN / diffusion)
│   │   │     1.1.2 Autoregressive transformers
│   │   │     1.1.3 Memory-augmented & long-horizon game worlds
│   │   ├── 1.2 Autonomous Driving
│   │   │     1.2.1 Multi-view camera  ·  1.2.2 Occupancy / BEV
│   │   │     1.2.3 LiDAR / 4D points  ·  1.2.4 Language-guided
│   │   ├── 1.3 Embodied AI & Robotics
│   │   │     1.3.1 Manipulation  ·  1.3.2 Navigation  ·  1.3.3 Locomotion
│   │   │     1.3.4 VLA & World Action Models (WAM)  ·  1.3.5 Real2Sim
│   │   ├── 1.4 3D / 4D Scene Generation
│   │   ├── 1.5 Scientific & Physical World Modeling
│   │   ├── 1.6 General Video World Models & Rollout Backbones
│   │   └── 1.7 Persistent Narrative & Multi-Shot Video World Models
│   │
│   └── By Architecture (cross-domain) — see Architecture Cheat Sheet
│       ├── Diffusion video WM
│       ├── Autoregressive transformer WM
│       ├── RSSM / Dreamer  ·  JEPA
│       ├── Occupancy / BEV  ·  3DGS / NeRF
│       └── LLM text WM  ·  World Action Models
│
├── 2. Representational World Models  [focus: learning structured internal state]
│   ├── 2.1 Latent Dynamics Models (RSSM / Dreamer)
│   ├── 2.2 Joint Embedding Predictive Architectures (JEPA)
│   ├── 2.3 Occupancy & BEV Representations
│   ├── 2.4 Multimodal, Text, Acoustic & Memory-Oriented World Models
│   └── 2.5 Symbolic & Knowledge-Graph World Models
│
└── 3. Agentic World Models  [focus: acting, planning, decision-making]
    │   (= World Foundation Model + Agentic Framework)
    ├── 3.1 Model-Based Reinforcement Learning (MBRL)
    ├── 3.2 World-Model-Guided Planning
    ├── 3.3 Closed-Loop Simulation & Evaluation
    ├── 3.4 Multi-Agent World Models
    ├── 3.5 Safety-Aware Agentic World Models
    └── 3.6 LLM / VLM / GUI Agents with World Models

Working definitions

TermDefinition
Mind World ModelThe biological and cognitive intuition that an intelligent system carries an internal model of the world and uses it for prediction, imagination, and counterfactual reasoning.
Generative World ModelPredicts or synthesizes plausible future observations, often pixels, video, occupancy, or point clouds.
Representational World ModelPredicts future state or latent structure without requiring photorealistic decoding.
World Foundation Model (WFM)A pretrained model of environment structure and dynamics that can support simulation, planning, forecasting, or data generation across downstream tasks.
JEPAA joint-embedding predictive architecture: predict future or missing representations, not pixels. The non-generative counterpoint to Section 1; see §2.2.
World Action Model (WAM)A model that jointly predicts futures and actions in one backbone, typically initialized from video generation. Distinct from a VLA, which maps observation + language directly to actions. See §1.3.4.
Agentic World ModelA WFM coupled with action selection, planning, memory, tool use, or policy optimization in a closed loop.
05 / Research practice

Reading Roadmap

Source ↗

Three tracks. Each step names papers that are already entries in this list; follow the section link to find full citations, badges, and code.

Track 1 — Newcomer (build the concept from zero)

Goal: understand what a world model is, where the idea comes from, and what the modern instantiations look like — roughly 7 stops.

  1. The definition. Read Definition and Scope and the working definitions under Taxonomic Overview. Ten minutes that prevent months of terminology confusion.
  2. The seminal paper. World Models (Ha & Schmidhuber, 2018) and its NeurIPS version Recurrent World Models Facilitate Policy Evolution — in §0.1–0.2. Learn V-M-C: compress perception, predict in latent space, train a controller entirely inside the dream.
  3. Latent dynamics done right. PlaNet (RSSM) and Dream to Control (Dreamer), then skim DreamerV2/V3 — in §0.2 and the comparison table in §2.1. This is the reinforcement-learning lineage of the field.
  4. The generative-interactive turn. Genie: Generative Interactive Environments, then the Genie 2 and Genie 3 lab reports — in §1.1.2. Latent actions learned from unlabeled video; playable worlds from a prompt.
  5. The non-generative counterpoint. V-JEPA (and LeCun's A Path Towards Autonomous Machine Intelligence, §0.1) — in §2.2. Predict representations, not pixels; understand why this is an argument, not just an architecture.
  6. One driving paper. GAIA-1 — in §1.2.1. The first large-scale autoregressive driving world model; sets up everything that follows in §1.2. (OccWorld in §1.2.2 is the natural second read for the geometry-first side.)
  7. One robotics paper. UniSim: Learning Interactive Real-World Simulators — in §1.3.1. A learned action-conditioned simulator of real-world interaction; the conceptual bridge to WAMs.

After these seven, the Historical Timeline and Architecture Cheat Sheet below will read as review rather than news.

Track 2 — Practitioner (build something this quarter)

Goal: pick a stack with open weights or code and a known deployment story. All of these have GitHub badges and live under Open Toolkits & Platforms plus their taxonomy homes.

StackWhat it gives youWhere
NVIDIA Cosmos (incl. Cosmos-Predict2.5, Cosmos 3)Open world-foundation-model platform for Physical AI: pretrained video WFMs, post-training recipes, driving pipeline (Cosmos-Drive-Dreams)Toolkits, §1.6
DreamerV3Reference latent-dynamics MBRL agent; single hyperparameter set across domains; the default baseline for imagination-based RL§2.1, §3.1
TD-MPC2Scalable latent MPC for continuous control (104 tasks); decoder-free, robust defaults§2.1, §1.3.3
V-JEPA 2Self-supervised video representation + action-conditioned latent planning; zero-shot manipulation recipes; official checkpoints on HuggingFace§2.2, §1.3.4
Matrix-Game (1.0 → 3.0)Open interactive game-world stack: real-time streaming rollouts, long-horizon memory in 3.0§1.1.1, Toolkits
Genie EnvisionerUnified robotic-manipulation world platform: imagination, policy evaluation, and data generation in one loop (AgiBot)§1.3.1, Toolkits

Supporting picks, depending on the problem: minWM and Causal Forcing for converting a video backbone into a real-time interactive world model (§1.6); OpenDWM for driving; stable-worldmodel and Nano World Models for controlled research baselines (all under Toolkits).

Track 3 — Researcher (find the frontier)

  1. Surveys first. From the Surveys & Position Papers tables: Understanding World or Predicting Future? for the broad taxonomy, Is Sora a World Simulator? for the generative debate, World Action Models: A Survey and World Action Models: The Next Frontier for the WAM consolidation, plus the domain surveys for driving and embodied AI.
  2. Theory and safety. The Safety & Theory table (When Does LeJEPA Learn a World Model?, General Agents Contain World Models, Critiques of World Models, identifiability and value-equivalence results in §2.1–§2.2) and §3.5 Safety-Aware Agentic World Models for the attack-surface literature.
  3. Benchmarks. Read Evaluation Dimensions below as the index, then go metric-shopping in Benchmarks & Evaluation. Pay attention to closed-loop utility benchmarks (World-in-World, WorldGym, WorldEval) versus rollout-quality benchmarks — they disagree, and that disagreement is a research topic.
  4. Open problems. The Open Problems section below distills what the 2025–2026 surveys actually argue about, with entry points into the taxonomy for each.
06 / Research practice

Historical Timeline

Source ↗

Two eras, compact by design. Every named work below is an entry in this list — follow the section link for the full citation, badges, and one-line rationale.

Cognitive and computational origins (1943–2018)

YearMilestoneWhy it matters
1943Craik, The Nature of Explanation — §0.1First articulation of the mind carrying a "small-scale model" of external reality used to try out alternatives before acting.
1948Tolman, Cognitive Maps in Rats and Men — §0.1Latent spatial representations inferred from behavior; the empirical ancestor of learned internal state.
1978O'Keefe & Nadel, The Hippocampus as a Cognitive Map — §0.1Place cells as the neural substrate of Tolman's map; foundation for spatial world-model research.
1983Johnson-Laird, Mental Models — §0.1Reasoning over constructed internal models of situations rather than formal logic.
1989Occupancy Grids (Elfes) — §0.1First computational spatial world model for a physical agent; ancestor of §2.3.
1990Schmidhuber, Making the World Differentiable — §0.2Recurrent controller–model architecture cited by Ha & Schmidhuber (2018) as the direct ancestor of learned neural world models.
1991Sutton, Dyna — §0.2Learning, planning, and reacting integrated through imagined experience; "Dyna-style rollouts" survive in the MBRL table.
1993Successor Representations (Dayan) — §0.1Encode future occupancy of states rather than immediate reward — predictive representation before deep learning.
1995Wolpert et al., internal models for sensorimotor integration — §0.1Forward models predict sensory consequences of motor commands — the neuroscience blueprint for action-conditioned prediction.
1999Rao & Ballard, predictive coding in visual cortex — §0.1Concrete computational predictive-coding model; higher areas predict lower-level activity.
2010Friston, free-energy principle — §0.1The brain as a hierarchical prediction-error-minimizing machine.
2011PILCO — §0.2Gaussian-process dynamics with analytic uncertainty; the data-efficiency reference point for model-based policy search.
2013Battaglia et al., mental simulation / intuitive physics — §0.1Humans run fast approximate physics simulations as a world model.
2015E2C and Action-Conditional Video Prediction — §0.2Latent dynamics from pixels, and deep action-conditioned next-frame prediction at Atari scale.
2017VPN, I2A, Successor Features — §0.2Value-equivalent abstract models, imagination-augmented agents, and deep successor representations.
2018World Models (Ha & Schmidhuber) + NeurIPS version; TDM; GQN — §0.1–0.2The term enters modern ML: V-M-C, training a controller inside the dream; also goal-conditioned implicit dynamics and neural scene rendering.

The scaling era (2018–2026)

YearMilestones (all entries in this list)Where
2019PlaNet (ICML) introduces RSSM and latent-space planning; MBPO (NeurIPS) formalizes Dyna-style model-based policy optimization; DPI-Net (ICLR) brings graph-network particle physics.§0.2, §3.1, §1.5.1
2020Dreamer (ICLR) trains actor-critic fully in imagination; MuZero (Nature) plans with a learned value-equivalent dynamics model, no rules given.§0.2, §3.1
2021DreamerV2 (ICLR) makes discrete latents work; EfficientZero (NeurIPS) reaches Atari sample-efficiency milestones; Pathdreamer (ICCV) is an early visual world model for navigation.§0.2, §3.1, §1.3.2
2022LeCun's position paper proposes JEPA-centered autonomous machine intelligence; TD-MPC (ICML) fuses TD learning with latent MPC; Iso-Dream (NeurIPS) disentangles controllable dynamics.§0.1, §2.1
2023DreamerV3 generalizes across domains with fixed hyperparameters; I-JEPA (CVPR) lands the JEPA program in vision; GAIA-1 (Wayve) is the first large-scale generative driving world model; UniPi turns text-guided video generation into policies; Pangu-Weather (Nature) and GraphCast (Science) show learned earth-system dynamics beating traditional simulation.§0.2, §2.2, §1.2.1, §1.3.4, §1.5.2
2024OpenAI frames Sora as a "world simulator", igniting the debate (Is Sora a World Simulator? survey; PhyWorld physical-law critique); Genie learns latent actions from unlabeled video; GameNGen runs DOOM in a diffusion model in real time; Oasis generates Minecraft token-by-token; V-JEPA (ICLR) and TD-MPC2 (ICLR) mature the representational side; OccWorld (ECCV) and Copilot4D (ICLR) establish occupancy/LiDAR world models; Genie 2 (December) generates playable 3D worlds from one image; DIAMOND (NeurIPS) shows diffusion world models paying off for RL.Surveys, §1.5.1, §1.1, §2.2, §2.1, §1.2.2–1.2.3, §3.1
2025NVIDIA Cosmos ships open world foundation models for Physical AI (January), extended by Cosmos-Predict2.5 and Cosmos-Drive-Dreams; GAIA-2 adds controllable multi-view driving; V-JEPA 2 demonstrates zero-shot robot manipulation from internet-scale video pretraining; Matrix-Game and Matrix-Game 2.0 open-source real-time interactive game worlds; Genie 3 (August) reaches real-time 24 fps text-to-world generation; Genie Envisioner unifies robot imagination, evaluation, and data generation; HunyuanWorld 1.0 generates explorable 3D worlds; DreamerV4 scales agent-side world-model training; PAN targets general long-horizon interactive simulation.Toolkits, §1.2.1, §2.2, §1.1, §1.3.1, §1.4.1, §3.1, §1.6
2026Cosmos 3 unifies language, image, video, audio, and action in one omnimodal WFM family; HY-World 2.0 and Matrix-Game 3.0 push open 3D/interactive stacks; V-JEPA 2.1 densifies JEPA video features; the World Action Model (WAM) wave consolidates — dedicated surveys (World Action Models: A Survey; The Next Frontier), open stacks (DreamZero), and a dense §1.3.4 of video-action models; memory, evaluation, and safety become first-class subfields (§1.1.3, §3.3, §3.5).§1.6, §1.4.1, §1.1.3, §2.2, §1.3.4, Surveys, §3.3, §3.5
07 / Research practice

Architecture Cheat Sheet

Source ↗

Eight architecture families that account for nearly every entry in this list. "State" is what the model carries between steps; "prediction target" is what it is trained to output; "failure modes" are the documented ones, not hypotheticals. Canonical papers are all entries here — follow the section links for citations and code.

FamilyStatePrediction targetAction conditioningStrengthsFailure modesCanonical papers (in this list)
Diffusion video WMImplicit — a window of recent frames or video latentsFuture frames (pixel or VAE-latent), denoisedActions/trajectories/text injected as conditioning; often weak by defaultVisual fidelity; inherits video-generation pretraining; multimodal futuresCompounding error over long rollouts; action conditioning ignored under classifier-free guidance; slow sampling without distillationGameNGen, DIAMOND (§1.1.1); Vista (§1.2.1); Cosmos family (§1.6)
AR transformer WMDiscrete token history (VQ codes) with KV cacheNext visual tokens / framesInterleaved action tokens or learned latent actionsStreaming and real-time by construction; unified with LLM tooling; latent actions learnable from unlabeled videoTokenizer artifacts; finite context → spatial forgetting; exposure biasGenie, Oasis, MineWorld (§1.1.2); iVideoGPT (§1.6); DrivingGPT (§1.2.4)
RSSM / Dreamer familyCompact deterministic + stochastic latentNext latent state (+ reward, value; decoder optional)Explicit action input to the transition functionExtremely cheap rollouts → sample-efficient RL in imagination; stable training recipesLimited visual capacity; mostly proven at simulator scale; latent hallucination outside the data manifoldPlaNet, Dreamer, DreamerV2/V3 (§2.1); DreamerV4 (§3.1); TD-MPC2 (decoder-free relative, §2.1)
JEPAEmbedding produced by a target encoderRepresentation of future/masked content — no pixel decoding (energy-based objective)Optional: action-conditioned predictor (V-JEPA 2, DUET-DINO)Ignores unpredictable pixel detail; strong transfer; cheap planning in representation spaceNo renderable output for humans; representation collapse without careful regularization; evaluation is indirectI-JEPA, V-JEPA, V-JEPA 2/2.1, LeWorldModel (§2.2)
Occupancy / BEV WMExplicit 3D voxel occupancy or BEV gridFuture occupancy / BEV framesEgo trajectory, agent commands, language (OccLLaMA)Metric geometry; direct planner interface; sensor-fusion friendlyResolution–memory trade-off; appearance-free (needs a renderer for photorealism); semantic sparsityOccWorld, Drive-OccWorld, DOME (§1.2.2); OccSora, BEVWorld (§2.3)
3DGS / NeRF worldsExplicit persistent 3D scene (Gaussians, fields, meshes)Novel-view renders + scene evolutionCamera trajectory; object-level edits; physics add-ons3D consistency by construction; revisitable, editable, engine-loadable worldsDynamics usually bolted on; costly scene construction; closed-world assumptionEmerNeRF, 4D Gaussian Splatting, HunyuanWorld 1.0, LayerPano3D (§1.4.1); GWM, GaussianWorld (§1.3.1, §1.2.2)
LLM text WMTextual / symbolic state descriptionNext state description, transition validity, or executable programText actions; tool callsAbstract and counterfactual reasoning; composable with agent frameworks; cheapState drift and hallucination over steps; weak physical/spatial grounding; hard to verifyLLM-Sim (§2.4); RAP, WebDreamer, CWM (§3.6); Text2World, PoE-World (§2.5)
WAM (world action model)Shared video–action latentJoint: future video (or latent) and actionsIntrinsic — action is an output as much as an inputPolicy and simulator in one model; transfers video pretraining into control; zero-shot policy resultsInference cost of imagining before acting; video-action generalization gap; evaluation protocols still immatureWorldVLA, UWM, UVA, DreamZero, LingBot-VA (§1.3.4); WAM surveys (Surveys)

Reading the table: the top half trades fidelity against control (diffusion vs. AR), the middle trades capacity against efficiency (RSSM/JEPA vs. pixel models), and the bottom half trades structure against openness (occupancy/3D worlds vs. text vs. joint video-action). Most 2026 systems are hybrids that pick one row as a backbone and borrow mechanisms from two others.

08 / Research practice

Open Problems

Source ↗

Twelve questions the 2025–2026 surveys and position papers in this list actually argue about — not a wish list. Each problem names its entry points here.

  1. Compounding error over long horizons. Autoregressive rollouts drift off-manifold; every mitigation (self-forcing, history guidance, rolling windows, distillation) trades something else away. When is drift a training-objective artifact versus a fundamental limit of learned single-step dynamics? Entry points: the forcing-family recipes and long-context models in §1.6; Orbis (§1.2.1).
  2. Action controllability and grounding. Video backbones absorb actions as weak conditioning and often ignore them; latent actions learned from unlabeled video may not align with executable controls. How do we guarantee — and measure — that actions cause futures? Entry points: ACT-Bench, VRAG Benchmark (Benchmarks); Genie's latent actions (§1.1.2); AdaWorld, WALA (§1.3.1, §1.3.4).
  3. Physical grounding vs. photorealism. Scaling improves appearance faster than physics; models interpolate visual statistics rather than learning laws. Does physical competence require explicit structure (Hamiltonian latents, differentiable simulators, occupancy) or only better data and probes? Entry points: PhyWorld, Physically Native World Models, PhysCoRe (§1.5.1, §1.3.1); Physical Grounding in World Models (Surveys).
  4. Memory and persistent state. Pixel-history context is not a world state: off-screen content decays, revisits contradict earlier generations. Explicit 3D state, retrieval, and hierarchical memory all help and all cost; none is settled. Entry points: §1.1.3 as a whole; Beyond Pixel Histories (§1.4.2); On Memory (§2.4); MBench, STEVO-Bench (Benchmarks).
  5. Evaluation itself. Benchmarks have multiplied faster than agreement on what they measure; rollout-quality metrics and closed-loop utility rank models differently, and most metrics need ground-truth futures that interventions destroy. Entry points: Evaluation Dimensions; Validate the Dream, Reference-Free Physical Consistency (§3.3); World-in-World (Benchmarks).
  6. Safety, robustness, and the trusted-imagination attack surface. Imagined futures now gate real actions, which makes the world model itself a target: adversarial contexts, data poisoning that only manifests downstream, and optimistic rollouts that hide failures. Entry points: BadWorld, World-Model Supply-Chain Poisoning, Trusted Imagination Attacks, World Models in Pieces (§3.5); RoboTrustBench (Benchmarks).
  7. Data: action labels are the bottleneck. Internet video is abundant but action-free; robot data is labeled but tiny and embodiment-specific. Latent actions, inverse dynamics, human-video transfer, and synthetic data engines each cover part of the gap. Entry points: WALA, EgoWAM, LaST-HD (§1.3.4); DreamDojo, PlayWorld (§1.3.1); Geographic Diversity for JEPA Driving WMs (§1.2.4).
  8. Sim-to-real and real-to-sim closure. When can a policy trained or validated inside a learned world model be trusted on hardware — and can real recordings be lifted into simulation-ready twins automatically? Entry points: §1.3.5; Efficient Sim-to-Real WAM (§1.3.4); RWM-U, Robotic World Model (§1.3.3); Quadrotor WM Generalization (§1.3.2).
  9. Multi-agent shared worlds. Almost everything in this list models one agent's view; shared, jointly consistent worlds with other goal-directed agents (traffic, multiplayer, social dynamics) are barely started. Entry points: §3.4; Solaris, Multiplayer Interactive World Models (§1.1.2); SceneDiffuser++ (§1.2.4).
  10. Are video generators world models? The Sora debate, still unresolved: implicit dynamics demonstrably emerge, and demonstrably violate physical law out of distribution. The productive version of the question is what additional structure converts one into the other. Entry points: Is Sora a World Simulator?, Mechanistic View on Video Generation as WMs, Critiques of World Models (Surveys); the §1.6/§1.7 boundary notes (§1.6, §1.7).
  11. Reconstruction vs. representation. Should the model predict pixels at all? Decoder-free (JEPA, TD-MPC2) and decoder-optional designs are cheaper and sometimes plan better, but are harder to inspect and evaluate. Entry points: Reconstruction or Semantics? (Surveys); ImageWAM, Fast-WAM (§1.3.4); §2.2 theory entries.
  12. Real-time inference economics. Interactive world models must generate under strict latency budgets; distillation, delta tokens, keyframe sparsity, and flash-style serving all exist because full-fidelity imagination is currently too slow to act on. Entry points: A Frame is Worth One Token (§1.1.1); SKIP, LaWAM (§1.3.1, §1.3.4); minWM, MoWorld, FlashDreams (§1.6, Toolkits).
09 / Research practice

Evaluation Dimensions

Source ↗

"Evaluating a world model" means at least eleven different things, and a single leaderboard number conflates them. This section is the conceptual index into Benchmarks & Evaluation; every benchmark named below is an entry in that table or in §3.3.

DimensionQuestion it answersTypical measurementsWhere to look in this list
1. Visual fidelityDo rollouts look like real observations?FVD/FID-style distances, human preference, per-frame qualityWorldModelBench, DrivingGen, EWMBench, WorldSimBench
2. Action controllabilityDo different actions produce correctly different futures?Action-following accuracy, instruction adherence, trajectory-conditioned errorACT-Bench, VRAG Benchmark, MiraBench, iWorld-Bench, MIND (control axis)
3. 3D / geometric consistencyIs the implied 3D world stable across viewpoints and revisits?Reprojection/loop-closure error, multi-view consistency, camera-controlled probingViewBench, WRBench, 4DWorldBench, WorldScore, PDI-Bench, RoboPhys-3D, Toward Memory-Aided World Models (LoopNav)
4. Physical plausibilityDoes the rollout obey mechanics, permanence, and conservation?Physics-law probes, intuitive-physics batteries, commonsense violation ratesPhysics-IQ Verified, VideoPhy-2, WorldBench, PhysWeep, PhysicsMind, Tailor-Bench, WorldOlympiad
5. Long-horizon memoryDoes content that left the view come back correct?Revisit consistency, occlusion probes, minute-scale drift metricsMBench, MIND, STEVO-Bench, R2M-Bench, GUI-CC, Toward Stable World Models, Omni-WorldBench
6. Closed-loop policy utilityDoes the model actually help an agent act?Real-task success of policies trained/evaluated inside the model; sim-vs-real ranking agreementWorld-in-World, WorldGym, WorldEval, PiL-World, GigaWorld-1 / WMBench, ReactSim-Bench, RoboWM-Bench (§3.3)
7. Sample efficiencyHow little real experience does model-based learning need?Score at fixed interaction budgetAtari 100k, DMControl Suite, ProcGen, Minecraft Diamond (DreamerV3)
8. Safety & robustnessDoes the model resist perturbations, misuse, and optimistic bias?Adversarial-context degradation, unsafe-instruction rejection, poisoning detection, optimism-bias probesRoboTrustBench, ARB4WM, MiraBench (optimism bias), MMBench2 (hallucination), plus the attack literature in §3.5
9. Counterfactual fidelityDo intervention-edited rollouts diverge the way the world would?Intervention/counterfactual probe accuracy, causal-consistency scoringWhat-If World, RoboTrustBench (counterfactual axis), WM-ABench (atomic internal-model probes)
10. Distributional calibrationDo repeated futures reproduce the range and probabilities of possible outcomes?Outcome-frequency calibration and distribution coverage under fixed initial state and actionPAWBench
11. Decision validityDoes a model rank actions and justify additional planning or adaptation compute?Fixed-candidate action ranking, matched intervention effects, update-versus-hold utility, total compute costARC-Bench; Intervention Gap (Surveys); Counterfactual Update Utility and Compute-Value Audit (§3.3)

Three cautions, all documented in entries here:

  • Dimensions 1 and 4 dissociate. High visual fidelity with broken physics is the normal failure mode, not the exception (PhyWorld, §1.5.1).
  • Dimensions 1–5 do not predict dimension 6. Rollout-quality metrics and policy-utility outcomes can rank models differently; this is why §3.3 exists as its own subsection, and why Validate the Dream argues for admissibility checks before trusting simulator verdicts.
  • Reference-free evaluation is still open. Most metrics need ground-truth futures that interventions make unavailable; see Reference-Free Physical Consistency in §3.3.
10 / Reference

Glossary

Source ↗

Precise working definitions, in the sense used throughout this list. Alphabetical. Terms in italics are cross-references within the glossary.

  • Action-conditioned rollout — Generating a future trajectory (frames, latents, occupancy) where each step is conditioned on a supplied action, so different action sequences must produce different futures. The minimum bar separating a world model from a video generator; benchmarked by ACT-Bench and the VRAG Benchmark (Benchmarks).
  • Agentic world model — A world foundation model coupled with action selection, planning, memory, tool use, or policy optimization in a closed loop. The organizing idea of Section 3.
  • BEV (bird's-eye view) — A top-down metric grid representation of a scene, standard in driving. BEV world models predict future BEV frames; see §2.3.
  • Causal forcing — A training/distillation recipe that converts bidirectional video diffusion into causal (past-only) autoregressive generation suitable for real-time interaction; named after the Causal Forcing line of work in §1.6.
  • Closed-loop vs. open-loop — Open-loop: the model predicts a future once, from a fixed prompt/context, and is scored against ground truth. Closed-loop: the model's outputs feed back into its own inputs (or a policy acts inside it) over many steps, so errors can compound and interventions matter. Closed-loop evaluation is the stricter and more decision-relevant regime; see §3.3.
  • Compounding error (rollout drift) — Accumulation of small per-step prediction errors during autoregressive rollout, driving generated futures off the data manifold. The central engineering obstacle of §1.6; mitigations include self-forcing, history guidance, memory modules, and 3D anchoring.
  • Counterfactual — A "what would have happened if" query: same initial state, different action or intervention. A world model with counterfactual fidelity produces futures that diverge correctly under such edits; benchmarked by What-If World and parts of RoboTrustBench (Benchmarks).
  • Diffusion forcing — A training objective mixing next-token-style causal prediction with full-sequence diffusion, giving per-frame noise levels; a backbone recipe for controllable causal video rollouts (§1.6).
  • Digital twin vs. world model — A digital twin is an instance-specific, engineered replica of one particular asset or site, kept synchronized with it. A world model is a learned, generalizing predictive model of environment dynamics. Twins can be built from world models (see Real2Sim, §1.3.5) and world models can be trained from twins, but the terms are not interchangeable; see the Digital Twin AI survey (Surveys).
  • Dyna-style rollout — Using a learned model to generate imagined transitions that augment real experience for policy learning (after Sutton's Dyna architecture). MBPO in §3.1 is the canonical deep-RL instantiation.
  • Energy-based JEPA — LeCun's formulation in which a predictor is trained to make representations of compatible (context, target) pairs low-energy, without reconstructing pixels — avoiding wasted capacity on unpredictable detail. The theoretical program behind §2.2.
  • Generative world model — Predicts or synthesizes plausible future observations (pixels, video, occupancy, point clouds), typically usable as a learned simulator or data engine. Section 1.
  • Imagination — Rolling the world model forward without touching the real environment, to train a policy (Dreamer), plan (MPC/MCTS), evaluate a policy, or synthesize data. "Training in imagination" means the policy never sees real transitions during optimization.
  • Inverse dynamics model (IDM) — A model that infers the action connecting two observed states. Used to label action-free video, to ground latent actions, and to turn generated videos into executable robot commands (§1.3.4).
  • Latent action — An action representation learned from unlabeled video rather than recorded controls (Genie, AdaWorld). Enables interactive control of models trained on internet-scale data without action labels; must be mapped to executable controls for robotics.
  • Latent dynamics model — A world model whose transition function operates on a compact learned state rather than observations; the RSSM/Dreamer family in §2.1 is the reference implementation.
  • Long-horizon memory — Mechanisms (explicit 3D state, retrieval, surfel/keyframe caches, state-space models) that keep a rollout consistent with content generated many steps ago, including content that left the field of view. See §1.1.3 and the memory benchmarks MBench, MIND, and STEVO-Bench (Benchmarks).
  • MBRL (model-based reinforcement learning) — RL that learns and exploits a dynamics model for sample efficiency, via imagined training, planning, or both. §3.1.
  • MPC (model-predictive control) — At each step, optimize a short action sequence against the world model's predicted futures, execute the first action, re-plan. The standard way to use latent world models for control without a learned policy (PlaNet, TD-MPC2).
  • Neural simulator — A learned model used in place of a hand-built simulator: action-in, observation-out, at interactive rates, with enough fidelity to train or evaluate policies (UniSim, RoboWorld, NVIDIA OmniDreams). The claim is functional, not architectural.
  • Occupancy (grid) — A voxelized representation marking which regions of 3D space are occupied (optionally with semantics). The oldest world-model formalization in this list (Elfes 1989, §0.1) and a mainline of driving world models (§1.2.2, §2.3).
  • Open-loop evaluation — See closed-loop vs. open-loop.
  • Physical plausibility vs. photorealism — Orthogonal axes: a rollout can look real while violating conservation laws, object permanence, or contact dynamics. PhyWorld, Physics-IQ Verified, and VideoPhy-2 (Benchmarks) measure the physics axis specifically.
  • Policy-in-the-loop evaluation — Scoring a world model by how well a policy trained or evaluated inside it transfers to the real environment — the utility-centric alternative to visual metrics. WorldGym, WorldEval, PiL-World, World-in-World (§3.3, Benchmarks).
  • Predictive coding — Hierarchical inference in which higher areas predict lower-level activity and only prediction errors are propagated (Rao & Ballard 1999; Friston's free-energy generalization). The neuroscientific ancestor of JEPA-style representation prediction (§0.1, §2.2).
  • Real2Sim / Sim-to-Real — Real2Sim: constructing simulation-ready scene twins from real recordings (§1.3.5). Sim-to-Real: transferring a policy or model trained in simulation (or imagination) to the physical world. World models sit on both bridges.
  • Representational world model — Predicts future state or latent structure without requiring photorealistic decoding. Section 2.
  • RSSM (recurrent state-space model) — The PlaNet/Dreamer transition architecture: a deterministic recurrent path plus a stochastic latent path, trained with variational objectives; supports fast latent-space planning and imagination.
  • Self-forcing — Training an autoregressive video model on its own generated prefixes rather than ground-truth frames, closing the train–test gap that causes rollout drift. Contrast teacher forcing; see §1.6.
  • Streaming / real-time interactivity — The system accepts new user input after a generated prefix already exists and continues from it at interactive latency. Stricter than autoregression or generation speed alone — this is the §1.7 bar for "interactive".
  • Successor representation / successor features — A predictive state encoding of expected future occupancy (Dayan 1993), generalized to deep features by Barreto et al. (2017). Dynamics are represented independently of the reward, which is why they sit with representational world models (§0.1, §0.2).
  • Teacher forcing — Training a sequence model with ground-truth history as input at every step. Efficient, but the model never learns to recover from its own mistakes — the root cause of exposure bias in rollouts.
  • VLA vs. WAM — A VLA (vision-language-action model) maps observations and language directly to actions; any world knowledge is implicit. A WAM (world action model) explicitly couples future prediction and action generation — it can imagine, then act, or co-generate both. The boundary cases (VLAs with latent world-model regularizers) live in §1.3.4; see also Do World Action Models Generalize Better than VLAs? (Surveys).
  • Value equivalence — A world model is value-equivalent to the environment if it yields the same values (and therefore the same policy) even when it is inaccurate as an observation predictor (Grimm et al., §0.2). The theoretical justification for MuZero-style abstract models and for decoder-free latent dynamics.
  • WAM (world action model) — A model that jointly learns environment dynamics and action generation in one backbone, typically initialized from video generation. The fastest-growing family in this list (§1.3.4).
  • WFM (world foundation model) — A pretrained model of environment structure and dynamics reusable across downstream tasks (simulation, planning, forecasting, data generation) — e.g. Cosmos, Genie, V-JEPA 2. "Foundation" refers to pretraining breadth, not architecture.
  • World model — An internal predictive model of an environment that helps an agent answer: what will happen if I act, wait, intervene, or imagine an alternative future? Intentionally broader than model-based RL and narrower than "any model that understands the world" — see Definition and Scope.
11 / Reference

FAQ

Source ↗

Q1. Is Sora a world model? Unresolved, and this list treats it that way. OpenAI's 2024 report frames video generation models as world simulators (Blogs); the survey Is Sora a World Simulator? and the physical-law study PhyWorld (Surveys, §1.5.1) document both the emergent dynamics and the systematic violations. Operationally: a video generator qualifies for the taxonomy here only when it demonstrates action conditioning, state persistence, or evaluable world-model use — the boundary stated at the top of §1.6.

Q2. What is the difference between a VLA, a world model, and a WAM? A VLA maps observation + language directly to actions; a world model predicts what happens next under actions or interventions; a WAM does both in one backbone — it jointly predicts futures and actions. See the VLA vs. WAM glossary entry, §1.3.4, and the empirical comparison Do World Action Models Generalize Better than VLAs? (Surveys).

Q3. Why are generic video-generation papers excluded? Because implicit visual dynamics alone are below the bar. An entry needs at least two of: models state; predicts state evolution under action/intervention; supports imagination, planning, evaluation, or controllable simulation (Definition and Scope). Video length, resolution, identity consistency, and multi-shot output do not qualify by themselves — §1.7 spells this out for the hardest boundary cases.

Q4. How are duplicates handled? Every paper has exactly one home section, chosen by main technical role. Intentional cross-references are plain-text pointers like (see §1.2.3), not repeated links. CI runs node scripts/check-arxiv-duplicates.mjs README.md, which fails on any arXiv ID appearing twice unless it is allowlisted in .github/arxiv-duplicate-allowlist.json with a reason.

Q5. How do I add a paper? For a lightweight suggestion, open a paper-suggestion issue. For a curated addition, open a PR following CONTRIBUTING.md: use the existing entry format, link primary sources, place it in the most appropriate section, keep the one-line description factual, and run the lint commands in the checklist. Taxonomy changes need their own justification (which entries move, and why the current tree fails them).

Q6. A paper fits both Generative and Representational — where does it go? By its main technical contribution, not its outputs. A model that predicts latents and optionally decodes pixels is representational; a model whose contribution is the synthesized observation stream is generative; a model wrapped in planning/acting machinery is agentic. Domain placement comes second — see the routing rule in Definition and Scope.

Q7. Is my perception / segmentation / trajectory-forecasting paper in scope? Usually not on its own. Pure perception answers "what is in the scene now", not "what happens if". It enters when it carries a genuine world-modeling role — e.g. occupancy forecasting under ego action (§1.2.2) rather than occupancy estimation.

Q8. Is a physics simulator or a digital twin a world model? Not by default. Hand-built simulators execute engineered dynamics; digital twins mirror one specific asset. This list includes them when they are learned, when they are constructed automatically from observation (Real2Sim, §1.3.5), or when a learned model plays the simulator's role (neural simulators — see the Glossary). The Digital Twin AI survey (Surveys) covers the relationship.

Q9. What exactly is a "World Foundation Model"? A pretrained model of environment structure and dynamics that supports multiple downstream uses — simulation, planning, forecasting, data generation — as defined in the working definitions. Cosmos, Genie, and V-JEPA 2 are the canonical examples here. It is a claim about pretraining breadth and reusability, not about any particular architecture.

Q10. Why is a famous paper missing? Three common reasons: it is out of scope under the two-of-three boundary (most video generation and most VLAs); it is a secondary source (commentary, re-implementations, news); or it genuinely slipped through — in which case, see Q5. Absence is a scope judgment before it is an omission.

Q11. What do the GitHub badges guarantee? Official code or the official project repository only. Unofficial reimplementations are deliberately not badged, per CONTRIBUTING.md. If an official repo appears later, PRs updating the badge are welcome.

Q12. How often is the list updated, and what does a "curation pass" mean? The header states the date of the last curation pass. The September 15, 2026 audit records reference-list comparisons, arXiv metadata checks, new-paper screening, exclusions, and validation results. External project and code links are checked where added or corrected; this is not a claim that every older external URL was retested. The News section records structural changes. Counts in List Statistics are recomputed on each pass from the rules in that section's comment.

Q13. Why are some famous video generators (Sora, FramePack, MAGI-1, …) not listed as papers? They fail the two-of-three inclusion boundary in Definition and Scope, or they are secondary reports already covered by the Sora debate in FAQ Q1 and Surveys. Length, resolution, identity consistency, and multi-shot output alone never qualify a paper; §1.6 and §1.7 state the extra bar for video backbones.

Q14. A paper appears under a different name than I searched for. How do I find it? Search the page for the arXiv ID (the arXiv badge label carries the full ID). If the paper is filed under a later version or a sibling name, the one-line rationale usually mentions the connection (e.g. AlayaWorld v1.1 is noted on the existing AlayaWorld entry rather than duplicated).

12 / About the list

News

Source ↗
  • [2026-09-16] 🔎 Research update. We have updated the world model research list with additional papers and refreshed existing entries across the main research areas.

  • [2026-08-25] 🧭 Comprehensive refresh. Handbook front-matter (how to use, reading roadmap, timeline, architecture cheat sheet, glossary, evaluation dimensions, labs, open problems, FAQ, list statistics) plus a large paper pass covering missing classics and July–August 2026 work across games, driving, robotics/WAMs, physics, JEPA, agentic systems, benchmarks, and workshops.

  • [2026-07-11] 🎉 WorldFoundry and its companion repository Awesome World Modeling are now open source! We welcome ⭐ stars, bug reports, feature requests, discussions, and pull requests from the community.

Internal & External "World Model"Historical Wave Map
internal & external world modelHistorical wave map
13 / About the list

List Statistics

Source ↗
StatisticValue
Unique arXiv papers1498
Total curated entries (taxonomy bullets + §2.1 / §3.1 / Surveys / Benchmarks table rows)1557
Entries with official code (GitHub badges)388
Official project pages (Project badges)410
Taxonomy sections and subsections (numbered headings in §0–3)44
Benchmarks tracked93
Surveys & position papers tracked84
Glossary terms37
Open problems12
FAQ entries14
Last full curation passSeptember 15, 2026

Counts are generated and checked by scripts/check-list-stats.mjs; arXiv IDs are independently checked by scripts/check-arxiv-duplicates.mjs. The September 2026 audit corrected an earlier total that included 56 non-paper bullets.

14 / About the list

Citation

Source ↗

If you find this repository useful in your research, please cite this curated list:

@misc{openenvision2026awesomeworldmodels,
  title={Awesome World Modeling},
  author={{OpenEnvision}},
  year={2026},
  howpublished={GitHub repository},
  url={https://github.com/OpenEnvision/Awesome-World-Modeling},
  note={A scope-aware, paper-first curated list of world model research}
}
15 / About the list

Contribution Guide

Source ↗

PRs are welcome. For lightweight suggestions, open a paper suggestion issue. For curated additions or taxonomy changes, please open a pull request and follow CONTRIBUTING.md.

The preferred entry format (matching the rest of this file) is:

- **Paper / Project Name** — Authors. "Full Paper Title." *Venue* YEAR.
  [![arXiv](https://img.shields.io/badge/arXiv-XXXX.XXXXX-b31b1b?logo=arxiv&logoColor=white)](https://arxiv.org/abs/XXXX.XXXXX)
  [![Project](https://img.shields.io/badge/Project-Page-0A66C2?logo=googlechrome&logoColor=white)](https://...)
  [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/...)
  > One-sentence factual reason this entry is in scope.

For table sections, keep the current columns and add only one concise scope or metric phrase.

Badge conventions:

  • arXiv — papers on arxiv.org (include the full arXiv ID in the badge label)
  • GitHub — official open-source code or the official project repository
  • Project — official project webpage
  • HuggingFace — official model, dataset, space, or leaderboard
  • Blog — technical blog post or official research-lab write-up
  • Paper — non-arXiv primary source: DOI, OpenReview, or proceedings page