WorldModelBench

About

WorldModelBench asks whether generated videos behave like world models. It contains 350 public test instances across 7 domains and 56 subdomains. Each instance provides a first-frame description, an instruction, and a first-frame image. Text-to-video models use the description plus instruction; image-to-video models use the first frame plus instruction.

WorldFoundry already includes the WorldModelBench evaluator wrapper and runtime under worldfoundry/evaluation/tasks/execution/runners/worldmodelbench. Do not clone the official repos for a WorldFoundry run. You still need the data assets and the VILA judge checkpoint.

What It Measures

WorldModelBench judges whether video generators behave as world models — not only whether clips look good, but whether they follow instructions, preserve temporal common sense, and respect physical laws. The benchmark crowdsourced 67K human labels to train a human-aligned VILA judge that automates evaluation.

Benchmark Design

Physical-adherence checks target subtle violations such as size drift, unnatural deformation, impossible fluid flow, object penetration, and gravity inconsistency — failures that generic video-quality metrics often miss.

Official References

• Paper: arXiv:2502.20694
• Project page and leaderboard: worldmodelbench-team.github.io
• Official source reference: github.com/WorldModelBench-Team/WorldModelBench
• Test data: Efficient-Large-Model/worldmodelbench
• Judge model: Efficient-Large-Model/vila-ewm-qwen2-1.5b
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/run_worldmodelbench_official_runner.py

Leaderboard Notes

The public leaderboard reports Instruction Following, Physics Adherence, Common Sense, and a Total Score. Official answers and explanations are withheld for test integrity; submit results to worldmodelbench.team@gmail.com per the project page. WorldFoundry normalizes local judge output into scorecard.json but does not host the upstream submission workflow.

Prepare Data And Assets

Run from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export PYTHONPATH="$PWD:${PYTHONPATH:-}"

Create a data root with the manifest, first-frame images, and evaluator file layout expected by the runner:

<worldmodelbench_data_root>/
  worldmodelbench.json
  images/
  evaluation.py

WorldFoundry ships the public 350-instance manifest:

export WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT=/path/to/worldmodelbench
mkdir -p "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}"
cp worldfoundry/data/benchmarks/assets/worldmodelbench/worldmodelbench.json \
  "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}/worldmodelbench.json"

If you use a Hugging Face mirror for the data assets, place it in the same root:

hf download Efficient-Large-Model/worldmodelbench \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}"

Download the judge checkpoint:

export WORLDFOUNDRY_WORLDMODELBENCH_JUDGE=/path/to/vila-ewm-qwen2-1.5b
hf download Efficient-Large-Model/vila-ewm-qwen2-1.5b \
  --local-dir "${WORLDFOUNDRY_WORLDMODELBENCH_JUDGE}"

Generate one video per manifest row. The output filename must match the first-frame stem:

images/69620089860948e38a4921dd4869d24f.jpg
=> <generated_videos>/69620089860948e38a4921dd4869d24f.mp4

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/worldmodelbench/generated_videos

Prompt construction follows the local README:

# text-to-video
" ".join([instance["text_first_frame"], instance["text_instruction"]])

# image-to-video
image = instance["first_frame"]
text = instance["text_instruction"]

Run With WorldFoundry

Public command for importing a completed worldmodelbench_results.json:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id worldmodelbench \
  --mode official-validation \
  --official-results-path /path/to/worldmodelbench_results.json \
  --benchmark-data-root "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/worldmodelbench/official-validation \
  --json

Direct runner command for judge execution through the bundled runtime:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/run_worldmodelbench_official_runner.py \
  --worldmodelbench-root worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/runtime \
  --data-root "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}" \
  --video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --judge "${WORLDFOUNDRY_WORLDMODELBENCH_JUDGE}" \
  --model-name "${WORLDFOUNDRY_MODEL_NAME:-candidate_model}" \
  --output-dir tmp/worldmodelbench/direct-runner \
  --json

Direct runner command for importing a completed result file:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/run_worldmodelbench_official_runner.py \
  --data-root "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}" \
  --video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --official-results-path /path/to/worldmodelbench_results.json \
  --output-dir tmp/worldmodelbench/direct-import \
  --json

Metrics

WorldModelBench raw category maxima are 3, 2, and 5 points. WorldFoundry reports both raw and normalized values when possible.

Outputs

The runner writes these files under --output-dir:

• scorecard.json: normalized scorecard with metric values and generated-video coverage.
• raw_metric_table.jsonl: one row per metric ID.
• judge_responses.jsonl: flattened VILA judge responses when available.
• upstream/worldmodelbench_results.json: generated by the direct judge path.
• upstream_stdout.log and upstream_stderr.log: direct judge execution logs.

Public benchmark-zoo runs can also write runner_runtime_report.json.

Limitations

The official answers and explanations are withheld, so local runs depend on the released prompts, generated videos, and the judge checkpoint. WorldFoundry can verify manifest coverage and result shape, but leaderboard evidence still requires the upstream submission process. In this revision, use the direct runner for judge execution; the public benchmark-zoo import path is supported, while the public judge-execution path does not pass all runner arguments through the catalog entry.

← Benchmark Hub