WorldModelBench

About

WorldModelBench asks whether generated videos behave like world models. It contains 350 public test instances across 7 domains and 56 subdomains. Each instance provides a first-frame description, an instruction, and a first-frame image. Text-to-video models use the description plus instruction; image-to-video models use the first frame plus instruction.

WorldFoundry already includes the WorldModelBench evaluator wrapper and runtime under worldfoundry/evaluation/tasks/execution/runners/worldmodelbench. Do not clone the official repos for a WorldFoundry run. You still need the data assets and the VILA judge checkpoint.

What It Measures

WorldModelBench judges whether video generators behave as world models — not only whether clips look good, but whether they follow instructions, preserve temporal common sense, and respect physical laws. The benchmark crowdsourced 67K human labels to train a human-aligned VILA judge that automates evaluation.

Benchmark Design

Physical-adherence checks target subtle violations such as size drift, unnatural deformation, impossible fluid flow, object penetration, and gravity inconsistency — failures that generic video-quality metrics often miss.

Official References

• Paper: arXiv:2502.20694
• Project page and leaderboard: worldmodelbench-team.github.io
• Official source reference: github.com/WorldModelBench-Team/WorldModelBench
• Test data: Efficient-Large-Model/worldmodelbench
• Judge model: Efficient-Large-Model/vila-ewm-qwen2-1.5b
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/run_worldmodelbench_official_runner.py

Leaderboard Notes

The public leaderboard reports Instruction Following, Physics Adherence, Common Sense, and a Total Score. Official answers and explanations are withheld for test integrity; submit results to worldmodelbench.team@gmail.com per the project page. WorldFoundry normalizes local judge output into scorecard.json but does not host the upstream submission workflow.

Prepare Data And Assets

Run from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export PYTHONPATH="$PWD:${PYTHONPATH:-}"

Create a data root with the manifest, first-frame images, and evaluator file layout expected by the runner:

<worldmodelbench_data_root>/
  worldmodelbench.json
  images/
  evaluation.py

WorldFoundry ships the public 350-instance manifest:

export WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT=/path/to/worldmodelbench
mkdir -p "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}"
cp worldfoundry/data/benchmarks/assets/worldmodelbench/worldmodelbench.json \
  "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}/worldmodelbench.json"

If you use a Hugging Face mirror for the data assets, place it in the same root:

hf download Efficient-Large-Model/worldmodelbench \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}"

Download the judge checkpoint:

export WORLDFOUNDRY_WORLDMODELBENCH_JUDGE=/path/to/vila-ewm-qwen2-1.5b
hf download Efficient-Large-Model/vila-ewm-qwen2-1.5b \
  --local-dir "${WORLDFOUNDRY_WORLDMODELBENCH_JUDGE}"

Generate one video per manifest row. The output filename must match the first-frame stem:

images/69620089860948e38a4921dd4869d24f.jpg
=> <generated_videos>/69620089860948e38a4921dd4869d24f.mp4

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/worldmodelbench/generated_videos

Prompt construction follows the local README:

# text-to-video
" ".join([instance["text_first_frame"], instance["text_instruction"]])

# image-to-video
image = instance["first_frame"]
text = instance["text_instruction"]

Run With WorldFoundry

Public command for importing a completed worldmodelbench_results.json:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id worldmodelbench \
  --mode official-validation \
  --official-results-path /path/to/worldmodelbench_results.json \
  --benchmark-data-root "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/worldmodelbench/official-validation \
  --json

Direct runner command for judge execution through the bundled runtime:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/run_worldmodelbench_official_runner.py \
  --worldmodelbench-root worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/runtime \
  --data-root "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}" \
  --video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --judge "${WORLDFOUNDRY_WORLDMODELBENCH_JUDGE}" \
  --model-name "${WORLDFOUNDRY_MODEL_NAME:-candidate_model}" \
  --output-dir tmp/worldmodelbench/direct-runner \
  --json

Direct runner command for importing a completed result file:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/worldmodelbench/run_worldmodelbench_official_runner.py \
  --data-root "${WORLDFOUNDRY_WORLDMODELBENCH_DATA_ROOT}" \
  --video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --official-results-path /path/to/worldmodelbench_results.json \
  --output-dir tmp/worldmodelbench/direct-import \
  --json

Metrics

WorldModelBench raw category maxima are 3, 2, and 5 points. WorldFoundry reports both raw and normalized values when possible.

Outputs

The runner writes these files under --output-dir:

• scorecard.json: normalized scorecard with metric values and generated-video coverage.
• raw_metric_table.jsonl: one row per metric ID.
• judge_responses.jsonl: flattened VILA judge responses when available.
• upstream/worldmodelbench_results.json: generated by the direct judge path.
• upstream_stdout.log and upstream_stderr.log: direct judge execution logs.

Public benchmark-zoo runs can also write runner_runtime_report.json.