WorldBench

About

WorldBench is represented in WorldFoundry as a physical-understanding benchmark with video-based and text-based result components. The cataloged public dataset is worldbenchmark/WorldBench, which redirects to worldbenchmark/IntuitivePhysics in the recorded Hugging Face metadata. The catalog describes 425 video-based scenes with RGB, normal, depth, flow, and segmentation assets, plus 181 text-based videos with multiple-choice and binary questions.

WorldFoundry already includes the WorldBench runner under worldfoundry/evaluation/tasks/execution/runners/worldbench. Do not clone official repos for this page. The current in-tree runner imports caller-provided JSON/JSONL/CSV/TSV results or materializes scores from a WorldFoundry artifact-score directory; no source-confirmed upstream evaluator was found in the local GitHub/Hugging Face references.

What It Measures

WorldBench asks how close generative world models are to the physical world. It evaluates video continuation from a short input clip and text-based physics reasoning, measuring whether models preserve object permanence, support relations, motion physics, and scale/perspective over time.

Benchmark Design

The official project uses Kubric (PyBullet + Blender) for physically accurate ground truth. Performance degrades with prediction horizon — the sharpest drop occurs after the first predicted frames.

Official References

• Paper: arXiv:2601.21282
• Project page: world-bench.github.io
• Dataset: worldbenchmark/WorldBench (redirects to worldbenchmark/IntuitivePhysics)
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/worldbench/run_worldbench_official_runner.py

Leaderboard Notes

The official site reports per-category mIoU for world foundation models and per-category accuracy for vision-language models. WorldFoundry does not currently ship a source-confirmed upstream evaluator; it imports caller-provided result tables or materializes scores from artifact-score directories. Leaderboard parity remains pending until the complete upstream scoring workflow is audited.

Prepare Data And Assets

Run from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export PYTHONPATH="$PWD:${PYTHONPATH:-}"

Download or stage the public dataset if you need local prompts, videos, or labels:

export WORLDFOUNDRY_WORLDBENCH_DATA_ROOT=/path/to/worldbenchmark__WorldBench
hf download worldbenchmark/WorldBench \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_WORLDBENCH_DATA_ROOT}"

Prepare one of these scoring inputs:

existing result file:
  results.json
  results.jsonl
  results.csv
  results.tsv

artifact-score directory:
  <artifact_score_dir>/
    one or more WorldFoundry metric artifacts that can be aggregated into
    video_based_accuracy, text_based_accuracy, multiple_choice_accuracy,
    binary_accuracy, or worldbench_average

The simplest supported result table is:

metric_id,score
video_based_accuracy,0.75
text_based_accuracy,0.75
multiple_choice_accuracy,0.75
binary_accuracy,0.75
worldbench_average,0.75

Per-sample rows are also accepted when they contain recognizable fields such as sample_id, component, question_type, prediction, answer, correct, or score.

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/worldbench/generated_videos
export WORLDFOUNDRY_WORLDBENCH_RESULTS_PATH=/path/to/worldbench/results.csv
export WORLDFOUNDRY_WORLDBENCH_ARTIFACT_SCORE_DIR=/path/to/worldbench/artifact_scores

Run With WorldFoundry

Public command for importing a completed WorldBench result file:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id worldbench \
  --mode official-validation \
  --official-results-path "${WORLDFOUNDRY_WORLDBENCH_RESULTS_PATH}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/worldbench/official-validation \
  --json

There is no public benchmark-zoo official-run command wired for WorldBench in this catalog entry. Use the direct runner when you want the in-tree artifact-score path.

Direct runner command for importing results:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/worldbench/run_worldbench_official_runner.py \
  --official-results-path "${WORLDFOUNDRY_WORLDBENCH_RESULTS_PATH}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/worldbench/direct-import \
  --json

Direct runner command for materializing scores from a WorldFoundry artifact-score directory:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/worldbench/run_worldbench_official_runner.py \
  --run-official \
  --artifact-score-dir "${WORLDFOUNDRY_WORLDBENCH_ARTIFACT_SCORE_DIR}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/worldbench/direct-artifact-scores \
  --json

The artifact-score command writes an intermediate result file at:

tmp/worldbench/direct-artifact-scores/upstream/worldbench_results.json

Metrics

Outputs

WorldFoundry writes these files under --output-dir:

• scorecard.json: normalized WorldBench scorecard.
• per_sample_scores.jsonl: normalized sample rows when present in the input.
• raw_metric_table.jsonl: metric rows with raw and normalized scores.
• upstream/worldbench_results.json: only when --run-official materializes artifact scores.