# WorldScore (/docs/evaluation/benchmark-hub/worldscore) ## About [#about] WorldScore evaluates world generation: a model starts from an observed scene and must extend it while preserving camera, object, visual, and motion constraints. The public benchmark covers 3,000 examples across 3D generation, 4D generation, image-to-video, and text-to-video systems. WorldFoundry already carries the benchmark runner and runtime under `worldfoundry/evaluation/tasks/execution/runners/worldscore`. Do not clone the official repos for a WorldFoundry run. Downloading the Hugging Face dataset and metric checkpoints is still expected. ## What It Measures [#what-it-measures] WorldScore is the first unified benchmark for **world generation**. It decomposes world building into a sequence of next-scene tasks with explicit camera-trajectory layout specs, then scores controllability, quality, and dynamics across 3D, 4D, I2V, and T2V systems. ## Benchmark Design [#benchmark-design] | Property | Detail | | ------------- | -------------------------------------------------------------------------------------------- | | Examples | 3,000 curated test cases spanning static and dynamic worlds | | Modalities | 3D generation, 4D generation, image-to-video, text-to-video | | Task form | Next-scene generation from an observed scene under camera or layout control | | Static split | Controllability + quality (camera control, object control, 3D/photometric/style consistency) | | Dynamic split | Adds motion accuracy, motion magnitude, and motion smoothness | | Aggregates | `WorldScore-Static` and `WorldScore-Dynamic` leaderboard variants | Unlike single-scene video benchmarks, WorldScore requires multi-scene continuity, long sequences, image conditioning, multi-style coverage, camera control, and 3D consistency in one protocol. ## Official References [#official-references] * Paper: [arXiv:2504.00983](https://arxiv.org/abs/2504.00983) * Project page and leaderboard: [haoyi-duan.github.io/WorldScore](https://haoyi-duan.github.io/WorldScore/) * Official source reference: [github.com/haoyi-duan/WorldScore](https://github.com/haoyi-duan/WorldScore) * Dataset: [Howieeeee/WorldScore](https://huggingface.co/datasets/Howieeeee/WorldScore) * In-tree runner: `worldfoundry/evaluation/tasks/execution/runners/worldscore/run_worldscore_official_runner.py` ## Leaderboard Notes [#leaderboard-notes] The official site publishes `WorldScore-Static` and `WorldScore-Dynamic` rankings with per-family sub-metrics (camera control, object control, content alignment, 3D/photometric/style consistency, subjective quality, motion accuracy/magnitude/smoothness). WorldFoundry records bounded GPU validation for one dynamic sample; full 3,000-example leaderboard parity requires complete `evaluation.json` trees and the full metric checkpoint stack. ## Prepare Data And Assets [#prepare-data-and-assets] Run from the WorldFoundry repository root: ```bash cd /path/to/WorldFoundry export PYTHONPATH="$PWD:${PYTHONPATH:-}" ``` Download or stage the dataset so the data root contains `WorldScore-Dataset/`: ```bash export WORLDFOUNDRY_WORLDSCORE_DATA_PATH=/path/to/Howieeeee__WorldScore hf download Howieeeee/WorldScore \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_WORLDSCORE_DATA_PATH}" ``` Prepare metric assets. WorldFoundry models these as reusable base-model dependencies, but the files still need to exist locally for full metric execution: * `WORLDFOUNDRY_WORLDSCORE_CONFIG_ROOT`: optional config override; defaults to `worldfoundry/data/benchmarks/assets/worldscore/config`. * `WORLDFOUNDRY_WORLDSCORE_ASSET_CHECKPOINT_DIR`: shared WorldScore metric checkpoint directory. * `WORLDFOUNDRY_DROID_SLAM_CKPT`: DROID-SLAM checkpoint, commonly `droid.pth`. * `WORLDFOUNDRY_GROUNDING_DINO_CKPT`: GroundingDINO Swin-T checkpoint. * `WORLDFOUNDRY_SAM_VIT_H_CKPT`: SAM ViT-H checkpoint. * `WORLDFOUNDRY_SAM2_CKPT`: SAM2.1 checkpoint. * `WORLDFOUNDRY_RAFT_THINGS_CKPT`: RAFT Things checkpoint. * `WORLDFOUNDRY_SEA_RAFT_CKPT`: SEA-RAFT checkpoint. * `WORLDFOUNDRY_FLOWFORMERPLUSPLUS_CKPT`: FlowFormer++ checkpoint. * `WORLDFOUNDRY_VFIMAMBA_CKPT`: VFIMamba checkpoint. Prepare candidate outputs in one of two shapes: ```text complete run: //worldscore_output/ static/.../input_image.png static/.../frames/000.png static/.../camera_data.json static/.../image_data.json static/.../evaluation.json dynamic/.../input_image.png dynamic/.../frames/000.png dynamic/.../videos/output.mp4 dynamic/.../image_data.json dynamic/.../evaluation.json worldscore.json bounded run: / candidate.mp4 ``` The bounded path is useful for local integration runs. It stages one generated video or frame directory into the dynamic WorldScore layout and is not leaderboard evidence. ```bash export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/worldscore_artifacts export WORLDFOUNDRY_WORLDSCORE_MODEL_NAME=wan2.1_i2v export WORLDFOUNDRY_WORLDSCORE_MODEL_PATH=/path/to/model_workspace export WORLDFOUNDRY_WORLDSCORE_HF_DATASET_ROOT="${WORLDFOUNDRY_WORLDSCORE_DATA_PATH}" ``` ## Run With WorldFoundry [#run-with-worldfoundry] Public benchmark command for the in-tree runtime: ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id worldscore \ --mode official-run \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --env WORLDFOUNDRY_WORLDSCORE_HF_DATASET_ROOT="${WORLDFOUNDRY_WORLDSCORE_HF_DATASET_ROOT}" \ --env WORLDFOUNDRY_WORLDSCORE_ASSET_CHECKPOINT_DIR="${WORLDFOUNDRY_WORLDSCORE_ASSET_CHECKPOINT_DIR}" \ --env WORLDFOUNDRY_WORLDSCORE_MODEL_NAME="${WORLDFOUNDRY_WORLDSCORE_MODEL_NAME}" \ --env WORLDFOUNDRY_WORLDSCORE_MODEL_PATH="${WORLDFOUNDRY_WORLDSCORE_MODEL_PATH}" \ --output-dir tmp/worldscore/official-run \ --json ``` Public command for importing an existing `worldscore.json`: ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id worldscore \ --mode official-validation \ --official-results-path /path/to/worldscore_output/worldscore.json \ --benchmark-data-root "${WORLDFOUNDRY_WORLDSCORE_DATA_PATH}" \ --generated-artifact-dir /path/to/worldscore_output \ --output-dir tmp/worldscore/official-validation \ --json ``` Direct runner command for staging and running from the bundled runtime: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/worldscore/run_worldscore_official_runner.py \ --worldscore-root worldfoundry/evaluation/tasks/execution/runners/worldscore/runtime/worldscore \ --worldscore-config-root worldfoundry/data/benchmarks/assets/worldscore/config \ --model-name "${WORLDFOUNDRY_WORLDSCORE_MODEL_NAME}" \ --model-path "${WORLDFOUNDRY_WORLDSCORE_MODEL_PATH}" \ --data-path "${WORLDFOUNDRY_WORLDSCORE_DATA_PATH}" \ --stage-dynamic-source "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --stage-target-frames 8 \ --stage-overwrite \ --output-dir tmp/worldscore/direct-runner \ --json ``` Direct runner command for importing a completed result file: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/worldscore/run_worldscore_official_runner.py \ --data-path "${WORLDFOUNDRY_WORLDSCORE_DATA_PATH}" \ --generated-root /path/to/worldscore_output \ --official-results-path /path/to/worldscore_output/worldscore.json \ --output-dir tmp/worldscore/direct-import \ --json ``` ## Metrics [#metrics] | Metric ID | Meaning | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `controllability` | Mean of camera-control and object-control adherence. Higher is better. | | `quality` | Mean visual/reconstruction quality family: content alignment, 3D consistency, photometric consistency, style consistency, and subjective quality when present. Higher is better. | | `dynamics` | Mean motion family: motion accuracy, motion magnitude, and motion smoothness. Higher is better. | | `worldscore_average` | Primary aggregate across available WorldScore metric families. WorldFoundry normalizes percent-like values to a unit score in the scorecard. | ## Outputs [#outputs] WorldFoundry writes these files under `--output-dir`: * `scorecard.json`: normalized benchmark scorecard consumed by reports and comparisons. * `per_sample_metrics.jsonl`: one row per discovered sample or imported sample metric row. * `raw_metric_table.jsonl`: metric rows with availability, raw score, normalized score, and source. * `official_stdout.log` and `official_stderr.log` for direct official-runtime execution. ## Limitations [#limitations] WorldFoundry has recorded bounded GPU validation for one dynamic sample, not full 3,000-example leaderboard parity. Full runs need complete `evaluation.json` trees, the WorldScore metric checkpoint stack, CUDA-capable dependencies, and enough GPU time to execute DROID-SLAM, segmentation, optical-flow, and frame-interpolation metrics. The runner evaluates already-generated outputs; it does not adapt 3D, 4D, or video models for generation. [← Benchmark Hub](/docs/evaluation/benchmark-hub)