# WBench (/docs/evaluation/benchmark-hub/wbench) ## About [#about] WBench is a multi-turn interactive video world-model benchmark. It evaluates whether a model can keep rendering quality, scene setting, user control, temporal consistency, and physical plausibility across repeated interactions. The public benchmark covers 289 cases and 1,058 interaction turns, with four interaction types: navigation, subject action, event editing, and perspective switching. The benchmark is designed for multiple control interfaces. Text-conditioned models run all 289 cases. Camera-conditioned and action-conditioned models run the 158 navigation cases, where WBench maps text, 6-DoF camera pose, and discrete actions into a comparable navigation protocol. WorldFoundry integrates the WBench runtime in tree. The official GitHub repository, paper, project page, Hugging Face data, and Hugging Face weights are protocol and asset references; the WorldFoundry workflow does not require an external benchmark source checkout. ## What It Measures [#what-it-measures] WBench is a **multi-turn interactive** video world-model benchmark. It evaluates whether a model can keep rendering quality, scene setting, user control, temporal consistency, and physical plausibility across repeated interactions — 289 cases, 1,058 turns, and 22 human-validated metrics. ## Sources [#sources] * Paper: [WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation](https://arxiv.org/abs/2605.25874) * Project page and leaderboard: [meituan-longcat.github.io/WBench](https://meituan-longcat.github.io/WBench/) * Official source reference: [github.com/meituan-longcat/WBench](https://github.com/meituan-longcat/WBench) * Dataset: [meituan-longcat/WBench](https://huggingface.co/datasets/meituan-longcat/WBench) * Example submissions: [meituan-longcat/WBench-examples](https://huggingface.co/datasets/meituan-longcat/WBench-examples) * Metric weights: [meituan-longcat/WBench-weights](https://huggingface.co/meituan-longcat/WBench-weights) ## Evaluation Protocol [#evaluation-protocol] Generated videos use the official WBench work directory layout: ```text work_dirs// videos/ case__combined.mp4 evaluation/ / case_.json report.json ``` `` is the case JSON id, not the case filename. A full multi-turn clip is stored as one `case__combined.mp4` file. If a model emits turns with unequal frame counts, prepare `turns.json` for leaderboard submission so per-turn metrics can split the combined video correctly. | Dimension | Metrics | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Quality | `aesthetic_quality`, `imaging_quality`, `temporal_flickering`, `dynamic_degree`, `motion_smoothness`, `hpsv3_quality` | | Setting | `scene_adherence`, `subject_adherence` | | Interaction | `navigation_trajectory`, `event_edit_adherence`, `subject_action_adherence`, `perspective_switch_adherence` | | Consistency | `background_consistency`, `segment_continuity`, `perspective_consistency`, `subject_consistency`, `geometric_consistency`, `photometric_consistency`, `spatial_consistency`, `gated_spatial_consistency` | | Physical | `visual_plausibility`, `causal_fidelity` | | Primary | `wbench_average`, computed from the five dimension scores when available | ## WorldFoundry Integration [#worldfoundry-integration] WorldFoundry's integrated runtime lives at: ```text worldfoundry/evaluation/tasks/execution/runners/wbench/runtime/wbench ``` The runner entry point is: ```text worldfoundry/evaluation/tasks/execution/runners/wbench/run_wbench_official_runner.py ``` The direct runner can execute the checked-in WBench runtime with `--run-official`, or it can ingest an existing `report.json` or per-metric `evaluation/` directory and write a WorldFoundry `scorecard.json`. The runtime uses WorldFoundry base-model assets for shared perception components where available, and WBench-specific weights are supplied through `WBENCH_WEIGHTS_DIR`. WBench has both local metric dependencies and API-backed VLM metrics. For full metric recomputation you need generated videos, WBench data cases and masks, metric weights, CUDA packages, and VLM credentials for the official VLM metrics. ## Prepare Data And Assets [#prepare-data-and-assets] Start from the WorldFoundry repository root: ```bash cd /path/to/WorldFoundry export WORLDFOUNDRY_REPO_ROOT="$PWD" export PYTHONPATH="$WORLDFOUNDRY_REPO_ROOT:${PYTHONPATH:-}" ``` Prepare the WBench runtime, data, work directory, and metric weights: ```bash export WORLDFOUNDRY_WBENCH_ROOT="$WORLDFOUNDRY_REPO_ROOT/worldfoundry/evaluation/tasks/execution/runners/wbench/runtime/wbench" export WORLDFOUNDRY_WBENCH_WORK_DIR=/path/to/wbench/work_dirs export WORLDFOUNDRY_WBENCH_MODEL_NAME=my_model export WBENCH_WEIGHTS_DIR=/path/to/wbench/weights hf download meituan-longcat/WBench \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_WBENCH_ROOT}/data" \ --exclude "splits/*" hf download meituan-longcat/WBench-weights \ --local-dir "${WBENCH_WEIGHTS_DIR}" ``` Place generated videos under: ```text ${WORLDFOUNDRY_WBENCH_WORK_DIR}/${WORLDFOUNDRY_WBENCH_MODEL_NAME}/videos/ ``` For a text-conditioned model, cover all 289 cases. For a camera-conditioned or action-conditioned model, cover the 158 navigation cases. Set `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR` to the same video directory so WorldFoundry can record coverage metadata: ```bash export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR="${WORLDFOUNDRY_WBENCH_WORK_DIR}/${WORLDFOUNDRY_WBENCH_MODEL_NAME}/videos" ``` For VLM metrics: ```bash export VLM_API_KEY= export VLM_API_URL=${VLM_API_URL:-https://ark.cn-beijing.volces.com/api/v3} export VLM_MODEL_NAME=${VLM_MODEL_NAME:-doubao-seed-2-0-lite-260215} ``` For local visual plausibility scoring, provide the PAVRM model path: ```bash export WORLDFOUNDRY_WBENCH_PAVRM_MODEL_DIR=/path/to/pavrm_model ``` ## Run Evaluation [#run-evaluation] Run the full in-tree WBench metric pipeline: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/wbench/run_wbench_official_runner.py \ --run-official \ --wbench-root "${WORLDFOUNDRY_WBENCH_ROOT}" \ --work-dir "${WORLDFOUNDRY_WBENCH_WORK_DIR}" \ --weights-dir "${WBENCH_WEIGHTS_DIR}" \ --model-name "${WORLDFOUNDRY_WBENCH_MODEL_NAME}" \ --phase all \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/wbench/official-run \ --json ``` For a staged run, the official runtime also supports phases: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/wbench/run_wbench_official_runner.py \ --run-official \ --wbench-root "${WORLDFOUNDRY_WBENCH_ROOT}" \ --work-dir "${WORLDFOUNDRY_WBENCH_WORK_DIR}" \ --model-name "${WORLDFOUNDRY_WBENCH_MODEL_NAME}" \ --phase report \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/wbench/report-only \ --json ``` If you already have `report.json` or an official `evaluation/` directory, import it through the public benchmark entry point: ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id wbench \ --mode official-validation \ --official-results-path "${WORLDFOUNDRY_WBENCH_WORK_DIR}/${WORLDFOUNDRY_WBENCH_MODEL_NAME}/evaluation/report.json" \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/wbench/official-validation \ --json ``` ## Outputs [#outputs] WorldFoundry writes: * `scorecard.json` * `raw_metric_table.jsonl` * `report.json` when the official runtime produced or imported it * WBench runtime logs under the selected `--output-dir` For leaderboard submission, the official package is: ```text / meta.json report.json turns.json videos/ case__combined.mp4 ``` The official WBench process accepts either self-evaluated scores plus videos, or videos only for the WBench team to evaluate. WorldFoundry does not run or host your model for that process; it prepares local scorecards and artifacts from the assets you provide. [← Benchmark Hub](/docs/evaluation/benchmark-hub)