# VBench-2.0 (/docs/evaluation/benchmark-hub/vbench-2.0) ## About [#about] VBench-2.0 extends the VBench series from basic technical quality toward intrinsic faithfulness. It targets next-generation video generators that must reason about people, physics, controllability, creativity, and commonsense, not just produce sharp short clips. WorldFoundry integrates the VBench-2.0 runtime in tree at `worldfoundry/evaluation/tasks/execution/runners/vbench_2_0`. The official VBench repository is a protocol reference; the runnable metric code used here lives in WorldFoundry. Official references: * Project page: [vchitect.github.io/VBench-2.0-project](https://vchitect.github.io/VBench-2.0-project/) * Paper: [arXiv:2503.21755](https://arxiv.org/abs/2503.21755) * Arena: [Vchitect/VBench2.0\_Video\_Arena](https://huggingface.co/spaces/Vchitect/VBench2.0_Video_Arena) * Sample videos: [Vchitect/VBench-2.0\_sampled\_videos](https://huggingface.co/datasets/Vchitect/VBench-2.0_sampled_videos) * Human annotation: [Vchitect/VBench-2.0\_human\_annotation](https://huggingface.co/datasets/Vchitect/VBench-2.0_human_annotation) * In-tree runner: `worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py` ## Evaluation Protocol [#evaluation-protocol] VBench-2.0 has 18 fine-grained dimensions grouped into five categories. | Group | Dimensions | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Creativity | `composition`, `diversity` | | Commonsense | `instance_preservation`, `motion_rationality` | | Controllability | `camera_motion`, `complex_landscape`, `complex_plot`, `dynamic_attribute`, `dynamic_spatial_relationship`, `human_interaction`, `motion_order_understanding` | | Human fidelity | `human_anatomy`, `human_clothes`, `human_identity` | | Physics | `material`, `mechanics`, `multi_view_consistency`, `thermotics` | WorldFoundry aggregates these into `vbench2_creativity`, `vbench2_commonsense`, `vbench2_controllability`, `vbench2_human_fidelity`, `vbench2_physics`, and the primary metric `vbench2_total`. The prompt assets are checked in: * `worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json` * `worldfoundry/data/benchmarks/assets/vbench-2.0/vbench2_prompts/prompt/*.txt` * `worldfoundry/data/benchmarks/assets/vbench-2.0/vbench2_prompts/meta_info/*.json` ## Data Preparation [#data-preparation] For leaderboard-style reproduction, generate videos for the official VBench-2.0 prompts and keep the same dimension split. Put all generated videos under one root and pass it with `--videos-path`. For local custom-input evaluation, you can score a folder or video file with a prompt: ```text /path/to/vbench2/generated_videos/ sample_0001.mp4 sample_0002.mp4 ``` Set the artifact root: ```bash export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/vbench2/generated_videos ``` The runner also discovers optional official datasets. Download them only when you need dataset coverage, human annotation metadata, or reference sampled videos: ```bash export WORLDFOUNDRY_VBENCH2_DATASET_ROOT=/path/to/datasets/vbench2 hf download Vchitect/VBench-2.0_sampled_videos \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}/VBench-2.0_sampled_videos" hf download Vchitect/VBench-2.0_human_annotation \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}/VBench-2.0_human_annotation" ``` Some human-fidelity dimensions also use the VBench-2.0 human anomaly assets: ```bash hf download Vchitect/VBench-2.0_human_anomaly \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}/VBench-2.0_human_anomaly" ``` Candidate model training is model-specific. Train or run the generator through its own package, then export videos into `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR`. ## Checkpoints And Runtime [#checkpoints-and-runtime] Use the unified CUDA environment. The VBench-2.0 evaluator needs the VBench perception stack plus additional judge/perception assets for LLaVA-Video, Qwen, YOLO-World, ArcFace, anomaly detection, CoTracker, RAFT, CLIP/DINO, GroundingDINO, and SAM-family segmentation. Reusable code for many of these lives under `worldfoundry/base_models`. Useful cache variables: ```bash export WORLDFOUNDRY_VBENCH_CACHE_DIR=/path/to/cache/models/vbench export WORLDFOUNDRY_VBENCH2_CACHE_DIR=/path/to/cache/models/vbench2 export WORLDFOUNDRY_VBENCH_RETINAFACE_CKPT=/path/to/retinaface.pth ``` `human_identity`, `gender_bias`, and `skin_bias` style face-related metrics require RetinaFace-compatible assets when those dimensions are selected. ## Run A Single Dimension [#run-a-single-dimension] Use this command to score generated videos with one VBench-2.0 dimension: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \ --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --vbench2-dataset-root "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}" \ --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json" \ --dimension Diversity \ --mode custom_input \ --output-dir tmp/vbench-2.0/diversity \ --timeout 7200 \ --json ``` For a custom prompt shared by all videos: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \ --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json" \ --dimension Mechanics \ --mode custom_input \ --prompt "a glass ball rolls down a wooden ramp and collides with a metal block" \ --output-dir tmp/vbench-2.0/mechanics_custom \ --json ``` ## Run Multiple Dimensions [#run-multiple-dimensions] Run the five category groups only after all corresponding metric assets are staged: ```bash cd /path/to/WorldFoundry for DIMENSION in \ Composition Diversity Instance_Preservation Motion_Rationality \ Camera_Motion Complex_Landscape Complex_Plot Dynamic_Attribute \ Dynamic_Spatial_Relationship Human_Interaction Motion_Order_Understanding \ Human_Anatomy Human_Clothes Human_Identity \ Material Mechanics Multi-View_Consistency Thermotics do PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \ --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --vbench2-dataset-root "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}" \ --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json" \ --dimension "${DIMENSION}" \ --mode custom_input \ --output-dir "tmp/vbench-2.0/${DIMENSION}" \ --json done ``` ## Import Existing Results [#import-existing-results] Import an official-compatible VBench-2.0 `*_eval_results.json`: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \ --official-results-path /path/to/vbench2_eval_results.json \ --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --vbench2-dataset-root "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}" \ --output-dir tmp/vbench-2.0/imported \ --json ``` ## Outputs [#outputs] Each run writes: * `scorecard.json`: WorldFoundry scorecard with dimension scores and VBench-2.0 aggregates. * `raw_metric_table.jsonl`: metric rows used by the scorecard. * `dimension_scores.json`: normalized dimension-score summary. * `vbench2_dataset_manifest.json`: discovered official dataset metadata when available. * `vbench2_video_coverage.json`: generated-video coverage against discovered references. * `upstream_stdout.log` and `upstream_stderr.log`: logs from the in-tree runtime. [Back to Benchmark Hub](/docs/evaluation/benchmark-hub)