VBench-2.0

Integrated

Intrinsic-faithfulness video generation evaluation with in-tree VBench-2.0 runtime, prompt assets, datasets, and commands.

Markdown Source

Last updated: Jul 11, 2026

On this page

About Evaluation Protocol Data Preparation Checkpoints And Runtime Run A Single Dimension Run Multiple Dimensions Import Existing Results Outputs

About

VBench-2.0 extends the VBench series from basic technical quality toward intrinsic faithfulness. It targets next-generation video generators that must reason about people, physics, controllability, creativity, and commonsense, not just produce sharp short clips.

WorldFoundry integrates the VBench-2.0 runtime in tree at worldfoundry/evaluation/tasks/execution/runners/vbench_2_0. The official VBench repository is a protocol reference; the runnable metric code used here lives in WorldFoundry.

Official references:

Project page: vchitect.github.io/VBench-2.0-project
Paper: arXiv:2503.21755
Arena: Vchitect/VBench2.0_Video_Arena
Sample videos: Vchitect/VBench-2.0_sampled_videos
Human annotation: Vchitect/VBench-2.0_human_annotation
In-tree runner: worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py

Evaluation Protocol

VBench-2.0 has 18 fine-grained dimensions grouped into five categories.

Group	Dimensions
Creativity	`composition`, `diversity`
Commonsense	`instance_preservation`, `motion_rationality`
Controllability	`camera_motion`, `complex_landscape`, `complex_plot`, `dynamic_attribute`, `dynamic_spatial_relationship`, `human_interaction`, `motion_order_understanding`
Human fidelity	`human_anatomy`, `human_clothes`, `human_identity`
Physics	`material`, `mechanics`, `multi_view_consistency`, `thermotics`

WorldFoundry aggregates these into vbench2_creativity, vbench2_commonsense, vbench2_controllability, vbench2_human_fidelity, vbench2_physics, and the primary metric vbench2_total.

The prompt assets are checked in:

worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json
worldfoundry/data/benchmarks/assets/vbench-2.0/vbench2_prompts/prompt/*.txt
worldfoundry/data/benchmarks/assets/vbench-2.0/vbench2_prompts/meta_info/*.json

Data Preparation

For leaderboard-style reproduction, generate videos for the official VBench-2.0 prompts and keep the same dimension split. Put all generated videos under one root and pass it with --videos-path.

For local custom-input evaluation, you can score a folder or video file with a prompt:

/path/to/vbench2/generated_videos/
  sample_0001.mp4
  sample_0002.mp4

Set the artifact root:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/vbench2/generated_videos

The runner also discovers optional official datasets. Download them only when you need dataset coverage, human annotation metadata, or reference sampled videos:

export WORLDFOUNDRY_VBENCH2_DATASET_ROOT=/path/to/datasets/vbench2

hf download Vchitect/VBench-2.0_sampled_videos \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}/VBench-2.0_sampled_videos"

hf download Vchitect/VBench-2.0_human_annotation \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}/VBench-2.0_human_annotation"

Some human-fidelity dimensions also use the VBench-2.0 human anomaly assets:

hf download Vchitect/VBench-2.0_human_anomaly \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}/VBench-2.0_human_anomaly"

Candidate model training is model-specific. Train or run the generator through its own package, then export videos into WORLDFOUNDRY_GENERATED_ARTIFACT_DIR.

Checkpoints And Runtime

Use the unified CUDA environment. The VBench-2.0 evaluator needs the VBench perception stack plus additional judge/perception assets for LLaVA-Video, Qwen, YOLO-World, ArcFace, anomaly detection, CoTracker, RAFT, CLIP/DINO, GroundingDINO, and SAM-family segmentation. Reusable code for many of these lives under worldfoundry/base_models.

Useful cache variables:

export WORLDFOUNDRY_VBENCH_CACHE_DIR=/path/to/cache/models/vbench
export WORLDFOUNDRY_VBENCH2_CACHE_DIR=/path/to/cache/models/vbench2
export WORLDFOUNDRY_VBENCH_RETINAFACE_CKPT=/path/to/retinaface.pth

human_identity, gender_bias, and skin_bias style face-related metrics require RetinaFace-compatible assets when those dimensions are selected.

Run A Single Dimension

Use this command to score generated videos with one VBench-2.0 dimension:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \
  --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --vbench2-dataset-root "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}" \
  --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json" \
  --dimension Diversity \
  --mode custom_input \
  --output-dir tmp/vbench-2.0/diversity \
  --timeout 7200 \
  --json

For a custom prompt shared by all videos:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \
  --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json" \
  --dimension Mechanics \
  --mode custom_input \
  --prompt "a glass ball rolls down a wooden ramp and collides with a metal block" \
  --output-dir tmp/vbench-2.0/mechanics_custom \
  --json

Run Multiple Dimensions

Run the five category groups only after all corresponding metric assets are staged:

cd /path/to/WorldFoundry

for DIMENSION in \
  Composition Diversity Instance_Preservation Motion_Rationality \
  Camera_Motion Complex_Landscape Complex_Plot Dynamic_Attribute \
  Dynamic_Spatial_Relationship Human_Interaction Motion_Order_Understanding \
  Human_Anatomy Human_Clothes Human_Identity \
  Material Mechanics Multi-View_Consistency Thermotics
do
  PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
    worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \
    --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
    --vbench2-dataset-root "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}" \
    --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/vbench-2.0/VBench2_full_info.json" \
    --dimension "${DIMENSION}" \
    --mode custom_input \
    --output-dir "tmp/vbench-2.0/${DIMENSION}" \
    --json
done

Import Existing Results

Import an official-compatible VBench-2.0 *_eval_results.json:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vbench_2_0/run_vbench_2_0_official_runner.py \
  --official-results-path /path/to/vbench2_eval_results.json \
  --videos-path "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --vbench2-dataset-root "${WORLDFOUNDRY_VBENCH2_DATASET_ROOT}" \
  --output-dir tmp/vbench-2.0/imported \
  --json

Outputs

Each run writes:

scorecard.json: WorldFoundry scorecard with dimension scores and VBench-2.0 aggregates.
raw_metric_table.jsonl: metric rows used by the scorecard.
dimension_scores.json: normalized dimension-score summary.
vbench2_dataset_manifest.json: discovered official dataset metadata when available.
vbench2_video_coverage.json: generated-video coverage against discovered references.
upstream_stdout.log and upstream_stderr.log: logs from the in-tree runtime.

Back to Benchmark Hub

VBench-2.0

Related