# Video-Bench (/docs/evaluation/benchmark-hub/video-bench) ## About [#about] Video-Bench is a human-preference-aligned benchmark for generated video quality. It uses MLLM judges with few-shot scoring and chain-of-query prompts to evaluate video quality, motion, and text alignment in a way that correlates strongly with human annotations. WorldFoundry integrates the Video-Bench evaluator in tree at `worldfoundry/evaluation/tasks/execution/runners/videobench`. The official repository is a protocol reference; the runtime used by WorldFoundry is checked in and does not require an external code checkout. Official references: * Project page: [video-bench.github.io](https://video-bench.github.io/) * Paper: [arXiv:2504.04907](https://arxiv.org/abs/2504.04907) * Human annotation dataset: [Video-Bench/Video-Bench\_human\_annotation](https://huggingface.co/datasets/Video-Bench/Video-Bench_human_annotation) * Official videos dataset: [Video-Bench/Video-Bench\_videos](https://huggingface.co/datasets/Video-Bench/Video-Bench_videos) * In-tree runner: `worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py` ## Evaluation Protocol [#evaluation-protocol] Video-Bench evaluates three groups: | Group | Metrics | Scale | | -------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | Static quality | `imaging_quality`, `aesthetic_quality` | 1-5 | | Dynamic quality | `temporal_consistency`, `motion_effects` | 1-5 | | Video-text alignment | `video_text_consistency`, `object_class_consistency`, `color_consistency`, `action_consistency`, `scene_consistency` | 1-5 for overall alignment, 1-3 for fine-grained alignment | WorldFoundry normalizes these dimensions and reports `videobench_average` as the mean normalized score across available dimensions. The bundled prompt metadata is `worldfoundry/data/benchmarks/assets/video-bench/VideoBench_full.json`. If you download the official human annotation dataset, the runner can also build prompt metadata from its per-dimension JSON files. ## Data Preparation [#data-preparation] Video-Bench standard mode expects generated videos organized by dimension and model name: ```text /path/to/video-bench/generated_videos/ color/ your_model/ A red bird_0.mp4 A red bird_1.mp4 object_class/ your_model/ A train_0.mp4 action/ your_model/ A person is marching_0.mp4 video-text consistency/ your_model/ Close up of grapes on a rotating table._0.mp4 ``` The `action` folder is used for `action`, `temporal_consistency`, and `motion_effects`. The `video-text consistency` folder is used for `video-text consistency`, `imaging_quality`, and `aesthetic_quality`. Set: ```bash export WORLDFOUNDRY_VIDEOBENCH_GENERATED_VIDEO_DIR=/path/to/video-bench/generated_videos ``` Optional official datasets: ```bash export WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT=/path/to/datasets/Video-Bench_human_annotation export WORLDFOUNDRY_VIDEOBENCH_OFFICIAL_VIDEOS_ROOT=/path/to/datasets/Video-Bench_videos hf download Video-Bench/Video-Bench_human_annotation \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT}" hf download Video-Bench/Video-Bench_videos \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VIDEOBENCH_OFFICIAL_VIDEOS_ROOT}" ``` Candidate model training and inference are model-specific. Generate the videos with the model package first, then place them in the Video-Bench layout above. ## Judge Credentials [#judge-credentials] Video-Bench uses GPT-4o and GPT-4o-mini compatible APIs for judging. WorldFoundry writes a private runtime config under the output directory, so you do not need to edit a repository-level `config.json`. ```bash export WORLDFOUNDRY_VIDEOBENCH_GPT4O_API_KEY="${OPENAI_API_KEY}" export WORLDFOUNDRY_VIDEOBENCH_GPT4O_MINI_API_KEY="${OPENAI_API_KEY}" export WORLDFOUNDRY_VIDEOBENCH_GPT4O_BASE_URL="${OPENAI_BASE_URL:-}" export WORLDFOUNDRY_VIDEOBENCH_GPT4O_MINI_BASE_URL="${OPENAI_BASE_URL:-}" ``` If the judge credentials are missing, the runner writes a failed scorecard explaining the missing API keys instead of silently producing scores. ## Run Standard Mode [#run-standard-mode] Run one standard dimension for a generated model: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \ --videos-path "${WORLDFOUNDRY_VIDEOBENCH_GENERATED_VIDEO_DIR}" \ --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/video-bench/VideoBench_full.json" \ --dimension color \ --mode standard \ --models your_model \ --output-dir tmp/video-bench/color_your_model \ --json ``` Run all supported dimensions for one model: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \ --videos-path "${WORLDFOUNDRY_VIDEOBENCH_GENERATED_VIDEO_DIR}" \ --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/video-bench/VideoBench_full.json" \ --models your_model \ --output-dir tmp/video-bench/your_model_full \ --timeout 7200 \ --json ``` When `--dimension` is omitted, the runner evaluates the Video-Bench dimensions it knows about. ## Run Custom Mode [#run-custom-mode] Use `custom_static` for `imaging_quality` or `aesthetic_quality`: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \ --videos-path /path/to/custom/videos \ --dimension aesthetic_quality \ --mode custom_static \ --models your_model \ --prompt "a cinematic shot of a silver robot walking through rain" \ --output-dir tmp/video-bench/custom_static \ --json ``` Use `custom_nonstatic` for dynamic quality or video-text alignment dimensions: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \ --videos-path /path/to/custom/videos \ --dimension motion_effects \ --mode custom_nonstatic \ --models your_model \ --prompt-file /path/to/video_prompts.json \ --output-dir tmp/video-bench/custom_motion \ --json ``` `--prompt-file` should follow the official custom-mode mapping from sample index or video key to prompt. ## Import Existing Results [#import-existing-results] Import an official-compatible Video-Bench result JSON or result directory: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \ --official-results-path /path/to/video_bench_results_or_dir \ --annotation-root "${WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT}" \ --official-videos-root "${WORLDFOUNDRY_VIDEOBENCH_OFFICIAL_VIDEOS_ROOT}" \ --output-dir tmp/video-bench/imported \ --json ``` ## Outputs [#outputs] Each run writes: * `scorecard.json`: normalized Video-Bench scores and `videobench_average`. * `raw_metric_table.jsonl`: metric rows used by the scorecard. * `per_sample_scores.jsonl`: per-sample judge scores when available. * `upstream/`: official runtime result files. * `logs/`: per-dimension judge logs. * `videobench_config.private.json`: generated private API config with file mode `0600`. [Back to Benchmark Hub](/docs/evaluation/benchmark-hub)