Video-Bench

Integrated

Human-preference-aligned video generation evaluation with in-tree MLLM judge runtime and runnable commands.

Last updated: Jul 11, 2026

On this page

About Evaluation Protocol Data Preparation Judge Credentials Run Standard Mode Run Custom Mode Import Existing Results Outputs

About

Video-Bench is a human-preference-aligned benchmark for generated video quality. It uses MLLM judges with few-shot scoring and chain-of-query prompts to evaluate video quality, motion, and text alignment in a way that correlates strongly with human annotations.

WorldFoundry integrates the Video-Bench evaluator in tree at worldfoundry/evaluation/tasks/execution/runners/videobench. The official repository is a protocol reference; the runtime used by WorldFoundry is checked in and does not require an external code checkout.

Official references:

Project page: video-bench.github.io
Paper: arXiv:2504.04907
Human annotation dataset: Video-Bench/Video-Bench_human_annotation
Official videos dataset: Video-Bench/Video-Bench_videos
In-tree runner: worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py

Evaluation Protocol

Video-Bench evaluates three groups:

Group	Metrics	Scale
Static quality	`imaging_quality`, `aesthetic_quality`	1-5
Dynamic quality	`temporal_consistency`, `motion_effects`	1-5
Video-text alignment	`video_text_consistency`, `object_class_consistency`, `color_consistency`, `action_consistency`, `scene_consistency`	1-5 for overall alignment, 1-3 for fine-grained alignment

WorldFoundry normalizes these dimensions and reports videobench_average as the mean normalized score across available dimensions.

The bundled prompt metadata is worldfoundry/data/benchmarks/assets/video-bench/VideoBench_full.json. If you download the official human annotation dataset, the runner can also build prompt metadata from its per-dimension JSON files.

Data Preparation

Video-Bench standard mode expects generated videos organized by dimension and model name:

/path/to/video-bench/generated_videos/
  color/
    your_model/
      A red bird_0.mp4
      A red bird_1.mp4
  object_class/
    your_model/
      A train_0.mp4
  action/
    your_model/
      A person is marching_0.mp4
  video-text consistency/
    your_model/
      Close up of grapes on a rotating table._0.mp4

The action folder is used for action, temporal_consistency, and motion_effects. The video-text consistency folder is used for video-text consistency, imaging_quality, and aesthetic_quality.

Set:

export WORLDFOUNDRY_VIDEOBENCH_GENERATED_VIDEO_DIR=/path/to/video-bench/generated_videos

Optional official datasets:

export WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT=/path/to/datasets/Video-Bench_human_annotation
export WORLDFOUNDRY_VIDEOBENCH_OFFICIAL_VIDEOS_ROOT=/path/to/datasets/Video-Bench_videos

hf download Video-Bench/Video-Bench_human_annotation \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT}"

hf download Video-Bench/Video-Bench_videos \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VIDEOBENCH_OFFICIAL_VIDEOS_ROOT}"

Candidate model training and inference are model-specific. Generate the videos with the model package first, then place them in the Video-Bench layout above.

Judge Credentials

Video-Bench uses GPT-4o and GPT-4o-mini compatible APIs for judging. WorldFoundry writes a private runtime config under the output directory, so you do not need to edit a repository-level config.json.

export WORLDFOUNDRY_VIDEOBENCH_GPT4O_API_KEY="${OPENAI_API_KEY}"
export WORLDFOUNDRY_VIDEOBENCH_GPT4O_MINI_API_KEY="${OPENAI_API_KEY}"
export WORLDFOUNDRY_VIDEOBENCH_GPT4O_BASE_URL="${OPENAI_BASE_URL:-}"
export WORLDFOUNDRY_VIDEOBENCH_GPT4O_MINI_BASE_URL="${OPENAI_BASE_URL:-}"

If the judge credentials are missing, the runner writes a failed scorecard explaining the missing API keys instead of silently producing scores.

Run Standard Mode

Run one standard dimension for a generated model:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \
  --videos-path "${WORLDFOUNDRY_VIDEOBENCH_GENERATED_VIDEO_DIR}" \
  --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/video-bench/VideoBench_full.json" \
  --dimension color \
  --mode standard \
  --models your_model \
  --output-dir tmp/video-bench/color_your_model \
  --json

Run all supported dimensions for one model:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \
  --videos-path "${WORLDFOUNDRY_VIDEOBENCH_GENERATED_VIDEO_DIR}" \
  --full-json-dir "$PWD/worldfoundry/data/benchmarks/assets/video-bench/VideoBench_full.json" \
  --models your_model \
  --output-dir tmp/video-bench/your_model_full \
  --timeout 7200 \
  --json

When --dimension is omitted, the runner evaluates the Video-Bench dimensions it knows about.

Run Custom Mode

Use custom_static for imaging_quality or aesthetic_quality:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \
  --videos-path /path/to/custom/videos \
  --dimension aesthetic_quality \
  --mode custom_static \
  --models your_model \
  --prompt "a cinematic shot of a silver robot walking through rain" \
  --output-dir tmp/video-bench/custom_static \
  --json

Use custom_nonstatic for dynamic quality or video-text alignment dimensions:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \
  --videos-path /path/to/custom/videos \
  --dimension motion_effects \
  --mode custom_nonstatic \
  --models your_model \
  --prompt-file /path/to/video_prompts.json \
  --output-dir tmp/video-bench/custom_motion \
  --json

--prompt-file should follow the official custom-mode mapping from sample index or video key to prompt.

Import Existing Results

Import an official-compatible Video-Bench result JSON or result directory:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videobench/run_videobench_official_runner.py \
  --official-results-path /path/to/video_bench_results_or_dir \
  --annotation-root "${WORLDFOUNDRY_VIDEOBENCH_ANNOTATION_ROOT}" \
  --official-videos-root "${WORLDFOUNDRY_VIDEOBENCH_OFFICIAL_VIDEOS_ROOT}" \
  --output-dir tmp/video-bench/imported \
  --json

Outputs

Each run writes:

scorecard.json: normalized Video-Bench scores and videobench_average.
raw_metric_table.jsonl: metric rows used by the scorecard.
per_sample_scores.jsonl: per-sample judge scores when available.
upstream/: official runtime result files.
logs/: per-dimension judge logs.
videobench_config.private.json: generated private API config with file mode 0600.

Back to Benchmark Hub

Video-Bench

Related