VMBench

What It Measures

VMBench (ICCV 2025) evaluates text-to-video motion quality from a human-perception alignment perspective. Most T2V benchmarks emphasize temporal consistency or frame sharpness, but they can miss three recurring motion failures:

• videos that look stable because they barely move;
• videos that move, but not at the amplitude implied by the prompt;
• videos with unnatural or commonsense-breaking motion even when frames look sharp.

VMBench introduces five perception-aligned motion metrics (PAS, OIS, TCS, CAS, MSS) and reports that its metrics improve Spearman correlation with human preference annotations by about 35.3% over prior automatic motion scorers.

Official references:

• Paper: arXiv:2503.10076
• Project page: amap-ml.github.io/VMBench-Website
• Source reference: github.com/AMAP-ML/VMBench
• Checkpoint assets: GD-ML/VMBench
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py

WorldFoundry ships the benchmark code in-tree. The upstream README is a protocol reference; normal WorldFoundry evaluation uses the vendored evaluator and bundled prompt manifest.

Benchmark Design

Prompt suite

The official prompt library contains 1,050 prompts with stable four-digit indices (0001–1050). Prompts are produced through meta-guided LLM generation and human-AI validation, covering six dynamic-scene dimensions in the upstream release.

WorldFoundry bundles the manifest at worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json.

Motion metrics

All primary scores are higher-is-better. WorldFoundry normalizes them on a 0–1 scale in the scorecard.

Data And Artifacts

Start from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export WORLDFOUNDRY_REPO_ROOT="$PWD"
export PYTHONPATH="$WORLDFOUNDRY_REPO_ROOT:${PYTHONPATH:-}"

Use the bundled prompt manifest:

export WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST="$WORLDFOUNDRY_REPO_ROOT/worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json"

Generate videos into a flat directory. Filenames must match prompt indices:

/path/to/vmbench/generated_videos/
  0001.mp4
  0002.mp4
  ...
  1050.mp4

Then point WorldFoundry at that directory:

export WORLDFOUNDRY_VMBENCH_VIDEO_DIR=/path/to/vmbench/generated_videos
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR="${WORLDFOUNDRY_VMBENCH_VIDEO_DIR}"

Download the VMBench checkpoint bundle plus companion models:

export WORLDFOUNDRY_VMBENCH_ASSET_DIR=/path/to/vmbench/assets

hf download GD-ML/VMBench \
  --local-dir "${WORLDFOUNDRY_VMBENCH_ASSET_DIR}"

hf download q-future/one-align \
  --local-dir "${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/q-future/one-align"

hf download google-bert/bert-base-uncased \
  --local-dir "${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/google-bert/bert-base-uncased"

Set the checkpoint paths used by the in-tree runtime:

export WORLDFOUNDRY_GROUNDING_DINO_BERT_BASE_UNCASED="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/google-bert/bert-base-uncased"
export WORLDFOUNDRY_QALIGN_ONE_ALIGN_MODEL="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/q-future/one-align"
export WORLDFOUNDRY_VMBENCH_CAS_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/vit_g_vmbench.pt"

export WORLDFOUNDRY_VMBENCH_GROUNDING_DINO_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/groundingdino_swinb_cogcoor.pth"
export WORLDFOUNDRY_SAM_VIT_H_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/sam_vit_h_4b8939.pth"
export WORLDFOUNDRY_SAM2_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/sam2.1_hiera_large.pt"
export WORLDFOUNDRY_COTRACKER2_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/scaled_offline.pth"

The runner defaults to all five metrics. To run a subset while preparing assets:

export WORLDFOUNDRY_VMBENCH_METRICS="pas ois tcs cas mss"

In-tree paths:

worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py
worldfoundry/evaluation/tasks/execution/runners/vmbench/vmbench_official_runtime.py
worldfoundry/evaluation/tasks/execution/runners/vmbench/runtime/official
worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json

Run With WorldFoundry

Run full metric recomputation through the direct in-tree runner:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py \
  --run-official \
  --runtime-backend official \
  --video-dir "${WORLDFOUNDRY_VMBENCH_VIDEO_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST}" \
  --output-dir tmp/vmbench/direct-official-run \
  --json

If you already have VMBench-shaped metric artifacts from another WorldFoundry evaluator, import them with the artifact backend:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py \
  --run-official \
  --runtime-backend artifact \
  --video-dir "${WORLDFOUNDRY_VMBENCH_VIDEO_DIR}" \
  --artifact-score-dir /path/to/vmbench/metric_artifacts \
  --prompt-manifest "${WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST}" \
  --output-dir tmp/vmbench/artifact-run \
  --json

Use the public CLI when importing an existing results.json or scores.csv:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id vmbench \
  --mode official-validation \
  --official-results-path /path/to/vmbench/results.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/vmbench/official-validation \
  --json

The public CLI does not currently declare a VMBench local recomputation command. Use the direct runner above for pas, ois, tcs, cas, and mss execution.

Outputs

Direct full runs write:

tmp/vmbench/direct-official-run/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_metrics.jsonl
  upstream_stdout.log
  upstream_stderr.log
  upstream/
    results.json
    scores.csv
    pas.stdout.log
    pas.stderr.log
    ois.stdout.log
    ois.stderr.log
    tcs.stdout.log
    tcs.stderr.log
    cas.stdout.log
    cas.stderr.log
    mss.stdout.log
    mss.stderr.log

Public CLI imports also write scorecard.json, raw_metric_table.jsonl, per_sample_metrics.jsonl, and runner log files under the requested output directory.

Leaderboard Notes

The official VMBench README publishes a leaderboard with scores on a 0–100 style scale. Representative upstream baselines:

The upstream README also reports roughly 6 hours 45 minutes of GPU evaluation time for all five metrics on 1,050 CogVideoX-5B videos (49 frames, 8 FPS), with TCS and MSS as the slowest stages.

WorldFoundry scorecards are local evidence. Leaderboard parity requires exactly 0001.mp4 through 1050.mp4 aligned with the bundled prompt order.

Known Limitations

• Full recomputation is heavyweight. The upstream README reports multi-hour runtime for 1,050 videos, and the WorldFoundry path still needs CUDA, PyTorch video/perception packages, and all metric checkpoints.
• The cas metric uses torchrun by default. Set WORLDFOUNDRY_VMBENCH_USE_TORCHRUN=0 if you need single-process execution.
• The in-tree runtime stops after the first metric process failure, then writes a partial scorecard showing which metric IDs were unavailable.
• Leaderboard parity requires exactly 1,050 generated videos named 0001.mp4 through 1050.mp4, matching the bundled prompt order.
• Hugging Face access for checkpoint files can vary by environment. Mirror the files locally and set the environment variables above when shared cache paths differ.

← Benchmark Hub