# VMBench (/docs/evaluation/benchmark-hub/vmbench)



## What It Measures [#what-it-measures]

VMBench (ICCV 2025) evaluates **text-to-video motion quality from a human-perception alignment perspective**. Most T2V benchmarks emphasize temporal consistency or frame sharpness, but they can miss three recurring motion failures:

* videos that look stable because they barely move;
* videos that move, but not at the amplitude implied by the prompt;
* videos with unnatural or commonsense-breaking motion even when frames look sharp.

VMBench introduces five perception-aligned motion metrics (PAS, OIS, TCS, CAS, MSS) and reports that its metrics improve Spearman correlation with human preference annotations by about **35.3%** over prior automatic motion scorers.

Official references:

* Paper: [arXiv:2503.10076](https://arxiv.org/abs/2503.10076)
* Project page: [amap-ml.github.io/VMBench-Website](https://amap-ml.github.io/VMBench-Website/)
* Source reference: [github.com/AMAP-ML/VMBench](https://github.com/AMAP-ML/VMBench)
* Checkpoint assets: [GD-ML/VMBench](https://huggingface.co/GD-ML/VMBench)
* In-tree runner: `worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py`

WorldFoundry ships the benchmark code in-tree. The upstream README is a protocol reference; normal WorldFoundry evaluation uses the vendored evaluator and bundled prompt manifest.

## Benchmark Design [#benchmark-design]

### Prompt suite [#prompt-suite]

The official prompt library contains **1,050** prompts with stable four-digit indices (`0001`–`1050`). Prompts are produced through meta-guided LLM generation and human-AI validation, covering six dynamic-scene dimensions in the upstream release.

WorldFoundry bundles the manifest at `worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json`.

### Motion metrics [#motion-metrics]

| Metric ID                     | Upstream | What it tests                                                   |
| ----------------------------- | -------- | --------------------------------------------------------------- |
| `perceptible_amplitude_score` | PAS      | Whether motion amplitude is large enough to be perceived        |
| `object_integrity_score`      | OIS      | Whether moving objects preserve identity and structure          |
| `temporal_coherence_score`    | TCS      | Whether motion stays coherent over time                         |
| `commonsense_adherence_score` | CAS      | Whether depicted actions follow commonsense motion expectations |
| `motion_smoothness_score`     | MSS      | Whether motion is smooth rather than jittery                    |
| `vmbench_average`             | Avg      | Mean of available PAS/OIS/TCS/CAS/MSS scores                    |

All primary scores are higher-is-better. WorldFoundry normalizes them on a 0–1 scale in the scorecard.

## Data And Artifacts [#data-and-artifacts]

Start from the WorldFoundry repository root:

```bash
cd /path/to/WorldFoundry
export WORLDFOUNDRY_REPO_ROOT="$PWD"
export PYTHONPATH="$WORLDFOUNDRY_REPO_ROOT:${PYTHONPATH:-}"
```

Use the bundled prompt manifest:

```bash
export WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST="$WORLDFOUNDRY_REPO_ROOT/worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json"
```

Generate videos into a flat directory. Filenames must match prompt indices:

```text
/path/to/vmbench/generated_videos/
  0001.mp4
  0002.mp4
  ...
  1050.mp4
```

Then point WorldFoundry at that directory:

```bash
export WORLDFOUNDRY_VMBENCH_VIDEO_DIR=/path/to/vmbench/generated_videos
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR="${WORLDFOUNDRY_VMBENCH_VIDEO_DIR}"
```

Download the VMBench checkpoint bundle plus companion models:

```bash
export WORLDFOUNDRY_VMBENCH_ASSET_DIR=/path/to/vmbench/assets

hf download GD-ML/VMBench \
  --local-dir "${WORLDFOUNDRY_VMBENCH_ASSET_DIR}"

hf download q-future/one-align \
  --local-dir "${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/q-future/one-align"

hf download google-bert/bert-base-uncased \
  --local-dir "${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/google-bert/bert-base-uncased"
```

Set the checkpoint paths used by the in-tree runtime:

```bash
export WORLDFOUNDRY_GROUNDING_DINO_BERT_BASE_UNCASED="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/google-bert/bert-base-uncased"
export WORLDFOUNDRY_QALIGN_ONE_ALIGN_MODEL="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/q-future/one-align"
export WORLDFOUNDRY_VMBENCH_CAS_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/vit_g_vmbench.pt"

export WORLDFOUNDRY_VMBENCH_GROUNDING_DINO_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/groundingdino_swinb_cogcoor.pth"
export WORLDFOUNDRY_SAM_VIT_H_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/sam_vit_h_4b8939.pth"
export WORLDFOUNDRY_SAM2_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/sam2.1_hiera_large.pt"
export WORLDFOUNDRY_COTRACKER2_CKPT="${WORLDFOUNDRY_VMBENCH_ASSET_DIR}/scaled_offline.pth"
```

The runner defaults to all five metrics. To run a subset while preparing assets:

```bash
export WORLDFOUNDRY_VMBENCH_METRICS="pas ois tcs cas mss"
```

In-tree paths:

```text
worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py
worldfoundry/evaluation/tasks/execution/runners/vmbench/vmbench_official_runtime.py
worldfoundry/evaluation/tasks/execution/runners/vmbench/runtime/official
worldfoundry/data/benchmarks/assets/vmbench/prompts/prompts.json
```

## Run With WorldFoundry [#run-with-worldfoundry]

Run full metric recomputation through the direct in-tree runner:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py \
  --run-official \
  --runtime-backend official \
  --video-dir "${WORLDFOUNDRY_VMBENCH_VIDEO_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST}" \
  --output-dir tmp/vmbench/direct-official-run \
  --json
```

If you already have VMBench-shaped metric artifacts from another WorldFoundry evaluator, import them with the artifact backend:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/vmbench/run_vmbench_official_runner.py \
  --run-official \
  --runtime-backend artifact \
  --video-dir "${WORLDFOUNDRY_VMBENCH_VIDEO_DIR}" \
  --artifact-score-dir /path/to/vmbench/metric_artifacts \
  --prompt-manifest "${WORLDFOUNDRY_VMBENCH_PROMPT_MANIFEST}" \
  --output-dir tmp/vmbench/artifact-run \
  --json
```

Use the public CLI when importing an existing `results.json` or `scores.csv`:

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id vmbench \
  --mode official-validation \
  --official-results-path /path/to/vmbench/results.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/vmbench/official-validation \
  --json
```

The public CLI does not currently declare a VMBench local recomputation command. Use the direct runner above for `pas`, `ois`, `tcs`, `cas`, and `mss` execution.

## Outputs [#outputs]

Direct full runs write:

```text
tmp/vmbench/direct-official-run/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_metrics.jsonl
  upstream_stdout.log
  upstream_stderr.log
  upstream/
    results.json
    scores.csv
    pas.stdout.log
    pas.stderr.log
    ois.stdout.log
    ois.stderr.log
    tcs.stdout.log
    tcs.stderr.log
    cas.stdout.log
    cas.stderr.log
    mss.stdout.log
    mss.stderr.log
```

Public CLI imports also write `scorecard.json`, `raw_metric_table.jsonl`, `per_sample_metrics.jsonl`, and runner log files under the requested output directory.

## Leaderboard Notes [#leaderboard-notes]

The official VMBench README publishes a leaderboard with scores on a 0–100 style scale. Representative upstream baselines:

| Model                |  Avg |  CAS |  MSS |  OIS |  PAS |  TCS |
| -------------------- | ---: | ---: | ---: | ---: | ---: | ---: |
| Wan2.1               | 78.4 | 62.8 | 84.2 | 66.0 | 17.9 | 97.8 |
| HunyuanVideo         | 63.4 | 51.9 | 81.6 | 65.8 | 26.1 | 96.3 |
| CogVideoX-5B         | 60.6 | 50.6 | 61.6 | 75.4 | 24.6 | 91.0 |
| OpenSora-Plan-v1.3.0 | 58.9 | 39.3 | 76.0 | 78.6 |  6.0 | 94.7 |
| Mochi 1              | 53.2 | 37.7 | 62.0 | 68.6 | 14.4 | 83.6 |
| OpenSora-v1.2        | 51.6 | 31.2 | 61.9 | 73.0 |  3.4 | 88.5 |

The upstream README also reports roughly **6 hours 45 minutes** of GPU evaluation time for all five metrics on 1,050 CogVideoX-5B videos (49 frames, 8 FPS), with TCS and MSS as the slowest stages.

WorldFoundry scorecards are local evidence. Leaderboard parity requires exactly `0001.mp4` through `1050.mp4` aligned with the bundled prompt order.

## Known Limitations [#known-limitations]

* Full recomputation is heavyweight. The upstream README reports multi-hour runtime for 1,050 videos, and the WorldFoundry path still needs CUDA, PyTorch video/perception packages, and all metric checkpoints.
* The `cas` metric uses `torchrun` by default. Set `WORLDFOUNDRY_VMBENCH_USE_TORCHRUN=0` if you need single-process execution.
* The in-tree runtime stops after the first metric process failure, then writes a partial scorecard showing which metric IDs were unavailable.
* Leaderboard parity requires exactly 1,050 generated videos named `0001.mp4` through `1050.mp4`, matching the bundled prompt order.
* Hugging Face access for checkpoint files can vary by environment. Mirror the files locally and set the environment variables above when shared cache paths differ.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)