# T2V-CompBench (/docs/evaluation/benchmark-hub/t2v-compbench)



## About [#about]

T2V-CompBench is a CVPR 2025 benchmark for compositional text-to-video generation. It tests whether generated videos bind the right attributes to the right objects, preserve dynamic attributes over time, satisfy spatial relationships, bind motion and actions correctly, model object interactions, and generate the requested object counts. The official prompt suite contains 1,400 prompts across seven categories, with 200 prompts per category.

WorldFoundry integrates the official evaluation scripts in tree. The runner is `worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py`; the vendored runtime code is `worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/runtime/t2v_compbench`; prompt and metadata assets live under `worldfoundry/data/benchmarks/assets/t2v-compbench`.

Official references:

* Project page: [t2v-compbench-2025.github.io](https://t2v-compbench-2025.github.io/)
* Paper: [arXiv:2407.14505](https://arxiv.org/abs/2407.14505)
* Leaderboard: [Kaiyue/T2V-CompBench\_Leaderboard](https://huggingface.co/spaces/Kaiyue/T2V-CompBench_Leaderboard)
* Released videos dataset: [Kaiyue/T2V-CompBench-Videos](https://huggingface.co/datasets/Kaiyue/T2V-CompBench-Videos)
* Official source reference: [KaiyueSun98/T2V-CompBench](https://github.com/KaiyueSun98/T2V-CompBench)

## Evaluation Protocol [#evaluation-protocol]

Primary metric: `t2v_compbench_average`.

| Metric                         | Evaluator family            | Video subdirectory                   |
| ------------------------------ | --------------------------- | ------------------------------------ |
| `consistent_attribute_binding` | MLLM                        | `consistent_attr/`                   |
| `dynamic_attribute_binding`    | MLLM                        | `dynamic_attr/`                      |
| `spatial_relationships`        | Detection + depth           | `spatial_relationships/`             |
| `motion_binding`               | Segmentation + DOT tracking | `motion_binding/`                    |
| `action_binding`               | MLLM                        | `action_binding/`                    |
| `object_interactions`          | MLLM                        | `interaction/`                       |
| `generative_numeracy`          | Detection                   | `generative_numeracy/`               |
| `t2v_compbench_average`        | Aggregate                   | Mean over available category scores. |

Each category expects 200 generated videos named like the official prompts:

```text
/path/to/t2v-compbench/videos/
  consistent_attr/
    0001.mp4
    ...
    0200.mp4
  dynamic_attr/
    0001.mp4
    ...
  spatial_relationships/
  motion_binding/
  action_binding/
  interaction/
  generative_numeracy/
```

The runner also accepts common official dataset aliases such as `consistent_attr_1`, `dynamic_attr_2`, `spatial_3`, `motion_4`, `interaction_6`, and `numeracy_7`.

## Data And Checkpoint Preparation [#data-and-checkpoint-preparation]

Start from the WorldFoundry repository root:

```bash
cd /path/to/WorldFoundry
export WORLDFOUNDRY_T2V_COMPBENCH_ASSETS="$PWD/worldfoundry/data/benchmarks/assets/t2v-compbench"
export WORLDFOUNDRY_T2V_COMPBENCH_ROOT="$PWD/worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/runtime/t2v_compbench"
```

Download the released reference video dataset if you need official comparison assets:

```bash
hf download Kaiyue/T2V-CompBench-Videos \
  --repo-type dataset \
  --local-dir /path/to/datasets/T2V-CompBench-Videos

export WORLDFOUNDRY_T2V_COMPBENCH_DATASET_ROOT=/path/to/datasets/T2V-CompBench-Videos
```

Prepare metric model checkpoints:

```bash
export WORLDFOUNDRY_T2V_COMPBENCH_LLAVA_MODEL_PATH=/path/to/llava-v1.6-34b
export WORLDFOUNDRY_T2V_COMPBENCH_GROUNDINGDINO_CKPT=/path/to/groundingdino_swint_ogc.pth
export WORLDFOUNDRY_T2V_COMPBENCH_SAM_CKPT=/path/to/sam_vit_h_4b8939.pth
```

For `motion_binding`, stage the DOT checkpoints expected by the official DOT script under:

```text
worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/runtime/t2v_compbench/dot/checkpoints/
```

The LLaVA, GroundingDINO, SAM, depth, and DOT code paths reuse WorldFoundry base-model packages instead of external source trees. Checkpoint files can still be downloaded or staged locally.

## Generate Candidate Videos [#generate-candidate-videos]

Use the prompt files under `WORLDFOUNDRY_T2V_COMPBENCH_ASSETS/prompts/` to run your candidate text-to-video model. Put outputs into the category layout shown above, then set:

```bash
export WORLDFOUNDRY_T2V_COMPBENCH_VIDEO_ROOT=/path/to/t2v-compbench/videos
export WORLDFOUNDRY_T2V_COMPBENCH_MODEL_NAME=my_t2v_model
```

Candidate model inference and training are model-specific. For a WorldFoundry-integrated generator, run the model's synthesis workflow on each category prompt file, then export the final videos into the expected category folders.

## Run Category Scoring [#run-category-scoring]

Run an MLLM category:

```bash
cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py \
  --video-root "${WORLDFOUNDRY_T2V_COMPBENCH_VIDEO_ROOT}" \
  --dataset-root "${WORLDFOUNDRY_T2V_COMPBENCH_DATASET_ROOT}" \
  --model-name "${WORLDFOUNDRY_T2V_COMPBENCH_MODEL_NAME}" \
  --category consistent_attribute_binding \
  --llava-model-path "${WORLDFOUNDRY_T2V_COMPBENCH_LLAVA_MODEL_PATH}" \
  --output-dir tmp/t2v-compbench/consistent-attribute \
  --json
```

Run a detection/depth category:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py \
  --video-root "${WORLDFOUNDRY_T2V_COMPBENCH_VIDEO_ROOT}" \
  --dataset-root "${WORLDFOUNDRY_T2V_COMPBENCH_DATASET_ROOT}" \
  --model-name "${WORLDFOUNDRY_T2V_COMPBENCH_MODEL_NAME}" \
  --category spatial_relationships \
  --grounded-checkpoint "${WORLDFOUNDRY_T2V_COMPBENCH_GROUNDINGDINO_CKPT}" \
  --sam-checkpoint "${WORLDFOUNDRY_T2V_COMPBENCH_SAM_CKPT}" \
  --device cuda \
  --output-dir tmp/t2v-compbench/spatial \
  --json
```

Run motion binding with the required two-stage path:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py \
  --video-root "${WORLDFOUNDRY_T2V_COMPBENCH_VIDEO_ROOT}" \
  --dataset-root "${WORLDFOUNDRY_T2V_COMPBENCH_DATASET_ROOT}" \
  --model-name "${WORLDFOUNDRY_T2V_COMPBENCH_MODEL_NAME}" \
  --category motion_binding \
  --run-motion-two-stage \
  --grounded-checkpoint "${WORLDFOUNDRY_T2V_COMPBENCH_GROUNDINGDINO_CKPT}" \
  --sam-checkpoint "${WORLDFOUNDRY_T2V_COMPBENCH_SAM_CKPT}" \
  --device cuda \
  --output-dir tmp/t2v-compbench/motion \
  --json
```

Once all assets and metric dependencies are staged, you can run the full suite:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py \
  --video-root "${WORLDFOUNDRY_T2V_COMPBENCH_VIDEO_ROOT}" \
  --dataset-root "${WORLDFOUNDRY_T2V_COMPBENCH_DATASET_ROOT}" \
  --model-name "${WORLDFOUNDRY_T2V_COMPBENCH_MODEL_NAME}" \
  --category all \
  --run-motion-two-stage \
  --llava-model-path "${WORLDFOUNDRY_T2V_COMPBENCH_LLAVA_MODEL_PATH}" \
  --grounded-checkpoint "${WORLDFOUNDRY_T2V_COMPBENCH_GROUNDINGDINO_CKPT}" \
  --sam-checkpoint "${WORLDFOUNDRY_T2V_COMPBENCH_SAM_CKPT}" \
  --device cuda \
  --output-dir tmp/t2v-compbench/official-run \
  --json
```

## Import Existing CSV Results [#import-existing-csv-results]

The official leaderboard submission is a folder or zip containing up to eight CSV files:

```text
my_t2v_model_consistent_attr_score.csv
my_t2v_model_dynamic_attr_score.csv
my_t2v_model_spatial_score.csv
my_t2v_model_motion_score.csv
my_t2v_model_motion_back_fore.csv
my_t2v_model_action_binding_score.csv
my_t2v_model_object_interactions_score.csv
my_t2v_model_numeracy_video.csv
```

To convert those CSVs into a WorldFoundry scorecard:

```bash
cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2v_compbench/run_t2v_compbench_official_runner.py \
  --csv-dir /path/to/t2v-compbench/eval_results_csv \
  --video-root "${WORLDFOUNDRY_T2V_COMPBENCH_VIDEO_ROOT}" \
  --dataset-root "${WORLDFOUNDRY_T2V_COMPBENCH_DATASET_ROOT}" \
  --model-name "${WORLDFOUNDRY_T2V_COMPBENCH_MODEL_NAME}" \
  --output-dir tmp/t2v-compbench/imported \
  --json
```

## Outputs [#outputs]

The run writes:

* `scorecard.json`: WorldFoundry scorecard with seven category metrics and `t2v_compbench_average`.
* `raw_metric_table.jsonl`: normalized metric rows.
* `per_sample_scores.jsonl`: per-video rows parsed from category CSVs when available.
* `leaderboard_csv_manifest.json`: discovered official CSV files and checksums.
* `generated_video_manifest.json`: category video coverage summary.
* `dataset_manifest.json`: local reference dataset summary.
* `upstream_stdout.log` and `upstream_stderr.log`: logs from in-tree category scripts.

[Back to Benchmark Hub](/docs/evaluation/benchmark-hub)