T2VWorldBench

About

T2VWorldBench evaluates whether text-to-video models can generate videos that express world knowledge rather than only surface visual detail. The WACV 2026 benchmark covers six categories: physics, nature, activity, culture, causality, and object knowledge. The paper uses a hybrid protocol that combines human assessment with VLM-based evaluation; the released code implements the VLM assessment component.

WorldFoundry integrates the official runtime in tree. The runner, result parser, and scorecard path live under worldfoundry/evaluation/tasks/execution/runners/t2vworldbench; the vendored official eval.py stays in runtime/t2vworldbench, while prompt and metadata assets live under worldfoundry/data/benchmarks/assets/t2vworldbench.

Official references:

• Paper: arXiv:2507.18107
• Official source reference: magiclinux/world_knowledge
• Full metadata and sample videos: official Google Drive folder
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/t2vworldbench/run_t2vworldbench_official_runner.py

What It Measures

T2VWorldBench evaluates whether text-to-video models express world knowledge — physics, nature, activity, culture, causality, and object understanding — rather than only surface visual detail. The WACV 2026 benchmark uses 1,200 prompts across 6 categories and 60 subcategories with a hybrid human + VLM protocol.

Benchmark Design

Primary aggregate: world_knowledge_average (mean over available category scores).

Evaluation Protocol

The paper prompt CSV is bundled here:

worldfoundry/data/benchmarks/assets/t2vworldbench/prompt/prompt.csv

The official VLM evaluator consumes a JSON metadata file. Each item should include:

The generated video directory can contain either all videos for the run or a single video. For a full reproduction, stage the complete official metadata JSON and one generated video per metadata item:

/path/to/t2vworldbench/generated_videos/
  Chopsticks.mp4
  Mongolia.mp4
  ...

/path/to/t2vworldbench/meta_data.json

If you use numbered filenames such as 0001.mp4, make sure id or name in the metadata JSON is also 0001; number is used for sorting, not prompt lookup.

Primary metric: world_knowledge_average.

Data And Model Preparation

Start from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export WORLDFOUNDRY_T2VWORLDBENCH_RUNTIME="$PWD/worldfoundry/evaluation/tasks/execution/runners/t2vworldbench/runtime/t2vworldbench"
export WORLDFOUNDRY_T2VWORLDBENCH_ASSETS="$PWD/worldfoundry/data/benchmarks/assets/t2vworldbench"
export WORLDFOUNDRY_T2VWORLDBENCH_PROMPT_CSV="$WORLDFOUNDRY_T2VWORLDBENCH_ASSETS/prompt/prompt.csv"
export WORLDFOUNDRY_T2VWORLDBENCH_PROMPT_FILE="$WORLDFOUNDRY_T2VWORLDBENCH_ASSETS/data/meta_data/meta_data.json"

Download or prepare the official metadata JSON. The repository includes a tiny example at:

worldfoundry/data/benchmarks/assets/t2vworldbench/data/meta_data/meta_data.json

The bundled file is only a small in-tree metadata asset. For leaderboard-style reproduction, use the full metadata from the official Drive release:

export WORLDFOUNDRY_T2VWORLDBENCH_PROMPT_FILE=/path/to/t2vworldbench/meta_data.json

The VLM evaluator uses the LLaVA-based code that has been integrated through WorldFoundry base-model modules. Stage the required LLaVA checkpoint according to your local base-model setup, then run with a Python environment that can import the WorldFoundry evaluation dependencies.

Generate Candidate Videos

Run your candidate T2V model on the prompts from the metadata file or from prompt.csv, then write videos using metadata-matching filenames:

/path/to/t2vworldbench/generated_videos/
  <id>.mp4
  <id>.mp4
  ...

Set the video root and model name before scoring:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/t2vworldbench/generated_videos
export WORLDFOUNDRY_T2VWORLDBENCH_MODEL_NAME=my_t2v_model

Candidate model inference and training are model-specific. For a WorldFoundry-integrated generator, run the model's synthesis workflow with these prompts, then export the final videos into the layout above.

Run VLM Scoring

Run the in-tree T2VWorldBench official runtime:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2vworldbench/run_t2vworldbench_official_runner.py \
  --run-official \
  --model-name "${WORLDFOUNDRY_T2VWORLDBENCH_MODEL_NAME}" \
  --prompt-file "${WORLDFOUNDRY_T2VWORLDBENCH_PROMPT_FILE}" \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/t2vworldbench/official-run \
  --json

The official evaluator writes <model_name>_video_assessment_scores.csv under the runner output directory. WorldFoundry then parses that CSV into the standard scorecard.

Import Existing Results

If you have already run the T2VWorldBench evaluator and have the official CSV, import it directly:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/t2vworldbench/run_t2vworldbench_official_runner.py \
  --official-results-path /path/to/my_t2v_model_video_assessment_scores.csv \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/t2vworldbench/imported \
  --json

The unified benchmark entry can import the same CSV:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id t2vworldbench \
  --mode official-validation \
  --official-results-path /path/to/my_t2v_model_video_assessment_scores.csv \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/t2vworldbench/official-validation \
  --json

Outputs

The run writes:

• scorecard.json: WorldFoundry scorecard with six category metrics and world_knowledge_average.
• raw_metric_table.jsonl: metric rows parsed from the official CSV.
• per_sample_scores.jsonl: per-video rows when the CSV exposes sample-level scores.
• upstream_stdout.log and upstream_stderr.log: logs from the in-tree official runtime.
• upstream/: raw evaluator output directory for --run-official.

Back to Benchmark Hub