VideoVerse

What It Measures

VideoVerse evaluates whether a text-to-video generator behaves like a video world model. The official benchmark focuses on event ordering, binary verification questions, static scene knowledge, dynamic event consistency, interaction, camera behavior, mechanics, and world-knowledge checks.

The canonical prompt suite has 300 prompts, 815 event units, and 793 verification checks. Each prompt record includes a t2v_prompt to generate from, event-order information, and yes/no checks used by the judge.

Official References

Assets And Data

WorldFoundry keeps the VideoVerse runner and prompt assets in-tree:

The optional public dataset asset is tracked as videoverse_dataset_assets and resolves to the Hugging Face dataset NNaptmn/VideoVerse when local dataset material is needed.

Generated Artifacts

Generate one video per prompt key. Put the videos directly under one directory and name each file with the prompt dictionary key:

/path/to/videoverse/generated/
  8f348e44-546c-4319-aefa-b860c02d9cbc.mp4
  dc4fa681-8b4a-413d-9571-29af7aa36c2e.mp4
  <prompt-id>.mp4

Supported video suffixes are .mp4, .mov, .mkv, .webm, .avi, and .m4v. For full-suite scoring, the directory should cover all 300 canonical prompt ids.

Dependencies

For judge execution, set one of these backends:

Gemini execution also needs the google-genai Python package in the runtime environment.

Commands

Set the generated video directory:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/videoverse/generated
export WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR=tmp/videoverse/run

Run the in-tree judge and score the produced eval_res.json:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoverse/run_videoverse_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" \
  --json

Import an existing official-shaped eval_res.json or metric JSON/JSONL:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id videoverse \
  --mode official-validation \
  --official-results-path /path/to/eval_res.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/videoverse/import \
  --json

Use --limit N on the direct runner only for bounded development runs. Do not use limited runs for leaderboard comparison.

Import Paths

If you need the scoring formula without launching the CLI, import:

from worldfoundry.evaluation.tasks.execution.runners._benchmark_metrics.formulas import videoverse_subquestion_metrics

The CLI runner itself is importable as:

from worldfoundry.evaluation.tasks.execution.runners.videoverse.run_videoverse_official_runner import (
    normalize_videoverse_results,
    run_official_videoverse,
)

Outputs

The output directory contains:

Metrics emitted by the runner are qa_accuracy, event_coverage, temporal_causality, world_knowledge_consistency, static_scene_consistency, dynamic_event_consistency, and videoverse_average. Higher is better for all seven metrics.

← Benchmark Hub