IPV-Bench

What It Measures

IPV-Bench evaluates physically impossible videos. The local Impossible Videos README describes two tracks:

• Text-to-video generation: generate videos from impossible-scene prompts, then score visual quality and prompt following.
• Video understanding: answer judgement, multiple-choice, and open-ended QA about impossible videos.

The official benchmark defines a taxonomy with 4 domains and 14 categories, plus an IPV-Txt prompt suite. WorldFoundry keeps the prompt suite and result parser in tree at worldfoundry/evaluation/tasks/execution/runners/ipv_bench and worldfoundry/data/benchmarks/assets/ipv-bench. Do not make a separate copy of the official repo for normal WorldFoundry use; the local /Impossible-Videos repo is a protocol reference.

Official References

Prepare Data And Assets

Checked-in assets:

• worldfoundry/data/benchmarks/assets/ipv-bench/ipv_txt_prompt_suite.json
• worldfoundry/data/benchmarks/assets/ipv-bench/judgement_question.json
• worldfoundry/data/benchmarks/assets/ipv-bench/judgement_answer.json
• worldfoundry/data/benchmarks/assets/ipv-bench/mcqa_question.json
• worldfoundry/data/benchmarks/assets/ipv-bench/mcqa_answer.json
• worldfoundry/data/benchmarks/assets/ipv-bench/openqa_question.json
• worldfoundry/data/benchmarks/assets/ipv-bench/openqa_answer.json
• worldfoundry/data/benchmarks/assets/ipv-bench/sample_results.csv

The prompt suite contains 260 canonical text-to-video prompts. Generated videos should be named by prompt_id so coverage can be computed:

/path/to/ipv/generated_videos/
  1.mp4
  2.mp4
  3.mp4

The runner accepts .mp4, .mov, .mkv, .webm, and .avi.

HF data can be mirrored when you need the released videos or task files:

hf download showlab/ImpossibleVideos \
  --repo-type dataset \
  --local-dir /path/to/datasets/ImpossibleVideos

Result File Shape

You can import summary metric rows:

metric_id,score
visual_quality,0.82
prompt_following,0.76
impossible_video_score,0.61
judgement_accuracy,0.73
mcqa_accuracy,0.70
open_qa_score,0.65

You can also import per-sample rows with fields such as prompt_id, visual_quality, prompt_following, answer, pred, mcqa_answer, mcqa_pred, open_qa_score, and open_qa_pred. The in-tree artifact importer also looks for files named like ipv_results*.csv, ipv_results*.json, ipv_results*.jsonl, and *_pred_ipv_*.json.

Output Layout

Public CLI output:

tmp/ipv-bench/official-validation/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_scores.jsonl
  runner_runtime_report.json
  specialized_normalizer_stdout.log
  specialized_normalizer_stderr.log

Direct runner output includes the same score files and an additional metadata JSON file with the prompt manifest path, metric IDs, and source result path.

Public CLI

The catalog-supported public command is result import with official-validation:

cd "$WORLDFOUNDRY_REPO_ROOT"

worldfoundry-eval zoo benchmark-run \
  --benchmark-id ipv-bench \
  --mode official-validation \
  --official-results-path /path/to/ipv_results.csv \
  --generated-artifact-dir /path/to/ipv/generated_videos \
  --output-dir tmp/ipv-bench/official-validation \
  --json

Direct Runner

Import an existing CSV, JSON, or JSONL result file:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/ipv_bench/run_ipv_bench_official_runner.py \
  --official-results-path /path/to/ipv_results.csv \
  --generated-artifact-dir /path/to/ipv/generated_videos \
  --output-dir tmp/ipv-bench/direct-import \
  --json

Import a result file that is already inside the generated artifact directory:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/ipv_bench/run_ipv_bench_official_runner.py \
  --run-official \
  --generated-artifact-dir /path/to/ipv/generated_videos \
  --output-dir tmp/ipv-bench/direct-run \
  --json

For a custom prompt file, add:

--prompt-manifest /path/to/ipv_txt_prompt_suite.json

Metrics

All metrics are normalized to [0, 1], and higher is better.

Limitations And Gaps

• WorldFoundry does not generate human annotations. Generation scores need imported annotations or judge outputs.
• The direct --run-official path is an artifact importer: it searches for existing IPV result files and computes coverage against generated videos.
• Full leaderboard parity still depends on the official prompt suite, complete generated videos, and the same human or judge protocol used by the paper.
• Open-ended QA is only as strong as the imported score. The current exact-match fallback is simple and may not reflect semantic answer quality.

Back to Benchmark Hub