VideoScience-Bench

Integrated

Scientific-reasoning video generation evaluation with WorldFoundry's checked-in VideoScience runner, data assets, commands, metrics, and current gaps.

Markdown Source

Last updated: Jul 11, 2026

On this page

What It Measures Official References Prepare Data And Assets Input And Output Layout Public CLI Direct Runner Metrics Limitations And Gaps

What It Measures

VideoScience-Bench evaluates whether generated videos are scientifically correct, not only visually plausible. The local VideoScience README describes 200 undergraduate-level physics and chemistry scenarios: 160 text-to-video examples, 40 image-to-video examples, 12 topics, and 103 scientific concepts. Each prompt usually requires multiple interacting concepts.

WorldFoundry uses the checked-in runner at worldfoundry/evaluation/tasks/execution/runners/videoscience_bench/run_videoscience_bench_official_runner.py and the in-tree runtime under worldfoundry/evaluation/tasks/execution/runners/videoscience_bench/runtime/videoscience_bench. Do not make a separate copy of the official repo for normal WorldFoundry use. The local /VideoScience and /videoscience-bench repos are protocol references.

Official References

Resource	Link
Paper	arXiv:2512.02942
GitHub	github.com/hao-ai-lab/VideoScience
HF dataset	lmgame/VideoScienceBench
In-tree runner	`worldfoundry/evaluation/tasks/execution/runners/videoscience_bench/run_videoscience_bench_official_runner.py`

Prepare Data And Assets

Checked-in assets:

worldfoundry/data/benchmarks/assets/videoscience-bench/database/data.csv
worldfoundry/data/benchmarks/assets/videoscience-bench/database/data_filtered.jsonl
worldfoundry/data/benchmarks/assets/videoscience-bench/database/annotations.json
worldfoundry/data/benchmarks/assets/videoscience-bench/database/annotations_analysis.json
worldfoundry/data/benchmarks/assets/videoscience-bench/sample_results.csv

Optional HF dataset mirror:

hf download lmgame/VideoScienceBench \
  --repo-type dataset \
  --local-dir /path/to/datasets/VideoScienceBench

Set the generated videos root and experiments table:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/videoscience/generated_videos
export WORLDFOUNDRY_VIDEOSCIENCE_EXPERIMENTS_CSV="$PWD/worldfoundry/data/benchmarks/assets/videoscience-bench/database/data.csv"

Configure a VLM judge provider for direct execution:

export WORLDFOUNDRY_VIDEOSCIENCE_PROVIDER=openai
export WORLDFOUNDRY_VIDEOSCIENCE_MODEL=gpt-4o
export OPENAI_API_KEY=...

The in-tree provider layer also supports Gemini and Anthropic when the matching packages and API keys are installed.

Input And Output Layout

The runner matches generated videos to data.csv rows using Unique ID. Accepted filenames include:

vid_<id>_run_1.mp4
vid_<id>.mp4
<id>.mp4

Zero-padded forms such as vid_094_run_1.mp4 are also accepted.

Example generated video layout:

/path/to/videoscience/generated_videos/
  vid_94_run_1.mp4
  vid_106_run_1.mp4
  vid_101_run_1.mp4

Public CLI output:

tmp/videoscience-bench/official-validation/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_scores.jsonl
  runner_runtime_report.json
  specialized_normalizer_stdout.log
  specialized_normalizer_stderr.log

Direct judge output:

tmp/videoscience-bench/direct-run/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_scores.jsonl
  upstream_stdout.log
  upstream_stderr.log
  upstream/
    videoscience_judge_results.json
    per_sample/

Public CLI

The catalog-supported public command is result import with official-validation:

cd "$WORLDFOUNDRY_REPO_ROOT"

worldfoundry-eval zoo benchmark-run \
  --benchmark-id videoscience-bench \
  --mode official-validation \
  --official-results-path /path/to/videoscience_judge_results.json \
  --generated-artifact-dir /path/to/videoscience/generated_videos \
  --output-dir tmp/videoscience-bench/official-validation \
  --json

Direct Runner

Import an existing VideoScience judge JSON or CSV:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoscience_bench/run_videoscience_bench_official_runner.py \
  --official-results-path /path/to/videoscience_judge_results.json \
  --output-dir tmp/videoscience-bench/direct-import \
  --json

Run the checked-in VLM judge over generated videos:

cd "$WORLDFOUNDRY_REPO_ROOT"

export OPENAI_API_KEY=...

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoscience_bench/run_videoscience_bench_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --experiments-csv "${WORLDFOUNDRY_VIDEOSCIENCE_EXPERIMENTS_CSV}" \
  --judge-provider "${WORLDFOUNDRY_VIDEOSCIENCE_PROVIDER:-openai}" \
  --judge-model "${WORLDFOUNDRY_VIDEOSCIENCE_MODEL:-gpt-4o}" \
  --max-frames 24 \
  --output-dir tmp/videoscience-bench/direct-run \
  --json

For a bounded local run:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoscience_bench/run_videoscience_bench_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --experiments-csv "${WORLDFOUNDRY_VIDEOSCIENCE_EXPERIMENTS_CSV}" \
  --judge-max-runs 10 \
  --ready-only \
  --output-dir tmp/videoscience-bench/direct-run-first10 \
  --json

--ready-only restricts the run to rows marked Done in the checked-in table.

Metrics

Metric ID	Meaning
`prompt_consistency`	Whether the generated video follows the requested setup and procedure.
`phenomenon_congruency`	Whether the expected scientific phenomenon appears and is scientifically correct.
`correct_dynamism`	Whether motions and interactions obey relevant physical laws beyond the main phenomenon.
`immutability`	Whether objects keep stable identity and attributes unless the prompt requires change.
`spatio_temporal_coherence`	Whether the video avoids flicker, teleportation, identity swaps, and unnatural transitions.
`videoscience_average`	Mean aggregate over available VideoScience metrics; primary metric.

The judge rubric is 1-4. WorldFoundry normalizes rubric scores to [0, 1] with (score - 1) / 3, and higher is better.

Limitations And Gaps

The public catalog command imports existing judge outputs. Use the direct runner for in-tree VLM judge execution.
Direct judge runs require provider credentials and can be slow or costly.
Missing generated videos are skipped. A run with no evaluated videos fails.
The in-tree runtime focuses on VLM grading over sampled frames. Optional CV evidence described by the upstream README is not fully mirrored in the current WorldFoundry runner.
Full leaderboard parity requires the same prompt selection, generated videos, provider/model settings, and judge behavior used for the reference evaluation.

Back to Benchmark Hub

VideoScience-Bench

Related