PhysVidBench

About

PhysVidBench evaluates physical commonsense in everyday text-to-video scenarios. The local source README describes a three-step flow: generate videos from prompts, extract eight AuroraCap caption tracks per video, then answer physical commonsense questions over those captions with Gemini.

WorldFoundry includes the runnable PhysVidBench runner under worldfoundry/evaluation/tasks/execution/runners/physvidbench. Do not clone the upstream code repositories for WorldFoundry runs. Downloading the Hugging Face dataset Anonymousny/PhysVidBench is fine for the full prompt/question data; code execution should still go through the in-tree runner.

Official References

Prepare Assets

• Prompt/question CSV: bundled fixture at worldfoundry/data/benchmarks/assets/physvidbench/prompts_questions.csv; override with WORLDFOUNDRY_PHYSVIDBENCH_PROMPT_MANIFEST or --prompt-manifest.
• Caption tracks: eight text files with base name cogvideo2b and suffixes _FP, _OP, _SR, _TD, _AU, _MT, _FM, and no suffix. Override with WORLDFOUNDRY_PHYSVIDBENCH_CAPTIONS_DIR, WORLDFOUNDRY_PHYSVIDBENCH_CAPTION_BASE, --captions-dir, or --caption-base.
• Full dataset: use the HF dataset Anonymousny/PhysVidBench or your own materialized prompt/question CSV when running beyond the bundled fixture.
• Candidate videos: set WORLDFOUNDRY_GENERATED_ARTIFACT_DIR to generated videos.
• Gemini judge: set GEMINI_API_KEY, GOOGLE_API_KEY, or WORLDFOUNDRY_PHYSVIDBENCH_GEMINI_API_KEY; install google-genai. For local fixture checks, set WORLDFOUNDRY_PHYSVIDBENCH_JUDGE_BACKEND=mock.

Generated Artifact Layout

PhysVidBench prompt IDs are numeric. Generated videos should be flat files using zero-padded names:

generated_videos/
  0000.mp4
  0001.mp4
  ...

The runner strips numeric leading zeroes during coverage checks, so 0.mp4 and 0000.mp4 both map to prompt ID 0. Caption files are line-indexed by PromptID, so each caption track must contain enough lines for the prompt IDs you evaluate.

Run With WorldFoundry

Import an existing PhysVidBench QA result CSV through the public CLI:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id physvidbench \
  --mode official-validation \
  --official-results-path /path/to/output.csv \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/physvidbench/validation \
  --json

Run the in-tree Gemini QA path through the public CLI:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated_videos
export GEMINI_API_KEY=...

worldfoundry-eval zoo benchmark-run \
  --benchmark-id physvidbench \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/physvidbench/run \
  --json

Direct in-tree runner:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated_videos
export WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR=tmp/physvidbench/direct
export GEMINI_API_KEY=...

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/physvidbench/run_physvidbench_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" \
  --json

Direct result import:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/physvidbench/run_physvidbench_official_runner.py \
  --official-results-path /path/to/output.csv \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/physvidbench/direct-validation \
  --json

For a bounded local fixture run without Gemini:

WORLDFOUNDRY_PHYSVIDBENCH_JUDGE_BACKEND=mock \
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/physvidbench/run_physvidbench_official_runner.py \
  --run-official \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/physvidbench/mock \
  --json

Metrics

All metrics are higher-is-better and are reported as 0..1 accuracies.

Outputs

The output directory contains:

• scorecard.json: run status, metrics, prompt/video coverage, QA backend, and leaderboard-validity flags.
• output.csv: written by the in-tree QA path, with PromptID, Question, Types, Model_Answer, and Match.
• raw_metric_table.jsonl: one row per declared metric.
• per_sample_scores.jsonl: question-level rows copied from the QA result.
• specialized_normalizer_stdout.log and specialized_normalizer_stderr.log when invoked through worldfoundry-eval zoo benchmark-run.

Limitations

• The bundled prompt/caption assets in this checkout are not a full 383-prompt benchmark pack; they are enough to document and exercise the wiring. Use the HF dataset or explicit overrides for full evaluation.
• The in-tree QA runner consumes captions; it does not run AuroraCap over videos. You must prepare the eight caption tracks yourself.
• The default non-mock backend requires Gemini API access and google-genai.
• Full leaderboard parity requires full prompt coverage, complete caption tracks, generated videos, and normalized QA output.

Back to Benchmark Hub