VideoPhy-2

About

VideoPhy-2 is an action-centric benchmark for physical commonsense in text-to-video generation. Instead of only asking whether a video looks plausible, it asks whether a generated action follows the requested caption and the physical rules implied by the scene. The official release contains 200 diverse actions, detailed synthesis prompts, human evaluation annotations, and VideoPhy-2-AutoEval for automatic scoring.

WorldFoundry integrates the prompt materialization, metric computation, and AutoEval inference path in tree. The official repository is a protocol reference; benchmark execution here uses worldfoundry/evaluation/tasks/execution/runners/videophy2 and worldfoundry/base_models/llm_mllm_core/mllm/videophy2_autoeval.

Official references:

• Project page: videophy2.github.io
• Paper: arXiv:2503.06800
• Test dataset: videophysics/videophy2_test
• Train dataset: videophysics/videophy2_train
• AutoEval weights: videophysics/videophy_2_auto
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/videophy2/run_videophy2_official_runner.py

Official References

Evaluation Protocol

WorldFoundry uses the bundled prompt manifest at worldfoundry/data/benchmarks/assets/videophy2/prompts.json. It contains the 200 prompt IDs, captions, and physical-rule metadata needed to reproduce the VideoPhy-2 test protocol.

Each candidate model must generate one video per prompt. The expected filename is:

<prompt_id>.mp4

For the full official suite, the generated artifact directory should contain 1.mp4 through 200.mp4.

Primary metric: videophy2_average.

The official AutoEval model runs three tasks internally: sa for semantic adherence, pc for physical commonsense, and rule for physical-rule grounding.

Data And Checkpoint Preparation

The prompt manifest is already checked in:

cd "$WORLDFOUNDRY_REPO_ROOT"
export WORLDFOUNDRY_VIDEOPHY2_PROMPT_MANIFEST="$PWD/worldfoundry/data/benchmarks/assets/videophy2/prompts.json"

Download the AutoEval checkpoint through Hugging Face:

export WORLDFOUNDRY_VIDEOPHY2_AUTOEVAL_CKPT=/path/to/checkpoints/videophy_2_auto
hf download videophysics/videophy_2_auto \
  --local-dir "${WORLDFOUNDRY_VIDEOPHY2_AUTOEVAL_CKPT}"

If you want to compare against the official released test assets or annotations, download the dataset separately:

hf download videophysics/videophy2_test \
  --repo-type dataset \
  --local-dir /path/to/datasets/videophy2_test

The benchmark reproduction path does not require training the AutoEval model. The official training dataset is public, but WorldFoundry only vendors the AutoEval inference runtime needed for scoring. If you retrain or fine-tune an evaluator, pass the resulting checkpoint directory with --autoeval-checkpoint.

Generate Candidate Videos

Run your candidate text-to-video model on the 200 prompts in WORLDFOUNDRY_VIDEOPHY2_PROMPT_MANIFEST, then write the videos into one directory using the official names:

/path/to/videophy2/generated_videos/
  1.mp4
  2.mp4
  ...
  200.mp4

Set the artifact directory before scoring:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/videophy2/generated_videos

Candidate model training and inference are model-specific. For WorldFoundry-integrated generators, use the model's synthesis page or package entrypoint, then copy or export the final video for each prompt ID into the layout above.

Run AutoEval Scoring

Run the in-tree VideoPhy-2 AutoEval path:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videophy2/run_videophy2_official_runner.py \
  --run-official \
  --prompt-manifest "${WORLDFOUNDRY_VIDEOPHY2_PROMPT_MANIFEST}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --autoeval-checkpoint "${WORLDFOUNDRY_VIDEOPHY2_AUTOEVAL_CKPT}" \
  --batch-size "${WORLDFOUNDRY_VIDEOPHY2_BATCH_SIZE:-1}" \
  --num-frames "${WORLDFOUNDRY_VIDEOPHY2_NUM_FRAMES:-32}" \
  --output-dir tmp/videophy2/official-run \
  --strict \
  --json

Use --strict for leaderboard reproduction. It fails if the 200 expected videos are not all present. For a partial research run, remove --strict; the scorecard will still report video coverage.

Useful runtime variables:

export WORLDFOUNDRY_VIDEOPHY2_BATCH_SIZE=1
export WORLDFOUNDRY_VIDEOPHY2_NUM_FRAMES=32
export WORLDFOUNDRY_VIDEOPHY2_AUTOEVAL_PYTHON="${WORLDFOUNDRY_UNIFIED_PYTHON:-python}"

Import Existing Results

If AutoEval has already produced a VideoPhy-2 result file, import it through the same runner:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videophy2/run_videophy2_official_runner.py \
  --official-results-path /path/to/videophy2_results.json \
  --prompt-manifest "${WORLDFOUNDRY_VIDEOPHY2_PROMPT_MANIFEST}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/videophy2/imported \
  --json

The unified benchmark entry can also import an existing result:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id videophy2 \
  --mode official-validation \
  --official-results-path /path/to/videophy2_results.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/videophy2/official-validation \
  --json

Outputs

The run writes:

• scorecard.json: WorldFoundry scorecard with semantic_adherence, physical_commonsense, joint_score, rule_classification_accuracy, and videophy2_average.
• videophy2_results.json: merged AutoEval per-prompt result JSON when --run-official is used.
• raw_metric_table.jsonl: metric rows used by the scorecard.
• per_sample_scores.jsonl: per-prompt SA/PC/joint/rule rows.
• videophy2_autoeval_inputs/: generated CSV inputs for sa, pc, and rule.
• videophy2_autoeval_outputs/: raw AutoEval CSV outputs.
• logs/: AutoEval stdout/stderr logs.

Back to Benchmark Hub