VideoPhy

About

VideoPhy evaluates whether text-to-video models can generate videos that both follow the input caption and obey everyday physical commonsense. The benchmark contains 688 captions covering real-world material interactions such as solid-solid, solid-fluid, and fluid-fluid behavior. The original paper reports that even strong text-to-video systems often fail because they miss the requested action, violate intuitive physics, or do both at the same time.

WorldFoundry integrates the VideoPhy evaluation path in tree. The prompt manifest, metric normalization, VideoCon-Physics invocation, scorecard writing, and result import are all implemented under worldfoundry/evaluation/tasks/execution/runners/videophy, while the reusable VideoCon-Physics inference code lives under worldfoundry/base_models/llm_mllm_core/mllm/videocon_physics.

Official references:

• Project page: videophy.github.io
• Paper: arXiv:2406.03520
• Test dataset: videophysics/videophy_test_public
• Train dataset: videophysics/videophy_train_public
• VideoCon-Physics weights: videophysics/videocon_physics
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/videophy/run_videophy_official_runner.py

Official References

Evaluation Protocol

WorldFoundry uses the bundled prompt manifest:

worldfoundry/data/benchmarks/assets/videophy/prompts.json

The manifest contains the 688 official captions and prompt IDs used for generation and coverage accounting. Each candidate model must generate one video per prompt. The expected filename is:

<prompt_id>.mp4

For the full benchmark suite, the generated artifact directory should contain 1.mp4 through 688.mp4.

Primary metric: videophy_average.

VideoCon-Physics runs two entailment-style passes internally: one for semantic adherence and one for physical commonsense. The runner merges those per-video predictions into the metrics above.

Data And Checkpoint Preparation

Start from the WorldFoundry repository root:

cd "$WORLDFOUNDRY_REPO_ROOT"
export WORLDFOUNDRY_VIDEOPHY_PROMPT_MANIFEST="$PWD/worldfoundry/data/benchmarks/assets/videophy/prompts.json"

Download the VideoCon-Physics checkpoint through Hugging Face:

export WORLDFOUNDRY_VIDEOPHY_VIDEOCON_CKPT=/path/to/checkpoints/videocon_physics
hf download videophysics/videocon_physics \
  --local-dir "${WORLDFOUNDRY_VIDEOPHY_VIDEOCON_CKPT}"

If you need to compare with the released test videos, annotations, or train split, download them as datasets:

hf download videophysics/videophy_test_public \
  --repo-type dataset \
  --local-dir /path/to/datasets/videophy_test_public

hf download videophysics/videophy_train_public \
  --repo-type dataset \
  --local-dir /path/to/datasets/videophy_train_public

Leaderboard reproduction only needs the 688 prompts, candidate generated videos, and the VideoCon-Physics checkpoint. Training VideoCon-Physics is optional research work; if you train or fine-tune a replacement evaluator, pass that checkpoint directory with --videocon-checkpoint.

Generate Candidate Videos

Run your text-to-video model on the captions in WORLDFOUNDRY_VIDEOPHY_PROMPT_MANIFEST, then stage the outputs in one directory:

/path/to/videophy/generated_videos/
  1.mp4
  2.mp4
  ...
  688.mp4

Set the artifact directory before scoring:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/videophy/generated_videos

Candidate model inference is model-specific. For a WorldFoundry-integrated generator, run the model's synthesis entrypoint or model page workflow, then export the final video for each prompt ID into the layout above.

Run VideoCon-Physics Scoring

Run the in-tree VideoPhy official path:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videophy/run_videophy_official_runner.py \
  --run-official \
  --prompt-manifest "${WORLDFOUNDRY_VIDEOPHY_PROMPT_MANIFEST}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --videocon-checkpoint "${WORLDFOUNDRY_VIDEOPHY_VIDEOCON_CKPT}" \
  --batch-size "${WORLDFOUNDRY_VIDEOPHY_BATCH_SIZE:-1}" \
  --num-frames "${WORLDFOUNDRY_VIDEOPHY_NUM_FRAMES:-32}" \
  --output-dir tmp/videophy/official-run \
  --strict \
  --json

Keep --strict when reproducing the official leaderboard-style suite. It fails if any of the 688 expected videos are missing. For partial research runs, remove --strict; the scorecard will still report video coverage.

Useful runtime variables:

export WORLDFOUNDRY_VIDEOPHY_BATCH_SIZE=1
export WORLDFOUNDRY_VIDEOPHY_NUM_FRAMES=32
export WORLDFOUNDRY_VIDEOPHY_VIDEOCON_PYTHON="${WORLDFOUNDRY_UNIFIED_PYTHON:-python}"
export WORLDFOUNDRY_VIDEOPHY_SA_THRESHOLD=0.5
export WORLDFOUNDRY_VIDEOPHY_PC_THRESHOLD=0.5

Import Existing Results

If you already have a VideoPhy result JSON or CSV with per-sample sa and pc fields, or a summary CSV with metric IDs and scores, import it through the same runner:

cd "$WORLDFOUNDRY_REPO_ROOT"

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videophy/run_videophy_official_runner.py \
  --official-results-path /path/to/videophy_results.json \
  --prompt-manifest "${WORLDFOUNDRY_VIDEOPHY_PROMPT_MANIFEST}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/videophy/imported \
  --json

The unified benchmark entry can also import an existing result:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id videophy \
  --mode official-validation \
  --official-results-path /path/to/videophy_results.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/videophy/official-validation \
  --json

Outputs

The run writes:

• scorecard.json: WorldFoundry scorecard with semantic_adherence, physical_commonsense, joint_score, and videophy_average.
• videophy_results.json: merged per-prompt VideoCon-Physics result JSON when --run-official is used.
• raw_metric_table.jsonl: metric rows used by the scorecard.
• per_sample_scores.jsonl: per-prompt SA/PC/joint rows.
• videocon_physics_inputs/: generated CSV inputs for semantic-adherence and physical-commonsense passes.
• videocon_physics_outputs/: raw VideoCon-Physics CSV outputs.
• logs/: VideoCon-Physics stdout/stderr logs.

Back to Benchmark Hub