PhyGenBench

About

PhyGenBench evaluates text-to-video physical commonsense with 160 prompts spanning 27 physical laws and four physics domains. The local source README describes PhyGenEval as a staged evaluator: key physical-phenomena checks, order verification, and full-video naturalness scoring.

WorldFoundry includes the runnable PhyGenBench code under worldfoundry/evaluation/tasks/execution/runners/phygenbench. Runtime code such as PhyGenEval/overall.py stays with the runner, while prompt/question assets live under worldfoundry/data/benchmarks/assets/phygenbench. Do not clone the upstream repository for WorldFoundry runs. Downloading Hugging Face judge checkpoints is fine when you choose the full upstream-style judge path; the benchmark code stays in-tree.

Official References

Prepare Assets

• Prompt suite: bundled at worldfoundry/data/benchmarks/assets/phygenbench/PhyGenBench/prompts.json.
• Optional prompt override: pass --prompt-manifest or set WORLDFOUNDRY_PHYGENBENCH_PROMPT_MANIFEST.
• Candidate videos: set WORLDFOUNDRY_GENERATED_ARTIFACT_DIR to generated videos.
• Existing result import: pass --official-results-path or set WORLDFOUNDRY_PHYGENBENCH_RESULTS_PATH. JSON, JSONL, CSV, and YAML-like official dimension rows are accepted by the metric loader.
• Full judge assets: the README names VQAScore / CLIP-FlanT5, GPT-4o or LLaVA-Interleave, and InternVideo2 for the staged evaluator. WorldFoundry does not bundle those model weights or API credentials.

Generated Artifact Layout

The expected generated videos are flat files named by prompt index:

generated_videos/
  output_video_1.mp4
  output_video_2.mp4
  ...
  output_video_160.mp4

WorldFoundry also accepts bare numeric stems such as 1.mp4 for coverage checks, but materialized WorldFoundry model runs are copied into output_video_{id}.mp4.

Run With WorldFoundry

Import an existing PhyGenEval result file through the public CLI:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id phygenbench \
  --mode official-validation \
  --official-results-path /path/to/phygenbench_results.json \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/phygenbench/validation \
  --json

Run the in-tree scorer wrapper through the public CLI. By default this uses the deterministic local fixture backend; set WORLDFOUNDRY_PHYGENBENCH_JUDGE_BACKEND=official only after placing the staged PhyGenEval outputs where overall.py expects them:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated_videos

worldfoundry-eval zoo benchmark-run \
  --benchmark-id phygenbench \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/phygenbench/run \
  --json

Direct in-tree runner:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated_videos
export WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR=tmp/phygenbench/direct

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/phygenbench/run_phygenbench_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" \
  --json

Direct result import:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/phygenbench/run_phygenbench_official_runner.py \
  --official-results-path /path/to/phygenbench_results.json \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/phygenbench/direct-validation \
  --json

Metrics

Scores are higher-is-better. Stage scores on 0..3, semantic scores on 0..5, and percentage-like rows are normalized into 0..1.

Outputs

The output directory contains:

• scorecard.json: run status, metric table, coverage, selected backend, and leaderboard-validity flags.
• raw_metric_table.jsonl: one row per declared metric.
• per_sample_scores.jsonl: prompt-level rows with stage scores when available.
• phygenbench_results.json: written by the in-tree scoring wrapper.
• specialized_normalizer_stdout.log and specialized_normalizer_stderr.log when invoked through worldfoundry-eval zoo benchmark-run.

Limitations

• The default local backend is deterministic fixture scoring for integration validation, not a leaderboard result.
• Full PhyGenBench parity requires generated videos for all 160 prompts plus VQAScore, order-verification, and video-naturalness judge outputs.
• The in-tree official backend dispatches the bundled PhyGenEval/overall.py, but it expects upstream-style intermediate result files to already exist.
• leaderboard_valid remains false unless the full judge stack and prompt coverage are supplied.

Back to Benchmark Hub