PhyEduVideo

About

PhyEduVideo evaluates text-to-video models as physics-education video generators. The source README frames the task around curriculum-aligned visual explanations: each physics concept is split into teaching points, and each teaching point has a prompt for generating an explanatory clip.

WorldFoundry includes the runnable PhyEduVideo integration under worldfoundry/evaluation/tasks/execution/runners/phyeduvideo and bundles the prompt/rubric JSON files under worldfoundry/data/benchmarks/assets/phyeduvideo. Do not clone the upstream repository for a WorldFoundry run. Hugging Face checkpoint downloads are fine for any external judge stack you choose to run, but the benchmark wrapper code is in-tree.

Official References

Prepare Assets

• Prompt suite: worldfoundry/data/benchmarks/assets/phyeduvideo/Prompts/Prompts.json contains 60 concepts and 205 teaching-point prompts.
• Rubric files: bundled SA.json, PC-1.json, PC-2.json, PC-3.json, and cap.json.
• Candidate videos: set WORLDFOUNDRY_GENERATED_ARTIFACT_DIR to generated videos named by prompt ID.
• Existing result import: pass --official-results-path or set WORLDFOUNDRY_PHYEDUVIDEO_RESULTS_PATH. CSV and JSON result files are accepted.
• Judge/checkpoint assets: full parity with the README requires the upstream InternVL / CLIP or VQA / video-quality judging stack. WorldFoundry currently imports those scores instead of running every upstream judge.

Generated Artifact Layout

Prompt IDs are built as Id{concept_id}_{teaching_point_id}. The generated-artifact directory should be flat:

generated_videos/
  Id1_1.mp4
  Id1_2.mp4
  ...
  phyeduvideo_results.csv

WorldFoundry materializes integrated model outputs into that filename layout before scoring. Coverage checks compare video stems with prompt IDs from Prompts.json.

Run With WorldFoundry

Import an existing PhyEduVideo result file through the public CLI:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id phyeduvideo \
  --mode official-validation \
  --official-results-path /path/to/phyeduvideo_results.csv \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/phyeduvideo/validation \
  --json

Run the in-tree scoring wrapper through the public CLI. This path expects a CSV/JSON result file from WORLDFOUNDRY_PHYEDUVIDEO_RESULTS_PATH or under the generated-artifact directory:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated_videos
export WORLDFOUNDRY_PHYEDUVIDEO_RESULTS_PATH=/path/to/phyeduvideo_results.csv

worldfoundry-eval zoo benchmark-run \
  --benchmark-id phyeduvideo \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/phyeduvideo/run \
  --json

Direct in-tree runner:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated_videos
export WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR=tmp/phyeduvideo/direct
export WORLDFOUNDRY_PHYEDUVIDEO_RESULTS_PATH=/path/to/phyeduvideo_results.csv

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/phyeduvideo/run_phyeduvideo_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" \
  --json

Direct result import:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/phyeduvideo/run_phyeduvideo_official_runner.py \
  --official-results-path /path/to/phyeduvideo_results.csv \
  --generated-artifact-dir /path/to/generated_videos \
  --output-dir tmp/phyeduvideo/direct-validation \
  --json

Metrics

Scores are higher-is-better and are normalized into 0..1 when source rows use larger scales.

Outputs

The output directory contains:

• scorecard.json: run status, metrics, prompt/video coverage, selected backend, and leaderboard-validity flags.
• raw_metric_table.jsonl: one row per declared metric.
• per_sample_scores.jsonl: concept/teaching-point rows when the input result file contains sample-level fields.
• phyeduvideo_results.csv or phyeduvideo_results.json: copied result file when the in-tree wrapper imports one.
• specialized_normalizer_stdout.log and specialized_normalizer_stderr.log when invoked through worldfoundry-eval zoo benchmark-run.

Limitations

• The in-tree scoring wrapper currently imports already-computed result files. It does not run the full InternVL / CLIP / VQA judge stack end to end.
• Full leaderboard parity requires generated videos for all 205 teaching points and upstream judge outputs for semantic, physics, motion, and flicker metrics.
• The bundled prompt/rubric assets are sufficient for request materialization and result aggregation, but not for recomputing every upstream metric from raw videos.
• leaderboard_valid remains false until complete judge evidence and full prompt coverage are supplied.

Back to Benchmark Hub