PhyGround

What It Measures

PhyGround evaluates physical plausibility for text-plus-image-to-video generation. Each prompt has a text description and a first-frame conditioning image. Videos are judged on three general dimensions and prompt-specific physical-law criteria across 13 laws:

• General: persistence, physical temporal validity, and semantic adherence.
• Solid-body mechanics: gravity, inertia, momentum, impenetrability, collision, and material behavior.
• Fluid dynamics: buoyancy, displacement, flow dynamics, boundary interaction, and fluid continuity.
• Optics: reflection and shadow behavior.

The official benchmark uses a 1-5 ordinal judge scale and includes 250 prompts. WorldFoundry imports PhyGround scores.json outputs and normalizes them into the local scorecard format. The runner does not launch benchmark-local shell scripts; judge execution is expected to happen through the WorldFoundry/base-model path or to be supplied as an existing result artifact.

Official References

Data And Artifacts

A standard local data root looks like this:

<phyground_data_root>/
  prompts/phyground.json
  first_images/
  videos/<model_id>/
    <video_id>.mp4
  annotations/

The in-tree prompt resolver also accepts data/prompts/phyground.json and data/first_images/, matching the official project layout.

WorldFoundry resolves prompts from:

• --prompt-manifest <path>
• WORLDFOUNDRY_PHYGROUND_PROMPT_MANIFEST
• bundled benchmark assets
• --benchmark-data-root <root>
• WORLDFOUNDRY_PHYGROUND_DATA_ROOT or WORLDFOUNDRY_BENCHMARK_DATA_ROOT
• --phyground-root <root> or WORLDFOUNDRY_PHYGROUND_ROOT

Generate one video for each prompt and name it by the prompt video field. If the prompt id is ball_fall_0001, the expected output is:

<generated_videos>/ball_fall_0001.mp4

Supported suffixes are .mp4, .mov, .mkv, .webm, and .avi. Coverage is based on filename stems.

Dependencies And Scores

The HF dataset is NU-World-Model-Embodied-AI/phyground; it provides prompts/phyground.json and first_images/. The released judge model is NU-World-Model-Embodied-AI/phyjudge-9B, typically served through the WorldFoundry/base-model infrastructure or another compatible judge path before importing the resulting scores.json.

For the WorldFoundry runner, full scoring input is one of:

• --official-results-path /path/to/scores.json
• WORLDFOUNDRY_PHYGROUND_RESULTS_PATH=/path/to/scores.json
• <generated_videos>/scores.json
• <generated_videos>/phyground_scores.json

Supported Commands

Set candidate videos and, when importing by environment variable, the scores file:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/phyground/videos/my_model
export WORLDFOUNDRY_PHYGROUND_RESULTS_PATH=/path/to/scores.json

Import existing PhyGround scores through the public CLI:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id phyground \
  --mode official-validation \
  --benchmark-data-root /path/to/phyground_data_root \
  --official-results-path /path/to/scores.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --prompt-manifest /path/to/phyground.json \
  --output-dir tmp/phyground/import \
  --json

Use the direct runner when you need runner-specific arguments such as --phyground-root, --limit, or --run-fixture:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/phyground/run_phyground_official_runner.py \
  --official-results-path /path/to/scores.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --benchmark-data-root /path/to/phyground_data_root \
  --prompt-manifest /path/to/phyground.json \
  --output-dir tmp/phyground/import \
  --json

If scores.json is already colocated with generated videos or supplied through WORLDFOUNDRY_PHYGROUND_RESULTS_PATH, the direct --run-official path copies and normalizes it:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/phyground/run_phyground_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --benchmark-data-root /path/to/phyground_data_root \
  --output-dir tmp/phyground/run \
  --json

Import Paths

• Runner: worldfoundry.evaluation.tasks.execution.runners.phyground.run_phyground_official_runner
• Metrics: worldfoundry.evaluation.tasks.execution.runners.phyground.phyground_metrics
• Prompt layout: worldfoundry.evaluation.tasks.execution.runners.phyground.phyground_prompts
• Judge artifact adapter: worldfoundry.evaluation.tasks.execution.runners.phyground.phyground_runtime
• Official score aggregation: worldfoundry.evaluation.tasks.execution.runners.phyground.phyground_official_scoring

Result File Shape

The preferred input is the official-style scores.json with results rows:

{
  "num_videos": 250,
  "general_dimensions": ["SA", "PTV", "persistence"],
  "results": [
    {
      "video": "ball_fall_0001",
      "SA": 4,
      "PTV": 5,
      "persistence": 5,
      "physical": {
        "laws": {
          "gravity": { "score": 4, "status": "scored" }
        }
      }
    }
  ]
}

Summary CSV rows with metric_id and score are also accepted. Scores above 1.0 in summary rows are treated as percentages and divided by 100.

Outputs under --output-dir include:

• scorecard.json: run status, coverage, metric table, and result paths.
• scores.json: copied into the output directory when using --run-official.
• raw_metric_table.jsonl: one row per declared metric.
• per_sample_scores.jsonl: selected per-video judge fields.

Metrics

Primary metric: phyground_overall. Higher is better for every metric.

← Benchmark Hub