4DWorldBench

About

4DWorldBench is a comprehensive benchmark for 3D/4D world generation models. The paper and project page position world generation as a step beyond 2D video synthesis: a model should build a realistic, dynamic, physically plausible world from text, image, or video conditions while staying coherent over space, time, camera motion, and physical interactions.

WorldFoundry vendors the runnable evaluation code in tree at worldfoundry/evaluation/tasks/execution/runners/four_d_worldbench. The official repository is only a protocol reference; benchmark execution in this project uses the checked-in runtime and does not require an external GitHub checkout.

Official references:

• Project page: yeppp27.github.io/4DWorldBench.github.io
• Paper: arXiv:2511.19836
• Local runner: worldfoundry/evaluation/tasks/execution/runners/four_d_worldbench/run_four_d_worldbench_official_runner.py
• In-tree runtime: worldfoundry/evaluation/tasks/execution/runners/four_d_worldbench/runtime/four_d_worldbench

Evaluation Protocol

4DWorldBench scores generated 3D/4D world videos along four groups. WorldFoundry keeps the original metric IDs so that imported official results and newly computed results map to the same scorecard.

The primary metric is four_d_worldbench_average. Intermediate aggregates are perceptual_quality, condition_4d_alignment, physical_realism_score, and four_d_consistency.

Data Preparation

The benchmark is driven by an official dataset JSON. Each JSON file describes one evaluation split and one or more candidate models. The runner looks up the model by models[].model_name, then evaluates the generated_videos listed under that model.

Minimal structure:

{
  "dataset_info": {
    "base_path": "/path/to/4DWorldBench",
    "model_type": "text-to-3D | image-to-4D | video-to-4D",
    "condition_type": "text | image | video"
  },
  "models": [
    {
      "model_name": "your_model_name",
      "conditions": [
        {
          "condition_meta_info": "alignment_motion_control",
          "prompts": [
            {
              "prompt_id": "sample_0001",
              "prompt_key": "sample_0001",
              "condition_content": "/path/to/condition/file.txt",
              "condition_caption": "Text description of the condition",
              "generated_videos": [
                "your_model_name/sample_0001.mp4"
              ]
            }
          ]
        }
      ]
    }
  ]
}

Video paths can be absolute. Relative generated_videos are resolved against --generated-video-dir, then dataset_info.base_path, then the dataset JSON directory. For a clean run, keep the official JSON unchanged except for adding or updating the candidate model entry and generated-video paths.

Checkpoints And Dependencies

Start from the unified CUDA environment, then install the benchmark-specific metric dependencies:

cd /path/to/WorldFoundry
bash scripts/setup/model_env_install.sh --model 4dworldbench

The exact assets depend on which dimensions you run:

Useful environment variables:

export WORLDFOUNDRY_4DWORLDBENCH_KEYE_MODEL=/path/to/Kwai-Keye--Keye-VL-1_5-8B
export WORLDFOUNDRY_4DWORLDBENCH_FASTVQA_CKPT=/path/to/FAST_VQA_3D_1_1_Scr.pth
export WORLDFOUNDRY_4DWORLDBENCH_DROID_CKPT=/path/to/droid.pth
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/4dworldbench/generated_videos
export OPENAI_API_KEY=...

OPENAI_API_KEY is only needed for physics_realism; the perceptual metrics do not need it.

Generate Candidate Videos

4DWorldBench does not train or run the candidate generator. Train and run the generator through its own WorldFoundry model package or your external inference script, then write the resulting videos into the paths referenced by generated_videos.

For a leaderboard-style run, prepare one dataset JSON per official split/task and make sure:

• --model-name exactly matches models[].model_name.
• Each selected prompt has at least one video in generated_videos.
• The generated videos are readable by the active Python environment.

Run One Dimension

Run a single official dimension from the in-tree runtime:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/four_d_worldbench/run_four_d_worldbench_official_runner.py \
  --run-official \
  --dataset-json /path/to/4dworldbench_dataset.json \
  --model-name your_model_name \
  --dimension perceptual_clip_iqa_metrics \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/4dworldbench/your_model_name/perceptual_clip_iqa_metrics \
  --json

The upstream metric result is written under tmp/4dworldbench/your_model_name/perceptual_clip_iqa_metrics/upstream/4dworldbench_results.json; the normalized WorldFoundry scorecard is written to scorecard.json.

Run A Full Metric Set

Run the dimensions that are required for your split and asset set. This shell loop is useful when all dependencies are installed:

cd /path/to/WorldFoundry

export DATASET_JSON=/path/to/4dworldbench_dataset.json
export MODEL_NAME=your_model_name
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/4dworldbench/generated_videos

for DIMENSION in \
  perceptual_clip_iqa_metrics \
  perceptual_clip_aesthetic_metrics \
  perceptual_fastvqa \
  alignment_attribute_control \
  alignment_relationship_control \
  alignment_motion_control \
  alignment_event_control \
  alignment_scene_control \
  alignment_camera_error_metrics \
  physics_realism \
  consistency_viewpoint \
  consistency_motion_smoothness \
  consistency_motion_qa \
  consistency_style
do
  PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
    worldfoundry/evaluation/tasks/execution/runners/four_d_worldbench/run_four_d_worldbench_official_runner.py \
    --run-official \
    --dataset-json "${DATASET_JSON}" \
    --model-name "${MODEL_NAME}" \
    --dimension "${DIMENSION}" \
    --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
    --output-dir "tmp/4dworldbench/${MODEL_NAME}/${DIMENSION}" \
    --json
done

If you only want perceptual results, run the perceptual dimensions. If you want the full average, stage the judge/checkpoint assets for all four groups and run all dimensions.

Import Existing Results

If you already have official 4DWorldBench result JSON files, import them into the unified WorldFoundry scorecard format:

cd /path/to/WorldFoundry

worldfoundry-eval zoo benchmark-run \
  --benchmark-id 4dworldbench \
  --mode official-validation \
  --official-results-path /path/to/4dworldbench_results.json \
  --output-dir tmp/4dworldbench/imported \
  --json

The importer accepts a single result JSON or a directory containing result JSON files.

Outputs

Each run writes:

• scorecard.json: WorldFoundry scorecard with normalized metrics and aggregates.
• raw_metric_table.jsonl: one row per metric.
• per_sample_scores.jsonl: per-sample rows when the upstream result includes sample detail.
• upstream/4dworldbench_results.json: raw official runtime output when --run-official is used.

Back to Benchmark Hub