World-in-World

What It Measures

World-in-World evaluates visual world models by closed-loop embodied utility instead of open-loop video appearance. Task success in embodied environments is the primary signal — high visual quality does not guarantee high task success.

Benchmark Design

The platform provides a unified online planning strategy and standardized action API. Key findings from the official project: controllability matters more than visuals; action-observation post-training scaling beats upgrading pretrained generators; more inference-time rollouts improve closed-loop performance.

Overview

World-in-World evaluates visual world models by closed-loop embodied utility instead of open-loop video appearance alone. The task families are Active Recognition, Image-Goal Navigation, Active Embodied QA, and Robotic Manipulation.

WorldFoundry ships the benchmark adapter and metric aggregation code in-tree:

worldfoundry/evaluation/tasks/execution/runners/world_in_world/run_world_in_world_official_runner.py
worldfoundry/evaluation/tasks/execution/runners/world_in_world/world_in_world_runtime.py
worldfoundry/evaluation/tasks/execution/runners/world_in_world/world_in_world_official_runtime.py
worldfoundry/evaluation/tasks/execution/runners/world_in_world/world_in_world_metrics.py
worldfoundry/evaluation/tasks/execution/runners/world_in_world/runtime/official
worldfoundry/data/benchmarks/assets/world-in-world

Users should not clone the official repos for normal WorldFoundry evaluation. The upstream World-in-World README is the protocol reference for producing rollouts and task summaries. WorldFoundry currently consumes those summaries and writes normalized scorecards; it does not start the Habitat, VLM, SAM2, world-model, or manipulation servers.

Protocol references:

• Paper: World-in-World: World Models in a Closed-Loop World
• Project page: world-in-world.github.io
• Source reference: github.com/World-In-World/world-in-world
• Dataset page: zonszer/WIW_datasets

Prepare Data And Assets

Start from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export WORLDFOUNDRY_REPO_ROOT="$PWD"
export PYTHONPATH="$WORLDFOUNDRY_REPO_ROOT:${PYTHONPATH:-}"

WorldFoundry includes OpenEQA prompt assets for the default AEQA path:

worldfoundry/data/benchmarks/assets/world-in-world/
  subtrees/open-eqa/data/open-eqa-184.json
  subtrees/open-eqa/data/open-eqa-41.json
  sample_results.csv

For fuller AR, IGNav, AEQA, or manipulation evidence, prepare a local asset root. The runner can read task manifests from this root through WORLDFOUNDRY_WORLD_IN_WORLD_ASSETS_ROOT:

export WORLDFOUNDRY_WORLD_IN_WORLD_ASSETS_ROOT=/path/to/world-in-world/assets

hf download zonszer/WIW_datasets \
  eval_datasets/AR/episodes_AR.json.gz \
  eval_datasets/IGNav/episodes_IGNav.json.gz \
  eval_datasets/IGNav/igdataset_goal_imgs.zip \
  eval_datasets/AEQA/episodes_AEQA.json.gz \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_WORLD_IN_WORLD_ASSETS_ROOT}/data/WIW_datasets"

If you are producing the summaries with the upstream rollout workflow, also prepare the scene and policy assets expected by that workflow:

${WORLDFOUNDRY_WORLD_IN_WORLD_ASSETS_ROOT}/data/scene_datasets/hm3d/val/
${WORLDFOUNDRY_WORLD_IN_WORLD_ASSETS_ROOT}/data/scene_datasets/mp3d/
SAM2 or Grounding SAM2 weights for AR and AEQA when used
VLM policy checkpoints or hosted VLM credentials
World-model checkpoints used by the rollout server
Manipulation simulator assets and 3D-Diffuser/OpenPI checkpoints when used

Point WorldFoundry at the generated artifacts and result summary:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/world-in-world/generated_artifacts
export WORLDFOUNDRY_WORLD_IN_WORLD_RESULTS_PATH="${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}/world_in_world_metrics.json"
export WORLDFOUNDRY_WORLD_IN_WORLD_TASK=AEQA

The generated artifact directory should contain either metrics.json or world_in_world_metrics.json. Optional videos are used for prompt coverage checks and should be named by prompt id:

/path/to/world-in-world/generated_artifacts/
  world_in_world_metrics.json
  f2e82760-5c3c-41b1-88b6-85921b9e7b32.mp4
  traces/
    f2e82760-5c3c-41b1-88b6-85921b9e7b32.json

Accepted result summaries are JSON objects with direct metric IDs or task summaries, or CSV files with metric_id,score columns. For task summaries, the runner maps accuracy to Active Recognition, sr and spl to IGNav, mean_score and mean_efficiency to AEQA, and success_rate to Manipulation.

Run Commands

Run the direct in-tree runner against metrics under the generated artifact directory:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/world_in_world/run_world_in_world_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --task "${WORLDFOUNDRY_WORLD_IN_WORLD_TASK}" \
  --output-dir tmp/world-in-world/direct-official-run \
  --json

Import an explicit result file through the direct runner:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/world_in_world/run_world_in_world_official_runner.py \
  --official-results-path /path/to/world_in_world_metrics.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --task "${WORLDFOUNDRY_WORLD_IN_WORLD_TASK}" \
  --output-dir tmp/world-in-world/direct-official-validation \
  --json

Use the public CLI for the in-tree run path:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id world-in-world \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/world-in-world/official-run \
  --json

Use the public CLI to import an existing result summary:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id world-in-world \
  --mode official-validation \
  --official-results-path /path/to/world_in_world_metrics.json \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/world-in-world/official-validation \
  --json

Metrics

All primary World-in-World metrics are higher-is-better. Rates and normalized scores are reported on a 0 to 1 scale.

The in-tree parser can also carry video-quality values such as FVD, SSIM, PSNR, and LPIPS in the component metadata when they appear in a summary, but they are not part of the primary metric order above.

Output Layout

Direct --run-official writes:

tmp/world-in-world/direct-official-run/
  scorecard.json
  raw_metric_table.jsonl
  world_in_world_metrics.json

Direct explicit-result imports write:

tmp/world-in-world/direct-official-validation/
  scorecard.json
  raw_metric_table.jsonl

Public CLI runs also write a runner_runtime_report.json file and specialized runner stdout/stderr logs under the requested output directory. Treat scorecard.json and raw_metric_table.jsonl as the stable outputs.

Runtime Notes

World-in-World worker launch configuration is environment-variable driven; it does not contain machine-local conda paths. Worker commands default to the current Python interpreter and in-tree relative scripts. Override a worker with WORLDFOUNDRY_WORLD_IN_WORLD_<WORKER>_CMD, or override only its interpreter with WORLDFOUNDRY_WORLD_IN_WORLD_<WORKER>_PYTHON. Checkpoint-specific variables use the same prefix, for example WORLDFOUNDRY_WORLD_IN_WORLD_SAM2_WORKER_CKPT_PATH, WORLDFOUNDRY_WORLD_IN_WORLD_SAM2_WORKER_CFG_PATH, WORLDFOUNDRY_WORLD_IN_WORLD_FTWAN21_WORKER_LORA_PATH, and WORLDFOUNDRY_WORLD_IN_WORLD_NWM_WORKER_CKPT_PATH.

Limitations

• The current WorldFoundry runner is a result-summary adapter. It does not launch closed-loop rollout servers, VLM policy servers, SAM2 services, Habitat-sim, or manipulation simulation.
• Only AEQA OpenEQA prompt lists are bundled in tree. AR and IGNav episode files, scene datasets, and manipulation assets must be prepared locally when you need those task families.
• Leaderboard parity requires complete official task summaries from AR, IGNav, AEQA, and Manipulation plus matching generated videos or rollout artifacts.
• AEQA scoring in the upstream workflow can depend on an LLM judge credential. WorldFoundry imports the finished score summary rather than calling that judge from this page's commands.
• interaction_trace_consistency is reported only when the provided result summary includes trace-consistency evidence or a direct metric value.

← Benchmark Hub