iWorld-Bench

Integrated

Interactive world-model benchmark for action control, camera following, memory, and generated-video quality.

Last updated: Jul 11, 2026

On this page

About Benchmark Design Official References What To Prepare Dataset Generated Videos Metric Assets Metrics Reproduce A Leaderboard-Style Run Action Control / Diff Memory Ability / Mem Camera Following Public CLI Import Existing Results Outputs

About

iWorld-Bench evaluates camera-controllable video generation models as interactive world models. It is not a generic video-aesthetic benchmark: a model must preserve visual quality while responding to action or camera controls and remembering the scene when it revisits a location.

The official release provides evaluation code, packaged metadata, camera-trajectory resources, first-frame assets, and reference inference adapters. WorldFoundry integrates the evaluation code in tree under worldfoundry/evaluation/tasks/execution/runners/iworldbench/, so a WorldFoundry run does not require a separate official source checkout.

Benchmark Design

Task type	Description	Tasks
Action control D=1	Stationary + 9 basic translation/rotation actions	1,000
Action control D=2	Two-degree-of-freedom combined actions	1,000
Action control D=3	Three-degree-of-freedom combined actions	1,000
Action control D=4	Four-degree-of-freedom complex actions	1,000
Memory ability	Cyclic reciprocal paths requiring loop-closure memory	200
Camera following	Trajectory following from camera parameter files	700

The project curates 330K video clips down to 2.1K high-quality samples and produces 4.9K test tasks through a unified action interface Ct = [D, T, R, V] (difficulty, translation ID, rotation ID, validity).

Official References

Paper: arXiv:2605.03941
Project page: iworld-bench.com
Official source reference: github.com/EmbodiedCity/iWorld-Bench
Dataset: EmbodiedCity/iWorld-Bench-Dataset
In-tree runner: worldfoundry/evaluation/tasks/execution/runners/iworldbench/run_iworldbench_official_runner.py

What To Prepare

Dataset

Download the packaged metadata, first-frame assets, and camera-following metadata:

export WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT=/path/to/datasets/iWorld-Bench-Dataset

hf download EmbodiedCity/iWorld-Bench-Dataset \
  --repo-type dataset \
  --include "dataset/all_pack/**" \
  --local-dir "${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}"

Expected layout:

${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}/
`-- dataset/all_pack/
    |-- metadata.csv
    |-- camera_following_metadata.csv
    `-- assets/

metadata.csv is used for the Diff/action-control and Mem/memory tasks. camera_following_metadata.csv is used only for trajectory-input models.

Generated Videos

Generate videos with the model you want to evaluate before running the scorer. You can use a WorldFoundry model, an API batch job, or your own runner, as long as the output videos can be matched back to the iWorld-Bench metadata rows.

Use one directory per track:

/path/to/iworld/generated/
|-- action_control/
|   |-- <prompt_id>.mp4
|   `-- ...
|-- memory_ability/
|   |-- <prompt_id>.mp4
|   `-- ...
`-- camera_following/
    |-- <prompt_id>.mp4
    `-- ...

The filename stem should match the metadata prompt_id / sample id. If your generator writes a different official video name from the CSV, keep that name and pass the same metadata file to the runner.

Metric Assets

The in-tree runtime reuses WorldFoundry's VBench and VIPe-related code paths. For real metric execution, set the official backend and point VIPe to the in-tree base-model package or your compatible local package:

export WORLDFOUNDRY_IWORLD_BENCH_RUNTIME_BACKEND=official
export VIPE_ROOT="${VIPE_ROOT:-$PWD/worldfoundry/base_models/three_dimensions/general_3d/vipe}"
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"

Do not use the default mock backend for reported numbers. It only exists so local wiring can run without the full metric stack.

Metrics

Metric	Group	Meaning
`image_quality`	Generation quality	VBench/MUSIQ-backed frame quality.
`brightness_consistency`	Generation quality	Luminance stability across the generated clip.
`color_temperature_constraint`	Generation quality	Hue and white-balance stability.
`sharpness_retention`	Generation quality	Detail retention without blur or high-frequency artifacts.
`motion_smoothness`	Trajectory following	VBench-backed smoothness for generated motion.
`trajectory_accuracy`	Trajectory following	VIPe-estimated camera path agreement with the command.
`trajectory_tolerance`	Trajectory following	Robustness against reference trajectory NPZ files.
`memory_symmetry`	Memory ability	Loop-closure consistency on reciprocal paths.
`trajectory_alignment`	Memory ability	Bidirectional camera-trajectory consistency for memory tasks.
`iworldbench_average`	Aggregate	Mean over available official component metrics.

Reproduce A Leaderboard-Style Run

Run each official track against the videos generated for that track. The direct runner is the clearest entrypoint because it exposes both the task split and the metric selector.

Action Control / Diff

export WORLDFOUNDRY_IWORLD_BENCH_ACTION_DIR=/path/to/iworld/generated/action_control

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/iworldbench/run_iworldbench_official_runner.py \
  --run-official \
  --generated-videos-dir "${WORLDFOUNDRY_IWORLD_BENCH_ACTION_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}/dataset/all_pack/metadata.csv" \
  --split diff \
  --metric action_control \
  --output-dir tmp/iworld-bench/action-control \
  --json

Memory Ability / Mem

export WORLDFOUNDRY_IWORLD_BENCH_MEMORY_DIR=/path/to/iworld/generated/memory_ability

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/iworldbench/run_iworldbench_official_runner.py \
  --run-official \
  --generated-videos-dir "${WORLDFOUNDRY_IWORLD_BENCH_MEMORY_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}/dataset/all_pack/metadata.csv" \
  --split mem \
  --metric memory_ability \
  --output-dir tmp/iworld-bench/memory-ability \
  --json

Camera Following

export WORLDFOUNDRY_IWORLD_BENCH_CAMERA_DIR=/path/to/iworld/generated/camera_following

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/iworldbench/run_iworldbench_official_runner.py \
  --run-official \
  --generated-videos-dir "${WORLDFOUNDRY_IWORLD_BENCH_CAMERA_DIR}" \
  --prompt-manifest "${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}/dataset/all_pack/camera_following_metadata.csv" \
  --split camera_following \
  --metric camera_following \
  --output-dir tmp/iworld-bench/camera-following \
  --json

For public leaderboard claims, keep the generated videos, the scorecards from all reported tracks, and the raw reports/ CSV files written under each output directory.

Public CLI

The public CLI is useful for one-track runs. Pass environment overrides explicitly so the run uses the real evaluator:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id iworld-bench \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_IWORLD_BENCH_ACTION_DIR}" \
  --env WORLDFOUNDRY_IWORLD_BENCH_RUNTIME_BACKEND=official \
  --env WORLDFOUNDRY_IWORLD_BENCH_METRIC=action_control \
  --env WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT="${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}" \
  --output-dir tmp/iworld-bench/action-control-cli \
  --json

Use the direct runner for memory and camera-following runs because those require an explicit --split. The public CLI accepts a global --prompt-manifest option for some benchmarks, but iWorld-Bench split selection is clearer and less error-prone through the direct runner.

Import Existing Results

If you already have iWorld-Bench report CSVs or a reports/ directory, import them into a WorldFoundry scorecard:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
worldfoundry/evaluation/tasks/execution/runners/iworldbench/run_iworldbench_official_runner.py \
  --official-results-path /path/to/iworld/reports \
  --generated-videos-dir /path/to/iworld/generated/action_control \
  --prompt-manifest "${WORLDFOUNDRY_IWORLD_BENCH_DATASET_ROOT}/dataset/all_pack/metadata.csv" \
  --split diff \
  --output-dir tmp/iworld-bench/imported \
  --json

Outputs

Each run writes:

scorecard.json
raw_metric_table.jsonl
upstream_command.json when the official backend is executed
upstream reports/*.csv under the selected output directory

Open scorecard.json first. A run is only comparable to the public leaderboard when the requested split has complete video coverage and all metric assets used by the reported metric family were available.

Back to Benchmark Hub

iWorld-Bench

Related