EWMBench

About

EWMBench evaluates embodied world-model videos across scene stability, robot-motion correctness, semantic alignment, and generation diversity. It is useful when a model must preserve task-relevant state, not just produce visually plausible frames.

What It Measures

EWMBench evaluates embodied world-model videos for scene consistency, robot-motion correctness, semantic alignment, and generation diversity. It samples from AgiBot World manipulation episodes and asks whether generated rollouts preserve task-relevant state — not just visual plausibility.

Benchmark Design

The official dataset contains 4,644 rows (~270 MB) sampled from AgiBot World. Evaluation expects preprocessed gt_dataset/ and *_dataset/ frame directories with trajectory and gripper-detection outputs.

Official References

• Paper: arXiv:2505.09694
• Official source reference: github.com/AgibotTech/EWMBench
• Dataset: agibot-world/EWMBench
• Model weights: agibot-world/EWMBench-model
• In-tree runner: worldfoundry/evaluation/tasks/execution/runners/ewmbench/run_ewmbench_official_runner.py

Leaderboard Notes

Official leaderboard claims require the full Hugging Face dataset, EWMBench-style preprocessing, DINOv2/YOLO-World/Qwen/CLIP checkpoints, and upstream-compatible ewmbm_final_table.csv. WorldFoundry's default direct scorer backend is a deterministic wiring placeholder; use WORLDFOUNDRY_EWMBENCH_SCORER_BACKEND=official only after all assets are staged.

Data, Checkpoints, And Assets

For importing existing results, prepare one EWMBench result table:

• ewmbm_final_table.csv, or a JSON/CSV summary with the EWMBench metric fields.
• Generated videos in WORLDFOUNDRY_GENERATED_ARTIFACT_DIR if you want coverage metadata in the scorecard.

For full EWMBench scoring, prepare the official data and metric models:

• Hugging Face dataset: agibot-world/EWMBench.
• Ground-truth data in the EWMBench layout: gt_dataset/task_*/episode_*/prompt/, video/, and after processing, traj/ plus gripper_detection/.
• Generated samples in a directory whose name ends with _dataset, with task_*/episode_*/<trial>/video/frame_*.jpg and processed trajectory/detection outputs.
• Qwen checkpoint: Qwen/Qwen2.5-VL-7B-Instruct.
• CLIP checkpoints: openai/clip-vit-base-patch16 and ViT-B-32.pt.
• EWMBench model weights from agibot-world/EWMBench-model, including fine-tuned DINOv2 and YOLO-World weights.
• A config file matching worldfoundry/data/benchmarks/assets/ewmbench/config.template.yaml, with data.gt_path, data.val_base, save_path, and ckpt paths filled in.

WorldFoundry also ships a small task manifest at worldfoundry/data/benchmarks/assets/ewmbench/task_manifest.json. It is useful for command wiring and coverage checks, but it is not the full benchmark dataset.

Set these paths for repeatable runs:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/ewmbench/generated-videos
export WORLDFOUNDRY_EWMBENCH_RESULTS_PATH=/path/to/ewmbm_final_table.csv
export WORLDFOUNDRY_EWMBENCH_CONFIG_PATH=/path/to/ewmbench/config.yaml

Generated Output Layout

A successful run writes the stable artifacts below under the chosen output directory:

<output-dir>/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_scores.jsonl
  ewmbench_results.csv              # direct scoring route
  runner_runtime_report.json        # public CLI route

Use scorecard.json for aggregate results and raw_metric_table.jsonl for per-metric rows. per_sample_scores.jsonl mirrors rows loaded from the EWMBench table when available.

Run Commands

Public CLI: Import EWMBench Results

Use this when you already have ewmbm_final_table.csv or another supported result export.

worldfoundry-eval zoo benchmark-run \
  --benchmark-id ewmbench \
  --mode official-validation \
  --official-results-path "${WORLDFOUNDRY_EWMBENCH_RESULTS_PATH}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/ewmbench/official-validation \
  --json

Direct Runner: Import EWMBench Results

This calls the in-tree runner script directly and accepts the same result table.

PYTHONPATH=. ${WORLDFOUNDRY_UNIFIED_PYTHON:-python} \
  worldfoundry/evaluation/tasks/execution/runners/ewmbench/run_ewmbench_official_runner.py \
  --official-results-path "${WORLDFOUNDRY_EWMBENCH_RESULTS_PATH}" \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/ewmbench/direct-import \
  --json

Direct Runner: Execute The Bundled EWMBench Scorer Path

By default, the direct scorer backend is mock; it writes deterministic placeholder CSV values for CI and wiring checks. Use official only after the dataset, preprocessing, config, and checkpoints are ready.

export WORLDFOUNDRY_EWMBENCH_SCORER_BACKEND=official

PYTHONPATH=. ${WORLDFOUNDRY_UNIFIED_PYTHON:-python} \
  worldfoundry/evaluation/tasks/execution/runners/ewmbench/run_ewmbench_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --config-path "${WORLDFOUNDRY_EWMBENCH_CONFIG_PATH}" \
  --output-dir tmp/ewmbench/direct-run \
  --json

Metrics

Runtime Notes

EWMBench runs against in-tree reusable components. Scene consistency uses worldfoundry/base_models/perception_core/general_perception/dinov2 and the packaged worldfoundry/evaluation/tasks/execution/runners/ewmbench/runtime/ewmbench/dino_config.yaml. Set WORLDFOUNDRY_EWMBENCH_CONFIG_PATH only when you need to override the main EWMBench runtime config. json_repair and fastdtw remain optional: the runner falls back to the Python json module for strict JSON and to a deterministic NumPy DTW implementation when fastdtw is not installed.

Limitations

• The bundled task manifest is intentionally small. Use the Hugging Face dataset for full benchmark coverage.
• The default direct scorer backend is not a leaderboard scorer; it is deterministic placeholder scoring for local wiring checks.
• The official backend expects EWMBench-style preprocessed directories and checkpoint paths in config.yaml. It does not convert arbitrary flat videos into the full EWMBench dataset layout.
• The public CLI route imports existing result rows. Generated videos alone do not produce EWMBench metrics.
• Leaderboard-equivalent claims require the full dataset, preprocessing, DINOv2, YOLO-World, Qwen, CLIP checkpoints, and upstream-compatible result tables.

← Benchmark Hub