AI2-THOR

Integrated

AI2-THOR — metrics, requirements, and run commands.

Last updated: Jul 11, 2026

On this page

About Metrics Task YAML Primary Run Evaluation Set Candidate Artifacts Import Official Results Score Generated Artifacts Direct In-Tree Runner Requirements Inputs Environment Outputs

About

AI2-THOR external task manifest for native embodied closed-loop rollouts.

Metrics

AI2-THOR external task manifest for native embodied closed-loop rollouts.

Task YAML

worldfoundry/data/benchmarks/tasks/external/ai2thor.yaml

Primary

success_rate

Metric	Focus
`success_rate`	Share of episodes or tasks that complete successfully.
`task_success`	Share of episodes or tasks that complete successfully.
`episode_success`	Share of episodes or tasks that complete successfully.
`reward`	Cumulative AI2-THOR simulator reward across interactive household navigation and manipulation episodes toward the language goal.

Run Evaluation

After preparing the assets and candidate artifacts listed on this page, use the public run entry points below.

Set Candidate Artifacts

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/artifacts

Import Official Results

worldfoundry-eval zoo benchmark-run \
  --benchmark-id ai2thor \
  --mode normalizer \
  --official-results-path /path/to/official/results-or-report \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/ai2thor/normalizer \
  --json

Score Generated Artifacts

worldfoundry-eval zoo benchmark-run \
  --benchmark-id ai2thor \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/ai2thor/official-run \
  --json

Direct In-Tree Runner

worldfoundry-eval embodied run --config worldfoundry/data/benchmarks/eval_configs/embodied/ai2thor/floorplan1.yaml

Requirements

Inputs

AI2-THOR Python package
AI2-THOR Unity simulator build/cache
Headless CloudRendering or X11 display backend
Policy checkpoint or external policy server for non-zero rollouts

Environment

WORLDFOUNDRY_AI2THOR_RESULTS_PATH

Outputs

scorecard.json
run_manifest.json
results.jsonl
metrics/summary.json

← Benchmark Hub

AI2-THOR

Related