# AI2-THOR (/zh/docs/evaluation/benchmark-hub/ai2thor)



## 简介 [#简介]

AI2-THOR external task manifest for native embodied closed-loop rollouts.

## 指标 [#指标]

AI2-THOR external task manifest for native embodied closed-loop rollouts.

### Task YAML [#task-yaml]

`worldfoundry/data/benchmarks/tasks/external/ai2thor.yaml`

### Primary [#primary]

* `success_rate`

| Metric            | 关注点                                                |
| ----------------- | -------------------------------------------------- |
| `success_rate`    | episode / task 是否成功完成的比例或成功率。                      |
| `task_success`    | episode / task 是否成功完成的比例或成功率。                      |
| `episode_success` | episode / task 是否成功完成的比例或成功率。                      |
| `reward`          | AI2-THOR 交互式家庭导航与操作 episode 中，朝向语言目标的仿真器累计 reward。 |

## 运行评测 [#运行评测]

准备好该 benchmark 页面列出的资产和候选产物后，使用下面的公开运行入口。

### 设置候选产物目录 [#设置候选产物目录]

```bash
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/artifacts
```

### 导入已有官方结果 [#导入已有官方结果]

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id ai2thor \
  --mode normalizer \
  --official-results-path /path/to/official/results-or-report \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/ai2thor/normalizer \
  --json
```

### 从生成产物重新打分 [#从生成产物重新打分]

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id ai2thor \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/ai2thor/official-run \
  --json
```

### 直接调用仓内 runner [#直接调用仓内-runner]

```bash
worldfoundry-eval embodied run --config worldfoundry/data/benchmarks/eval_configs/embodied/ai2thor/floorplan1.yaml
```

## 依赖与产物 [#依赖与产物]

### Inputs [#inputs]

* AI2-THOR Python package
* AI2-THOR Unity simulator build/cache
* Headless CloudRendering or X11 display backend
* Policy checkpoint or external policy server for non-zero rollouts

### Environment [#environment]

* `WORLDFOUNDRY_AI2THOR_RESULTS_PATH`

### Outputs [#outputs]

* `scorecard.json`
* `run_manifest.json`
* `results.jsonl`
* `metrics/summary.json`

[← 返回 Benchmark Hub](/zh/docs/evaluation/benchmark-hub)