AI2-THOR
已接入
AI2-THOR — 指标、依赖与运行命令。
简介
AI2-THOR external task manifest for native embodied closed-loop rollouts.
指标
AI2-THOR external task manifest for native embodied closed-loop rollouts.
Task YAML
worldfoundry/data/benchmarks/tasks/external/ai2thor.yaml
Primary
success_rate
| Metric | 关注点 |
|---|---|
success_rate | episode / task 是否成功完成的比例或成功率。 |
task_success | episode / task 是否成功完成的比例或成功率。 |
episode_success | episode / task 是否成功完成的比例或成功率。 |
reward | AI2-THOR 交互式家庭导航与操作 episode 中，朝向语言目标的仿真器累计 reward。 |
运行评测
准备好该 benchmark 页面列出的资产和候选产物后，使用下面的公开运行入口。
设置候选产物目录
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/artifacts导入已有官方结果
worldfoundry-eval zoo benchmark-run \
--benchmark-id ai2thor \
--mode normalizer \
--official-results-path /path/to/official/results-or-report \
--generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
--output-dir tmp/ai2thor/normalizer \
--json从生成产物重新打分
worldfoundry-eval zoo benchmark-run \
--benchmark-id ai2thor \
--mode official-run \
--generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
--output-dir tmp/ai2thor/official-run \
--json直接调用仓内 runner
worldfoundry-eval embodied run --config worldfoundry/data/benchmarks/eval_configs/embodied/ai2thor/floorplan1.yaml依赖与产物
Inputs
- AI2-THOR Python package
- AI2-THOR Unity simulator build/cache
- Headless CloudRendering or X11 display backend
- Policy checkpoint or external policy server for non-zero rollouts
Environment
WORLDFOUNDRY_AI2THOR_RESULTS_PATH
Outputs
scorecard.jsonrun_manifest.jsonresults.jsonlmetrics/summary.json