# Benchmark Hub (/docs/evaluation/benchmark-hub) ## About [#about] Index of benchmarks and models tracked by WorldFoundry. Integration status, blockers, metrics, and evidence flags come from machine-readable manifests under `worldfoundry/data/benchmarks/` and `worldfoundry/data/models/`. This page does **not** publish leaderboard rankings. ## At a glance [#at-a-glance] * **3 families** — Embodied AI · Video Generation · World Models * **58 benchmarks** — 22 embodied / 27 video / 9 world models * **60 models** in core catalogs (VLA/VA/WAM, open video, world models) * **0 leaderboard-valid cells** claimed here → [Leaderboard Eligibility](#leaderboard-eligibility) ```bash worldfoundry-eval zoo benchmarks --json # refresh counts ``` ## Benchmark list [#benchmark-list] ### Embodied AI — 22 [#embodied-ai--22] Language-conditioned agents, manipulation, sim rollouts, action traces, offline policy eval. All rows currently `leaderboard_valid=false`. VLA/VA/WAM catalog: 15 models. ### Video Generation — 27 [#video-generation--27] T2V / I2V quality, alignment, dynamics, preference judges, safety, physics, and camera control. Open-video catalog: 19 models. ### World Models — 9 [#world-models--9] Long-horizon / interactive worlds (action, camera, state, memory). World-model catalog: 26 models. ## How to run one benchmark [#how-to-run-one-benchmark] Use the benchmark-specific page first. The exact prompt file, generated-artifact layout, checkpoint names, judge model, and official result format are benchmark-dependent. ### Import official result files [#import-official-result-files] ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id \ --mode official-validation \ --official-results-path \ --generated-artifact-dir \ --output-dir tmp//official-validation \ --json ``` ### Score generated artifacts [#score-generated-artifacts] Use this only when the benchmark page says the in-tree metric runtime can recompute scores from generated artifacts and the required metric checkpoints are staged. ```bash worldfoundry-eval run \ --benchmark \ --model \ --mode official-run \ --output-dir tmp// \ --json ``` If generated artifacts already exist: ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id \ --mode official-run \ --generated-artifact-dir \ --output-dir tmp//official-run \ --json ``` Use the per-benchmark page to match the expected artifact layout before scoring: prompt index videos, case id videos, prompt id videos, sampled frame directories, reports, or structured rollout traces. {/* benchmark-subpages */} ## Benchmarks [#benchmarks] Each page describes the concrete assets, output layout, and supported runner commands for that benchmark. ### Embodied AI [#embodied-ai] * [AI2-THOR](/docs/evaluation/benchmark-hub/ai2thor) * [BEHAVIOR-1K](/docs/evaluation/benchmark-hub/behavior1k) * [BridgeData V2](/docs/evaluation/benchmark-hub/bridgedata-v2) * [CALVIN](/docs/evaluation/benchmark-hub/calvin) * [Kinetix](/docs/evaluation/benchmark-hub/kinetix) * [LIBERO](/docs/evaluation/benchmark-hub/libero) * [LIBERO-Mem](/docs/evaluation/benchmark-hub/libero-mem) * [LIBERO-Para](/docs/evaluation/benchmark-hub/libero-para) * [LIBERO-Plus](/docs/evaluation/benchmark-hub/libero-plus) * [LIBERO-Pro](/docs/evaluation/benchmark-hub/libero-pro) * [ManiSkill](/docs/evaluation/benchmark-hub/maniskill) * [ManiSkill2](/docs/evaluation/benchmark-hub/maniskill2) * [Meta-World](/docs/evaluation/benchmark-hub/metaworld) * [MiKASA-Robo](/docs/evaluation/benchmark-hub/mikasa) * [MolmoSpaces-Bench](/docs/evaluation/benchmark-hub/molmospaces) * [RLBench](/docs/evaluation/benchmark-hub/rlbench) * [RoboCasa](/docs/evaluation/benchmark-hub/robocasa) * [RoboCerebra](/docs/evaluation/benchmark-hub/robocerebra) * [RoboMME](/docs/evaluation/benchmark-hub/robomme) * [RoboTwin 2.0](/docs/evaluation/benchmark-hub/robotwin) * [SimplerEnv](/docs/evaluation/benchmark-hub/simpler-env) * [VLABench](/docs/evaluation/benchmark-hub/vlabench) ### Video Generation [#video-generation] * [VBench](/docs/evaluation/benchmark-hub/vbench) * [VBench++](/docs/evaluation/benchmark-hub/vbench-plus-plus) * [VBench-2.0](/docs/evaluation/benchmark-hub/vbench-2.0) * [Video-Bench](/docs/evaluation/benchmark-hub/video-bench) * [VMBench](/docs/evaluation/benchmark-hub/vmbench) * [T2V-CompBench](/docs/evaluation/benchmark-hub/t2v-compbench) * [VideoScore](/docs/evaluation/benchmark-hub/videoscore) * [ChronoMagic-Bench](/docs/evaluation/benchmark-hub/chronomagic-bench) * [EvalCrafter](/docs/evaluation/benchmark-hub/evalcrafter) * [FETV](/docs/evaluation/benchmark-hub/fetv) * [AIGCBench](/docs/evaluation/benchmark-hub/aigcbench) * [MiraBench](/docs/evaluation/benchmark-hub/mirabench) * [DEVIL Dynamics](/docs/evaluation/benchmark-hub/devil-dynamics) * [GenAI-Bench](/docs/evaluation/benchmark-hub/genai-bench) * [T2VSafetyBench](/docs/evaluation/benchmark-hub/t2v-safety-bench) * [CameraBench](/docs/evaluation/benchmark-hub/camerabench) * [VideoVerse](/docs/evaluation/benchmark-hub/videoverse) * [PhysVidBench](/docs/evaluation/benchmark-hub/physvidbench) * [PhyGenBench](/docs/evaluation/benchmark-hub/phygenbench) * [VideoPhy](/docs/evaluation/benchmark-hub/videophy) * [VideoPhy2](/docs/evaluation/benchmark-hub/videophy2) * [Physics-IQ](/docs/evaluation/benchmark-hub/physics-iq) * [IPV-Bench](/docs/evaluation/benchmark-hub/ipv-bench) * [VideoScience-Bench](/docs/evaluation/benchmark-hub/videoscience-bench) * [PhyEduVideo](/docs/evaluation/benchmark-hub/phyeduvideo) * [PhyGround](/docs/evaluation/benchmark-hub/phyground) * [PhyFPS-Bench-Gen](/docs/evaluation/benchmark-hub/phyfps-bench-gen) ### World Models [#world-models] * [WorldModelBench](/docs/evaluation/benchmark-hub/worldmodelbench) * [WorldScore](/docs/evaluation/benchmark-hub/worldscore) * [WorldBench](/docs/evaluation/benchmark-hub/worldbench) * [WBench](/docs/evaluation/benchmark-hub/wbench) * [iWorld-Bench](/docs/evaluation/benchmark-hub/iworld-bench) * [T2VWorldBench](/docs/evaluation/benchmark-hub/t2vworldbench) * [WorldArena](/docs/evaluation/benchmark-hub/worldarena) * [World-in-World](/docs/evaluation/benchmark-hub/world-in-world) * [EWMBench](/docs/evaluation/benchmark-hub/ewmbench)