Embodied Official Runtime Setup

What you get from this repository

WorldFoundry ships result normalizers, CLI entry points, and simulator interfaces under worldfoundry.evaluation.tasks.embodied. Checkpoints, simulator assets, Docker images, and rollout outputs are not in git. You point WorldFoundry at them with environment variables or CLI flags on your machine.

integration_status: integrated means the surface exists. It does not mean leaderboard evidence exists. Scorecards stay official_benchmark_verified=false until you complete the official protocol.

Quick start (no assets required)

After cloning, verify the catalog and runnable evaluation surfaces:

worldfoundry-eval zoo benchmarks --json

This confirms the bundled libero, robomme, and robotwin entries are discoverable. Actual scoring uses official rollout outputs or simulator execution.

Step 1: Set your roots

All paths resolve from WORLDFOUNDRY_* environment variables or CLI arguments. Replace placeholders with directories on your machine:

export WORLDFOUNDRY_DATA_DIR="$HOME/worldfoundry-data"
export WORLDFOUNDRY_HFD_DATASET_ROOT="$HOME/worldfoundry-data"   # HF dataset mirror root; may equal DATA_DIR
export WORLDFOUNDRY_CKPT_DIR="$HOME/worldfoundry-checkpoints"
export WORLDFOUNDRY_MODEL_DIR="$HOME/worldfoundry-models"
export WORLDFOUNDRY_CACHE_DIR="$HOME/worldfoundry-cache"
export WORLDFOUNDRY_ARTIFACT_DIR="$HOME/worldfoundry-artifacts"
export WORLDFOUNDRY_MODEL_SOURCE_DIR="$HOME/worldfoundry-upstream-repos"  # Root for cloned upstream repos

Layout is flexible. Common patterns:

• Flat HF mirror under WORLDFOUNDRY_HFD_DATASET_ROOT (lerobot__libero, TianxingChen__RoboTwin2.0, …)
• Custom subtree under WORLDFOUNDRY_DATA_DIR/datasets/...

As long as the matching WORLDFOUNDRY_*_ROOT / WORLDFOUNDRY_*_CHECKPOINT variables point at real files, WorldFoundry resolves them correctly.

Step 2: Generate scaffold, then fill in your paths

Generate templates and placeholder directories (does not download large files for you):

PYTHONPATH=. ${WORLDFOUNDRY_UNIFIED_PYTHON} scripts/setup/prepare_embodied_official_assets.py \
  --create-dirs \
  --json

Outputs (default under tmp/benchmark_zoo/embodied_official_assets/, override with --output-root):

Load and override with your paths:

source tmp/benchmark_zoo/embodied_official_assets/embodied_official_env.sh
bash tmp/benchmark_zoo/embodied_official_assets/check_repo_assets.sh

export WORLDFOUNDRY_LIBERO_ROOT="$WORLDFOUNDRY_MODEL_SOURCE_DIR/LIBERO"
export WORLDFOUNDRY_LIBERO_POLICY_CHECKPOINT="$WORLDFOUNDRY_CKPT_DIR/your-policy-for-libero"
export WORLDFOUNDRY_LIBERO_RESULTS_PATH="$WORLDFOUNDRY_ARTIFACT_DIR/runs/libero/official_results.json"

Or copy worldfoundry/data/benchmarks/local_assets.example.yaml, edit ${WORLDFOUNDRY_*} placeholders, then:

export WORLDFOUNDRY_LOCAL_ASSET_MANIFEST=/path/to/your/local_assets_manifest.yaml

Step 3: Choose how far to validate

Environment strategy

Start with the unified CUDA environment:

bash scripts/setup/bootstrap_worldfoundry.sh --cuda cu128
source tmp/worldfoundry_unified_env.sh

List model env mappings: bash scripts/setup/model_env_install.sh --list

Benchmark environment variables

Each benchmark uses separate env vars for your checkout, checkpoint, and results:

Upstream code roots use WORLDFOUNDRY_<BENCHMARK>_ROOT. The full list is in the generated embodied_official_env.sh.

Validate and import

Import official results (when you already have rollout outputs):

worldfoundry-eval zoo benchmark-run \
  --benchmark-id libero \
  --mode official-validation \
  --official-results-path "$WORLDFOUNDRY_LIBERO_RESULTS_PATH" \
  --output-dir "$WORLDFOUNDRY_ARTIFACT_DIR/validation/libero" \
  --json

You can pass --official-results-path and --benchmark-data-root directly on the CLI without exporting env vars first.

Review scorecard.json, raw_results.jsonl, and evaluation/summary.json. official_benchmark_verified=false is expected for import-only runs.

Full rollout

Run the Docker-backed official evaluator or VLA evaluation harness with your policy checkpoint, write result files, then import with official-validation above.

Before rollout, record: benchmark id, upstream commit, simulator versions, dataset split, checkpoint path, action/observation contract, and result file path.

For a real policy, replace model_id in the eval config with a configured VLA policy, or point server.url at an external policy server. The policy can return {"token": "forward"} or an equivalent discrete action payload.

Bundled validation fixtures

Zero-asset validation fixtures live at:

worldfoundry/data/test_cases/evaluation/embodied_fixtures/

They cover libero, robomme, and robotwin and exercise the same import path as real rollouts, but do not constitute leaderboard evidence.

Maintainer notes

Release gates and official_benchmark_verified labeling are documented in Validation evidence.