RoboMME

Normalizer

RoboMME — metrics, requirements, and run commands.

Last updated: Jul 11, 2026

On this page

About Metrics Task YAML Primary Environment setup Runtime environment Evaluation data Checkpoints & assets Key environment variables Run Evaluation Set Candidate Artifacts Import Official Results Score Generated Artifacts Direct In-Tree Runner Requirements Inputs Environment Outputs

About

RoboMME external benchmark manifest for memory and multimodal embodied policy evaluation.

Metrics

RoboMME external benchmark manifest for memory and multimodal embodied policy evaluation.

Task YAML

worldfoundry/data/benchmarks/tasks/external/robomme.yaml

Primary

success_rate

Metric	Focus
`success_rate`	Share of episodes or tasks that complete successfully.
`counting_success_rate`	Share of episodes or tasks that complete successfully.
`permanence_success_rate`	Share of episodes or tasks that complete successfully.
`reference_success_rate`	Share of episodes or tasks that complete successfully.
`imitation_success_rate`	Share of episodes or tasks that complete successfully.

Environment setup

Runtime environment

Conda env: worldfoundry-unified-cu128.
Separate setup: defaults to the unified WorldFoundry env (needs_new_env: false); a benchmark-only conda env is usually not required.
WorldFoundry provides contract and official result normalization for caller-provided RoboMME outputs.

Evaluation data

Candidate model outputs: set WORLDFOUNDRY_ROBOMME_RESULTS_PATH to the generated-video or rollout artifact root.
${WORLDFOUNDRY_DATA_DIR}/datasets/robomme
${WORLDFOUNDRY_CACHE_DIR}/repos/RoboMME--robomme_policy_learning

Checkpoints & assets

This benchmark is primarily rollout- or official-result-driven; see catalog requires and the task YAML for policy/simulator checkpoints.

Key environment variables

WORLDFOUNDRY_ROBOMME_ROOT
WORLDFOUNDRY_ROBOMME_RESULTS_PATH

Run Evaluation

After preparing the assets and candidate artifacts listed on this page, use the public run entry points below.

Set Candidate Artifacts

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/artifacts

Import Official Results

worldfoundry-eval zoo benchmark-run \
  --benchmark-id robomme \
  --mode normalizer \
  --official-results-path /path/to/official/results-or-report \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/robomme/normalizer \
  --json

Score Generated Artifacts

worldfoundry-eval zoo benchmark-run \
  --benchmark-id robomme \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/robomme/official-run \
  --json

Direct In-Tree Runner

worldfoundry-eval zoo benchmark-run --benchmark-id robomme --mode official-validation --official-results-path <official_results.json> --output-dir <out> --json