CLI Reference

worldfoundry-eval is the canonical command-line entrypoint for WorldFoundry. It covers catalog discovery, readiness checks, typed model inference, benchmark execution, result normalization, reporting, and validation. Use the Quickstart for the first clone-to-run walkthrough; use this page when choosing a command or checking its contract.

Install and verify

Run commands from the repository root after activating the unified environment.

bash scripts/setup/bootstrap_worldfoundry.sh
source tmp/worldfoundry_unified_env.sh
conda activate "${WORLDFOUNDRY_UNIFIED_ENV_PREFIX}"
worldfoundry-eval --help

python -m pip install -e .

# Add an optional surface only when needed.
python -m pip install -e ".[tui]"
python -m pip install -e ".[mcp]"

python -m worldfoundry --help
python -m worldfoundry zoo models --json

Running worldfoundry-eval with no arguments prints a short first-run banner. Add --help for the complete top-level command list.

Entrypoints

Examples below use worldfoundry-eval so shell history, job scripts, and bug reports share one spelling.

Help at every level

Append --help after the command whose flags you need. Options belong to the leaf command, so --json is written after zoo models, not before it.

worldfoundry-eval --help
worldfoundry-eval zoo --help
worldfoundry-eval zoo model-show --help
worldfoundry-eval run --help

Captured from the current CLI. The screenshot can be regenerated with npm run cli:screenshots in docs/fumadocs.

Model-specific help

The positional form run MODEL loads that model's inference schema and adds typed --pipeline.*, --pipeline.load.*, and --runtime.* options to its help page.

worldfoundry-eval run self-forcing --help
worldfoundry-eval run self-forcing --model-status
worldfoundry-eval run self-forcing --print-config --json

• --pipeline.* controls request inputs and generation defaults.
• --pipeline.load.* controls checkpoint and model-loading fields.
• --runtime.* controls the execution device and runner settings.
• --model-status and --print-config resolve metadata only; they do not load weights.

Choose the right command

Several commands can reach the evaluation core, but they start from different user inputs.

Complete command map

Discovery and interactive surfaces

Execution and scoring

Data, plans, and validation

Reports and integrations

The parser is the source of truth for flags. Use worldfoundry-eval <command> [<subcommand>] --help instead of copying flags from an unrelated model or benchmark recipe.

Common workflows

1. Discover and check readiness

worldfoundry-eval zoo models
worldfoundry-eval zoo benchmarks --ready-now

worldfoundry-eval zoo model-show \
  --model-id <model-id> \
  --include-manifest \
  --json

worldfoundry-eval zoo benchmark-show \
  --benchmark-id <benchmark-id> \
  --include-spec \
  --json

# Checks the local cache; it does not download.
worldfoundry-eval zoo model-download \
  --model-id <model-id> \
  --check-local \
  --json

Use human tables while exploring. Add --json only when another program will consume the result.

2. Run one model directly

Inspect the resolved fields before allocating GPU resources:

worldfoundry-eval run self-forcing --model-status
worldfoundry-eval run self-forcing --print-config --json

Then provide only the overrides for this run:

worldfoundry-eval run self-forcing \
  --pipeline.prompt "A paper boat moving down a forest stream" \
  --pipeline.num-output-frames 33 \
  --pipeline.seed 7 \
  --pipeline.load.ckpt-path /path/to/self_forcing_dmd.pt \
  --runtime.device cuda \
  --output-dir tmp/self_forcing_run

Entries marked runnable_runner can execute directly. runner_candidate and listed_only entries remain inspectable but stop before model loading with a readiness explanation.

3. Plan or execute model × benchmark runs

Plan a potentially large matrix first:

worldfoundry-eval run \
  --all-benchmarks \
  --model <model-id> \
  --plan-only \
  --output-dir tmp/worldfoundry_plan \
  --json

Execute one cell after its assets and environment pass readiness checks:

worldfoundry-eval run \
  --benchmark <benchmark-id> \
  --model <model-id> \
  --mode official-run \
  --output-dir tmp/worldfoundry_run \
  --json

Repeat --model or --benchmark to create a matrix, or use --suite <suite-id>. Add --resume to reuse completed cells whose fingerprints still match.

4. Score existing artifacts or result rows

For a benchmark-owned artifact layout:

worldfoundry-eval score \
  --benchmark <benchmark-id> \
  --artifacts /path/to/generated-artifacts \
  --mode official-run \
  --output-dir tmp/score/<benchmark-id> \
  --plan-only \
  --json

Remove --plan-only after the prepared intent reports ready: true.

For already materialized WorldFoundry result rows:

worldfoundry-eval score \
  --results tmp/results.jsonl \
  --metric artifact_count \
  --metric required_artifacts_present \
  --required-artifact video \
  --output-dir tmp/score/results \
  --json

evaluate exposes the same deterministic core with lower-level request, task, runner, cache, and model-mode fields:

worldfoundry-eval evaluate \
  --results-path tmp/results.jsonl \
  --metric artifact_count \
  --required-artifact video \
  --output-dir tmp/worldfoundry_evaluate \
  --json

5. Import official-shaped benchmark results

worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-validation \
  --official-results-path /path/to/official-results.json \
  --generated-artifact-dir /path/to/generated-artifacts \
  --output-dir tmp/<benchmark-id>/official-validation \
  --json

Benchmark-specific inputs such as data roots, score directories, prompt manifests, judge credentials, and metric subsets live on the corresponding Benchmark Hub page.

Benchmark modes

Outputs and reproducibility

Evaluation commands write durable files below --output-dir. Exact benchmark-owned extras vary, but the shared run surface uses these artifacts:

Keep the command, environment/profile identity, catalog revision, and output directory together when reproducing a score. See Validation for schema checks and release evidence.

For deterministic model outputs, configure the generation cache explicitly:

worldfoundry-eval run \
  --suite <suite-id> \
  --generation-cache-dir tmp/generation-cache \
  --generation-cache-mode read-write \
  --output-dir tmp/worldfoundry_suite \
  --json

The cache uses SQLite plus audit.jsonl; hits are recorded in the run manifest. Cache reuse does not override model or benchmark compatibility checks.

Automation conventions

• Put --json on the leaf command: worldfoundry-eval zoo models --json.
• Repeat list flags instead of comma-joining unless that command explicitly accepts comma-separated values: for example, --metric a --metric b and --model a --model b.
• KEY=VALUE flags such as --model-parameter, --model-runtime, --benchmark-parameter, and --env parse JSON-compatible values where documented by --help.
• Use stable catalog IDs in scripts. Aliases are convenient interactively, but canonical IDs make stored commands easier to audit.
• Use NO_COLOR=1 for plain redirected help, or FORCE_COLOR=1 to keep ANSI color in a compatible terminal capture.
• Do not parse human tables. Use a command's --json output or the durable files under --output-dir.
• Add --fail-on-sample-error when partial sample failure must fail a batch or CI job.

Exit status

Individual commands document stricter behavior in --help. Flags such as --fail-on-issue, --fail-on-sample-error, and --fail-on-skipped intentionally change when non-success exits are returned.

Next steps