Evaluation

Evaluation turns model outputs into reviewable evidence: prepare inputs, generate or import results, score them, and write a scorecard. Model launch details stay in runners; benchmarks consume normalized GenerationResult rows. Scoring and leaderboard claims are separate — a successful framework import is not the same as an official leaderboard run.

For a first end-to-end path, use Usage. For flags and discovery commands, see CLI. For per-benchmark layouts, see Benchmark Hub. For metric recipes, see Metrics.

What to run

Default user shape is worldfoundry-eval run. Use evaluate when you already have materialized requests or results. Use zoo benchmark-run for integrated external benchmarks where you supply the data root, generated artifacts, or official result files.

Choose by intent

The CLI, TUI, and Workspace compile these user intents onto the same two execution requests; they do not maintain separate inference or metric implementations.

A custom-data score is useful evidence, but is not automatically comparable to an official leaderboard. Likewise, a bounded official component run may prove that the integration works without proving full benchmark comparability. prepared_evaluation.json records the classification, resolved request, configuration sources, and preflight diagnostics for every prepared run; model × benchmark plans also cap their claim at the benchmark catalog's checked-in evidence level.

Pipeline

1. Resolve mode, output directory, model/benchmark metadata, metrics, and optional runtime profile.
2. Normalize inputs into GenerationRequest rows (task YAML, dataset manifest, JSON/JSONL, or benchmark samples).
3. Produce GenerationResult rows via a WorldModelRunner or by importing existing results.
4. Score metrics against those rows.
5. Write the audit trail: run manifest, ledgers, metric summaries, report.md, and scorecard.json.

Common commands

# Materialize a small request ledger
worldfoundry-eval task materialize <task-name> \
  --benchmark <benchmark-id> \
  --dataset-manifest /path/to/dataset_manifest.json \
  --num-samples 8 \
  --output-jsonl tmp/requests.jsonl \
  --json

# Score existing results
worldfoundry-eval evaluate \
  --mode existing-results \
  --results-path tmp/results.jsonl \
  --output-dir tmp/evaluate_existing \
  --benchmark-id <benchmark-id> \
  --model-id <model-id> \
  --metric artifact_count \
  --json

# Apply VBench's complete protocol to videos generated elsewhere
worldfoundry-eval score \
  --benchmark vbench \
  --artifacts /path/to/videos \
  --output-dir tmp/vbench_custom \
  --plan-only \
  --json

# Run an in-tree model over a checksum-verified dataset manifest
worldfoundry-eval generate-score \
  --model <model-id> \
  --dataset-manifest /path/to/dataset_manifest.json \
  --metric artifact_count \
  --output-dir tmp/custom_dataset \
  --plan-only \
  --json

# Run one benchmark cell
worldfoundry-eval run \
  --benchmark <benchmark-id> \
  --model <model-id> \
  --output-dir tmp/benchmark_run \
  --mode official-run \
  --json

# Import official-shaped results (framework evidence, not a leaderboard claim)
worldfoundry-eval zoo benchmark-run \
  --benchmark-id <benchmark-id> \
  --mode official-validation \
  --official-results-path <official_results.json> \
  --generated-artifact-dir <generated_artifacts> \
  --output-dir tmp/benchmark_zoo/official_validation/<benchmark-id> \
  --json

Remove --plan-only after preflight reports ready: true.

Reproduction recipes

Checked-in reproduction profiles bind the known generator model, checkpoint, generation parameters, official prompts, and benchmark runtime without requiring users to re-enter them. Use an explicit profile when a benchmark has multiple published cells, or the benchmark id when one profile is declared as its default:

worldfoundry-eval reproduce \
  --profile vbench-zeroscope-aesthetic \
  --output-dir tmp/reproduce \
  --plan-only \
  --json

worldfoundry-eval reproduce --benchmark vbench --output-dir tmp/reproduce --plan-only --json

The checked-in VBench profile is a bounded one-sample, aesthetic_quality validation profile and is explicitly not leaderboard eligible. Profile scope and eligibility are part of the prepared plan rather than inferred from a successful process exit.

Use --recipe for a custom or newly published model cell. A reproduction recipe keeps upstream model and benchmark settings separate, so benchmark parameters cannot silently override orchestrator-owned paths or modes.

schema_version: worldfoundry-reproduction-recipe-v1
id: paper-table-1-vbench
benchmark:
  id: vbench
model:
  id: <model-id>
  variant: <variant-id>
  parameters:
    num_frames: 81
generation:
  num_inference_steps: 30
  guidance_scale: 7.5
evaluation:
  mode: official-run
  parameters: {}
  env: {}
data:
  requests_path: ./official_requests.jsonl
  id: vbench-official-prompts
reference:
  source: <paper-or-official-config-url>
  reported_score: <value>

Run worldfoundry-eval reproduce --recipe recipe.yaml --output-dir tmp/reproduce --plan-only --json first. The prepared record includes the recipe and requests SHA-256 values, source fields, model/benchmark revisions, scope, and leaderboard candidacy; execution still verifies the benchmark runner and local assets.

Benchmark-specific layouts and flags belong on each Benchmark Hub page.

Outputs

Modes

Use official-validation for framework import evidence. Use official-run only after datasets, checkpoints, environment, and official evaluator requirements are satisfied.

Contracts (short)

Subsystem ownership and data flow: Evaluation core. Flag reference: CLI.

Next

• Benchmark Hub — per-benchmark setup and run notes
• Metrics — in-tree scorers and compute APIs
• Supported models — catalog status
• Embodied official runtime — Docker / simulator path