Metrics

Quantitative evaluation sits at the center of progress in generative modeling. As systems move from single-frame synthesis toward video, editing, embodied rollouts, and long-horizon world simulation, the space of plausible outputs expands faster than human inspection can scale. Metrics provide the compression layer: they turn high-dimensional artifacts into scalar or vector summaries that can be compared across models, checkpoints, ablations, and benchmark suites. WorldFoundry ships an in-tree metric layer—a registry of scorers and distribution/perceptual compute routines that runners invoke through a stable contract. This hub is not a survey paper of every published score; it documents what the repository actually implements, how those implementations are discovered, and where their assumptions break down.

Benchmarks answer what to measure on which data; metrics answer how to turn predictions into numbers. In practice the boundary blurs: the same FID routine may appear in a video benchmark, a standalone ablation script, and a scorecard field named distribution.fid. Without a shared registry, teams re-implement preprocessing, checkpoint paths, and aggregation rules—and small divergences produce large leaderboard gaps that are impossible to attribute. We treat metrics as first-class artifacts, not ad hoc notebook cells. Each entry declares inputs (images, videos, feature arrays, text prompts), optional checkpoint dependencies, and the shape of its output. Runners reference metric ids; the registry resolves aliases, validates availability, and routes to the appropriate compute_* implementation. That separation keeps model code ignorant of scoring details while keeping scoring reproducible across machines.

The in-tree catalog spans four families, reflected in the sidebar. Multimodal scorers and quality metrics encode semantic agreement between conditioning and generation—CLIP-family, VQA, ITM, face, art, and open S2V judges. Distribution metrics (FID, IS, KID, FVD, JEDi, CMMD, Clean-FID, and feature-array variants) summarize population-level statistics rather than per-sample correctness. Perceptual pairwise metrics (LPIPS, SSIM, DINO similarity, mask accuracy, layout quality) measure local fidelity when paired supervision exists. Editing and layout metrics (SemSR, IRS, CAS, manipulation direction, object-wise consistency) target structured edits where the instruction, mask, or layout constraint is part of the evaluation contract. A fifth page, Registry & reference, documents discovery (worldfoundry-eval metric list), parameterized ids, checkpoint staging, and package layout—consult it when registering a new metric or debugging why a run skipped a score.

Metrics that depend on pretrained backbones declare those dependencies explicitly; missing checkpoints fail loudly rather than silently substituting proxies. Registry keys normalize case and hyphenation so runners and scorecards store canonical ids while implementations evolve behind a stable contract. Benchmark manifests choose which metrics apply to which task; this hub describes how each metric behaves when invoked. For benchmark-specific evaluator assets and commands, use Benchmark Hub. Not every leaderboard number in the literature has an in-tree implementation; individual pages call out approximations, unsupported modes, and cases where official benchmark scripts must be used instead of the generic compute API.

Getting started

Three steps:

1. source WorldFoundry/tmp/worldfoundry_unified_env.sh and pip install -e ".[distribution_metrics]"
2. worldfoundry-eval metric show <id> to resolve ids and aliases
3. Open the metric page below and copy the recipe for your id

Only built-in existing-results metrics use worldfoundry-eval evaluate; everything else is Python (compute_* or scorer classes).

Metric imports are lazy: checkpoints are required only when a metric loads a feature extractor, multimodal scorer, or API judge. Default caches follow Hugging Face / Torch / torchvision layout:

export WORLDFOUNDRY_HFD_ROOT=${HF_HOME:-$HOME/.cache/huggingface}
export WORLDFOUNDRY_T2V_METRICS_CACHE_DIR=$WORLDFOUNDRY_HFD_ROOT
export WORLDFOUNDRY_CKPT_DIR=$HOME/.cache/worldfoundry/checkpoints

Metric pages42 ids · click to expand

Pages

Related pages

• Evaluation — run contract, evaluate, and scorecard outputs.
• Benchmark Hub — benchmark-specific metric assets and evaluator requirements.
• Local asset preparation — checkpoint staging including FVD I3D weights.