Distribution metrics

Image and video set-level metrics — FID, FVD, JEDi, CMMD, Clean-FID, and feature-array distances.

Last updated: Jul 11, 2026

Set-level metrics comparing reference and generated image or video collections.

Jump to a metric23 ids · click to expand

Metric principles and computation

This section describes the principle and computation contract of each registry metric. The goal is to make every metric understandable without requiring users to inspect the implementation first.

Notation used below:

$X = \{x_i\}$ denotes reference samples.
$Y = \{y_i\}$ denotes generated samples.
$f(\cdot)$ denotes a feature extractor.
$\mu_X, \Sigma_X$ and $\mu_Y, \Sigma_Y$ denote empirical mean and covariance in feature space.
$\uparrow$ means higher is better.
$\downarrow$ means lower is better.

For benchmark-specific aggregation, filtering, or official leaderboard normalization, document the protocol on the corresponding benchmark page instead of redefining the reusable metric here.

Image distribution metrics

`fid` (↓)

Principle: Fréchet Inception Distance compares reference and generated image distributions in a neural feature space. The standard computation extracts Inception features, fits a Gaussian to each set, and computes the 2-Wasserstein distance between the two Gaussians:

\mathrm{FID}(X, Y) = \lVert \mu_X - \mu_Y \rVert^2 + \mathrm{Tr}\!\left(\Sigma_X + \Sigma_Y - 2\left(\Sigma_X \Sigma_Y\right)^{1/2}\right)

Lower values indicate closer generated and reference feature distributions. FID is useful for set-level image generation evaluation, but it is sensitive to feature extractor choice, preprocessing, sample count, and reference statistics. WorldFoundry aliases such as clip_fid, swav_fid, scene_fid, object_crop_fid, and fid_vid keep the same Fréchet-distance idea but change the feature extractor or the sample construction.

When: Compare two folders of images (reference vs generated).

Reference: Fréchet distance between Inception (or CLIP/SwAV) feature distributions. Paper: Heusel et al. Repo: toshas/torch-fidelity

Assets: Torch-fidelity InceptionV3 / CLIP / SwAV weights via torch.hub and torchvision caches. Offline: feature_extractor_weights_path or staged Torch cache.

from worldfoundry.evaluation.tasks.metrics import compute_fid

ref_dir = "/data/coco_val"
gen_dir = "/data/my_model_samples"

fid = compute_fid(ref_dir, gen_dir)
print(fid)  # lower is better

clip_fid = compute_fid(ref_dir, gen_dir, feature_extractor="clip-vit-b-32")
swav_fid = compute_fid(ref_dir, gen_dir, feature_extractor="swav-resnet50")

Returns: Float FID; lower is better.

`scene_fid` (↓)

\mathrm{FID}(X, Y) = \lVert \mu_X - \mu_Y \rVert^2 + \mathrm{Tr}\!\left(\Sigma_X + \Sigma_Y - 2\left(\Sigma_X \Sigma_Y\right)^{1/2}\right)

When: SceneFID with object crops from bbox annotations.

from worldfoundry.evaluation.tasks.metrics import compute_scene_fid

score = compute_scene_fid(
    "/data/ref",
    "/data/gen",
    reference_bboxes_json="/data/ref_bboxes.json",
)
print(score)

Returns: Scene-level FID float.

`inception_score` (↑)

Principle: Inception Score estimates whether generated images are both classifiable and diverse. The original formulation feeds generated images into an Inception classifier, obtains p(y | x), compares each conditional label distribution with the marginal label distribution p(y), and exponentiates the expected KL divergence:

\mathrm{IS} = \exp\!\left(\mathbb{E}_x\left[\mathrm{KL}\!\left(p(y|x)\,\|\,p(y)\right)\right]\right)

A high score means individual images produce confident predictions while the whole generated set covers many predicted classes. It does not use reference images and can be misleading for domains that do not match the classifier's training distribution.

When: Measure generated-set quality/diversity via Inception classifier.

Reference: Inception classifier confidence and label entropy over generated images. Paper: Salimans et al. Repo: toshas/torch-fidelity (vendored)

Assets: Torch-fidelity InceptionV3 via torch.hub / torchvision cache.

from worldfoundry.evaluation.tasks.metrics import compute_inception_score

result = compute_inception_score("/data/generated_only")
print(result)  # dict with mean/std IS

Returns: Dict with Inception Score mean/std; higher mean is better.

`kid` (↓)

Principle: Kernel Inception Distance compares reference and generated image sets using squared Maximum Mean Discrepancy over Inception features. A common implementation uses a polynomial kernel:

\mathrm{KID}(X, Y) = \mathrm{MMD}^2\!\left(f(X), f(Y)\right)

Unlike FID, KID does not fit Gaussian distributions and its estimator can be unbiased. Lower scores indicate closer feature distributions. It is often useful when sample counts are smaller or when users want an MMD-based complement to FID.

When: Kernel Inception Distance between two image sets.

Reference: Kernel Inception Distance (MMD with polynomial kernel on Inception features). Paper: Bińkowski et al. Repo: toshas/torch-fidelity

Assets: Torch-fidelity Inception features via torch.hub / torchvision cache.

from worldfoundry.evaluation.tasks.metrics import compute_kid

kid = compute_kid("/data/ref", "/data/gen")
print(kid["kernel_inception_distance_mean"])

Returns: Dict of KID statistics; lower mean is better.

`precision_recall` (↑)

Principle: Distribution precision and recall separate two failure modes that a single scalar such as FID can hide. Precision measures how much of the generated distribution lies on or near the real data manifold, while recall measures how much of the real data manifold is covered by generated samples. Implementations typically embed real and generated samples, estimate manifolds or support regions, then compute the fraction of samples falling inside the other distribution's support. Higher precision means more realistic samples; higher recall means better coverage.

When: Manifold precision/recall for coverage and fidelity.

Reference: Manifold precision and recall for generative model coverage/fidelity. Paper: Kynkäänniemi et al. Repo: toshas/torch-fidelity

Assets: Torch-fidelity Inception features via torch.hub / torchvision cache.

from worldfoundry.evaluation.tasks.metrics import compute_precision_recall

pr = compute_precision_recall("/data/ref", "/data/gen")
print(pr)

Returns: Dict with precision, recall, and related keys.

`improved_precision_recall` (↑)

Principle: Improved Precision and Recall uses non-parametric manifold estimates in feature space. For each real and generated sample, the method estimates a local radius using nearest neighbors, then checks cross-membership between real and generated manifolds. Precision is the fraction of generated samples that lie inside the real manifold; recall is the fraction of real samples covered by the generated manifold. Higher values are better. This metric is useful for distinguishing fidelity failure from coverage failure.

When: α-precision, β-recall, realism score (k-NN radii).

Reference: α-precision, β-recall, and realism score via k-NN radii in feature space. Paper: Kynkä et al. Repo: kynkaat/improved-precision-and-recall-metric

Assets: Torchvision VGG16 feature extractor (TORCH_HOME / torch cache).

from worldfoundry.evaluation.tasks.metrics import (
    compute_improved_precision_recall,
    compute_realism_score,
)

ipr = compute_improved_precision_recall("/data/ref", "/data/gen")
realism = compute_realism_score("/data/ref", "/data/gen")
print(ipr, realism)

Returns: Dict of IPR metrics; higher precision/recall/realism is better.

`fwd / cmmd / clean_fid / mind / trend` (↓)

`fwd`

Principle: Fréchet Wavelet Distance is a domain-agnostic alternative to neural-feature Fréchet metrics. Instead of extracting Inception features, it projects images into wavelet-packet coefficient space and computes a Fréchet distance between the resulting coefficient distributions. The core computation is:

\mathrm{FWD}(X, Y) = d_{\text{Fréchet}}\!\left(W_p(X), W_p(Y)\right)

where Wp is the wavelet packet transform. Lower values indicate closer distributions. Because the representation is frequency-based rather than classifier-based, FWD can be useful for domains where ImageNet-pretrained features are poorly matched.

`cmmd`

Principle: CMMD, or CLIP Maximum Mean Discrepancy, compares image sets in CLIP embedding space using MMD, usually with a Gaussian RBF kernel. The computation is:

\mathrm{CMMD}(X, Y) = \mathrm{MMD}^2\!\left(\mathrm{CLIP}_{\mathrm{img}}(X), \mathrm{CLIP}_{\mathrm{img}}(Y)\right)

Lower values indicate that generated and reference image distributions are closer in CLIP's semantic feature space. CMMD was proposed as an alternative to FID for modern image generation, especially text-to-image settings, because it avoids the Gaussian assumption used by FID and uses richer CLIP features.

`clean_fid`

Principle: Clean-FID keeps the FID principle but standardizes preprocessing details such as resizing, quantization, and reference statistics. The computation is still a Fréchet distance between feature Gaussians, but the implementation aims to reduce score drift caused by inconsistent image-processing libraries. Use clean_fid when you need FID values that are more comparable across papers and codebases. Do not mix Clean-FID and non-Clean-FID values in the same leaderboard unless explicitly documented.

`mind`

Principle: Monge Inception Distance compares generated and reference distributions using sliced Wasserstein distance over Inception-style features. Instead of estimating high-dimensional Gaussian moments as FID does, MIND projects features onto many one-dimensional directions, sorts projected samples, computes one-dimensional optimal transport distances, and averages them. Lower values indicate closer distributions. The motivation is better sample efficiency and robustness compared with moment-matching metrics such as FID.

`trend`

Principle: trend / trend_jsd compares reference and generated trend distributions, usually through a divergence such as Jensen-Shannon divergence. Given two categorical or binned distributions P and Q, the common computation is:

\mathrm{JSD}(P, Q) = \tfrac{1}{2}\,\mathrm{KL}(P \parallel M) + \tfrac{1}{2}\,\mathrm{KL}(Q \parallel M), \quad M = \tfrac{1}{2}(P + Q)

Lower values indicate that generated outputs preserve the reference trend distribution more closely. In WorldFoundry, this is best used when the benchmark defines explicit attributes, categories, temporal bins, or other trend statistics to compare.

When: Distribution distance on image folders (different feature backends).

Reference:

fwd: Fréchet Wavelet Distance between image distributions. Paper: Karras et al. Repo: WorldFoundry (vendored pytorchfwd)
cmmd: CLIP Maximum Mean Discrepancy between image sets. Paper: Jayasumana et al. Repo: WorldFoundry (adapted from CMMD repo)
clean_fid: Clean-FID with improved Inception preprocessing. Paper: Parmar et al. Repo: GaParmar/clean-fid
mind: Monge Inception Distance between image sets. Paper: John et al. Repo: WorldFoundry (torch-fidelity integration)
trend: Truncated entropy / trend divergence on feature histograms. Paper: Han et al. Repo: WorldFoundry (paper reimplementation)

Assets:

fwd: Torch-fidelity / wavelet backend weights via vendored distribution helpers and torch cache.
cmmd: CLIP vision encoder openai/clip-vit-large-patch14-336 in Hugging Face transformers cache (WORLDFOUNDRY_HFD_ROOT).
clean_fid: Clean-FID Inception network and optional reference statistics in the cleanfid runtime cache.
mind: Torch-fidelity feature extractors via torch.hub / torchvision cache.
trend: No model checkpoint when caller supplies the required statistics/features.

from worldfoundry.evaluation.tasks.metrics import (
    compute_fwd,
    compute_cmmd,
    compute_clean_fid,
    compute_mind,
    compute_trend,
)

ref, gen = "/data/ref", "/data/gen"
print("fwd", compute_fwd(ref, gen))
print("cmmd", compute_cmmd(ref, gen))
print("clean_fid", compute_clean_fid(ref, gen, dataset_name="cifar10", mode="clean"))
print("mind", compute_mind(ref, gen))
print("trend", compute_trend(ref, gen))

Returns: Float or dict per metric; lower is better for these ids.

`ppl` (↓)

Principle: Perceptual Path Length measures smoothness of a generator's latent space. The metric samples two latent codes, interpolates between them, takes a small step along the interpolation path, generates the corresponding images, and measures perceptual distance between the two nearby generated images. A simplified form is:

\mathrm{PPL} = \mathbb{E}\left[\frac{d\!\left(G(\mathrm{interp}(z_1, z_2, t)),\, G(\mathrm{interp}(z_1, z_2, t + \varepsilon))\right)}{\varepsilon^2}\right]

Lower values indicate smoother perceptual changes under latent interpolation. PPL is meaningful only when the model exposes a valid latent-space interpolation interface.

When: StyleGAN latent perceptual path length (smoothness).

Reference: Perceptual path length in StyleGAN latent space (smoothness). Paper: Karras et al. Repo: toshas/torch-fidelity

Assets: Torch-fidelity latent / perceptual path weights via torch.hub / torchvision cache.

from worldfoundry.evaluation.tasks.metrics import compute_ppl

ppl = compute_ppl("/path/to/generated_or_latent_inputs")
print(ppl)

Returns: PPL statistic; lower = smoother generator.

`vendi_score / rke / rnd` (↑)

`vendi_score`

Principle: Vendi Score measures diversity from a similarity matrix. Given samples and a similarity function, construct a normalized similarity matrix K, compute its eigenvalues λ, and return the exponential Shannon entropy:

\mathrm{Vendi}(X) = \exp\!\left(-\sum_i \lambda_i \log \lambda_i\right)

The score can be interpreted as an effective number of distinct samples under the chosen similarity function. Higher values indicate greater diversity. Because the metric is reference-free, it should be paired with a fidelity metric when evaluating generation quality.

`rke`

Principle: Rényi Kernel Entropy evaluates diversity through entropy in a kernel-induced feature space. Given a positive semidefinite kernel matrix, the method computes a Rényi-entropy-style functional over its eigenvalues. Higher scores indicate more effective modes or greater diversity under the chosen kernel. Because the metric is reference-free, it should be reported with the kernel, feature representation, and any normalization used.

`rnd`

Principle: Random Network Distillation score measures diversity or novelty using prediction error against random-network features. The method passes samples through a fixed random target network and a predictor or feature-matching path; greater prediction error or feature spread can indicate greater diversity depending on the implementation. Higher values are generally interpreted as more diversity or novelty. It is best treated as a diagnostic metric and should be paired with fidelity metrics.

When: Diversity from feature matrices or image folders.

Reference:

vendi_score: Vendi score (diversity) from feature matrix or image embeddings. Paper: Friedman & Dieng Repo: WorldFoundry (adapted)
rke: Rényi kernel entropy mode count on feature embeddings (diversity). Paper: Jalali et al. (NeurIPS 2023) Repo: mjalali/renyi-kernel-entropy
rnd: Random Network Distillation novelty score on feature vectors. Paper: Burda et al. Repo: WorldFoundry (paper reimplementation)

Assets:

vendi_score: Image helper: torchvision InceptionV3. Text helper: roberta-base and princeton-nlp/unsup-simcse-roberta-base under WORLDFOUNDRY_HFD_ROOT. Feature-array mode needs no extractor.
rke: No model checkpoint when caller supplies feature embeddings.
rnd: No model checkpoint when caller supplies feature embeddings.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import (
    compute_vendi_score,
    compute_rke,
    compute_rrke,
    compute_rnd,
    compute_rnd_from_images,
)

features = np.load("embeddings.npy")  # shape (N, D)
print("vendi", compute_vendi_score(features))
print("rke", compute_rke(features))
print("rrke", compute_rrke(ref_features, gen_features))
print("rnd", compute_rnd(features))
print("rnd_images", compute_rnd_from_images("/data/generated"))

Returns: Float diversity scores; higher vendi/rke/rnd = more diverse.

`rarity_score` (↑)

Principle: Rarity Score measures how uncommon each generated image is relative to a reference distribution. The method embeds images, uses nearest-neighbor distances to model how sparse the local region is, and assigns higher rarity to samples that lie in less crowded regions of feature space. A set-level score is usually an average or distribution summary over individual rarity scores. Higher rarity can indicate novelty, but not necessarily quality: rare samples may also be unrealistic or out-of-distribution.

When: How rare generated samples are vs a real feature bank.

Reference: Mean distance to k-nearest real neighbors (higher = rarer samples). Paper: Han et al. (ICLR 2023) Repo: hichoe95/Rarity-Score

Assets: Feature bank from real images; uses the metric's configured feature extractor cache when built from media.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import compute_mean_rarity_score

real_feats = np.load("real.npy")
gen_feats = np.load("gen.npy")
print(compute_mean_rarity_score(real_feats, gen_feats, k=5))

Returns: Mean rarity; higher = rarer samples.

`fld` (↓)

Principle: Feature Likelihood Divergence evaluates fidelity, diversity, and novelty using density estimation in feature space. The method fits or estimates feature likelihoods for train, test, and generated samples, then measures whether generated samples look like unseen test data rather than memorized training data. Lower values indicate better generalization under the FLD formulation. This metric is useful when overfitting or memorization is a concern and when train/test/generated feature sets are available.

When: Feature Likelihood Divergence on precomputed feature arrays.

Reference: Feature Likelihood Divergence between real and generated feature distributions. Paper: Richter & Rhodes (ICML 2022) Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint when caller supplies train/test/generated feature arrays.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import compute_fld

train = np.load("train_feats.npy")
test = np.load("test_feats.npy")
gen = np.load("gen_feats.npy")
print(compute_fld(train, test, gen))

Returns: Float FLD; lower is better.

`multimodal_mid` (↑)

Principle: Mutual Information Divergence is a multimodal generation metric based on CLIP-style image-text features. It estimates cross-modal mutual information behavior between paired modalities and uses a negative Gaussian cross-mutual-information formulation to score alignment. Higher WorldFoundry scores should indicate stronger multimodal consistency when the implementation exposes MID as a positive alignment score. Use it for paired multimodal generation such as text-to-image or image captioning, and document the exact feature model.

When: Mutual Information Divergence on paired image-text embeddings.

Reference: Mutual Information Divergence comparing joint image-text embedding distributions. Paper: Dhaka et al. (NeurIPS 2021) Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint when caller supplies paired feature arrays.

from worldfoundry.evaluation.tasks.metrics import compute_multimodal_mid

print(compute_multimodal_mid(image_feats, text_feats))

Returns: Float MID; higher is better.

`fjd` (↓)

Principle: Fréchet Joint Distance evaluates conditional generation by comparing joint distributions of generated content and conditioning variables. Instead of computing Fréchet distance only on image features, FJD builds joint embeddings, for example image features concatenated or combined with class, text, mask, or layout-condition features, then computes a Fréchet distance between real and generated joint distributions. Lower values indicate better joint fidelity, condition consistency, and intra-condition diversity.

When: Fréchet Joint Distance on paired image-text joint embeddings.

Reference: Fréchet Joint Distance over paired image-text embeddings. Paper: Lee et al. (ICCV 2023) Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint when caller supplies joint embeddings.

from worldfoundry.evaluation.tasks.metrics import compute_fjd_from_joint_embeddings

print(compute_fjd_from_joint_embeddings(real_joint, gen_joint))

Returns: Float FJD; lower is better.

`crosslid` (↓)

Principle: CrossLID evaluates how well generated and real manifolds coincide using local intrinsic dimensionality. The method estimates the local intrinsic dimensionality of real samples with respect to neighborhoods formed by generated samples. Lower CrossLID indicates that generated samples better match the local structure of the real data manifold. It is useful for diagnosing mode collapse and distribution mismatch beyond global moment statistics.

When: Cross local intrinsic dimensionality for diversity assessment.

Reference: Cross local intrinsic dimensionality for diversity assessment. Paper: Barua et al. (ICLR 2019) Repo: sukarnabarua/CrossLID

Assets: No model checkpoint when caller supplies feature arrays.

from worldfoundry.evaluation.tasks.metrics import compute_crosslid

print(compute_crosslid(features, k=5))

Returns: Float CrossLID; lower indicates less diversity.

`cfid` (↓)

Principle: Conditional FID generalizes FID to conditional distributions. Instead of comparing only unconditional generated and real image distributions, CFID measures discrepancies between conditional distributions, making it more sensitive to cases where outputs are realistic but unrelated to inputs. The closed-form version follows a conditional Gaussian/Wasserstein setup. Lower values indicate better conditional fidelity.

When: Conditional FID from paired conditional/unconditional embeddings.

Reference: Conditional FID from paired conditional/unconditional embeddings. Paper: Benny & Wolf (NeurIPS 2020) Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint when caller supplies conditional feature arrays.

from worldfoundry.evaluation.tasks.metrics import compute_cfid

print(compute_cfid(ref_pairs, gen_pairs))

Returns: Float cFID; lower is better.

`ssd` (↓)

Principle: Semantic Similarity Distance is a CLIP-based metric for text-image consistency. It compares generated images and text prompts from a distributional semantic-alignment perspective, combining text-image similarity and semantic variation differences. Lower values indicate better text-image semantic consistency. Use it for text-to-image benchmarks where prompt faithfulness matters more than pixel-level similarity.

When: Semantic Similarity Distance between text and image embeddings.

Reference: Semantic Similarity Distance between text and image embeddings. Paper: Tschannen et al. (ICLR 2018) Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint when caller supplies text/image embeddings.

from worldfoundry.evaluation.tasks.metrics import compute_ssd

print(compute_ssd(text_embeds, image_embeds))

Returns: Float SSD; lower is better.

`linear_separability` (↑)

Principle: Linear separability measures disentanglement in a latent or attribute space. The StyleGAN formulation asks whether latent-space samples corresponding to a binary attribute can be separated by a linear hyperplane. In WorldFoundry, the public helper computes a score from a confusion matrix, so the effective computation depends on classifier outcomes:

\text{linear\_separability} = \mathrm{score}(\mathbf{C})

Higher values indicate that the attribute or factor is more linearly separable. This metric is meaningful only when the upstream classifier, labels, and attribute definitions are stable.

When: StyleGAN latent linear separability from an SVM confusion matrix.

Reference: SVM linear separability score from StyleGAN latent confusion matrix. Paper: Karras et al. Repo: NVlabs/stylegan

Assets: No model checkpoint — pass the confusion matrix / separability inputs required by the API.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import compute_linear_separability

confusion = np.load("svm_confusion.npy")
print(compute_linear_separability(confusion))

Returns: Separability score; higher = more disentangled latents.

`fdd` (↓)

Principle: Fréchet Denoised Distance replaces the Inception feature extractor in FID with a denoising-autoencoder representation. The metric encodes real and generated images with a DAE, fits Gaussian statistics in the denoised latent space, and computes a Fréchet distance. Lower values indicate that generated images are closer to reference images in a structure-biased denoised feature space. FDD is especially relevant when structural plausibility matters more than low-level visual artifacts.

When: Fréchet Denoised Distance on DAE latent features.

Reference: Fréchet distance on denoising-autoencoder latent features (structural plausibility). Paper: Fan et al. (ECCV 2024) Repo: jiajie96/FDD_pytorch

Assets: WORLDFOUNDRY_FDD_DAE_CKPT or auto-download via gdown (Google Drive id 1j7MVFWYfRNZLQ3uChe7TGQG8L1Oaf9Gt).

from worldfoundry.evaluation.tasks.metrics import compute_fdd

# export WORLDFOUNDRY_FDD_DAE_CKPT=/path/to/dae.pt
score = compute_fdd("/data/ref", "/data/gen")
print(score)

Returns: Float FDD; lower = more structurally plausible.

`cis` (↑)

Principle: Conditional Inception Score generalizes Inception Score to class-conditional generation. It decomposes quality and diversity by conditioning class, using within-class and between-class components derived from classifier predictions. Higher values indicate better class-conditional quality and diversity. Use CIS only when generated samples have known class conditions and classifier predictions are meaningful for those classes.

When: Conditional Inception Score from class-probability batches.

Reference: Conditional Inception Score: BCIS × WCIS from class-bucketed softmax. Paper: Benny et al. (IJCV 2020) Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint when caller supplies class-probability batches.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import compute_cis, compute_cis_from_predictions

# probs: list of (N, num_classes) arrays, one bucket per condition
print(compute_cis(class_probs))
print(compute_cis_from_predictions(predictions, num_classes=1000))

Returns: Dict with BCIS, WCIS, and combined CIS.

`attribute_sad / attribute_pad` (↓)

`attribute_sad`

Principle: Attribute SaD is an interpretable attribute-distribution metric. Given high-level concept scores for real and generated images and a text attribute list, it measures how far the generated attribute distribution deviates from the real attribute distribution. Lower values indicate better preservation of marginal attribute statistics. This metric is useful when benchmark users need to know which semantic attributes are mismatched rather than only receiving a scalar realism score.

`attribute_pad`

Principle: Attribute PaD is the paired or relationship-sensitive counterpart to SaD. It evaluates whether generated samples preserve attribute relationships or paired attribute behavior relative to the reference set. Lower values indicate better attribute-pair consistency. Use it when attribute co-occurrence or implausible attribute relationships are important evaluation targets.

When: SadPaD attribute divergence from precomputed HCS histograms.

Reference:

attribute_sad: KL / JS / W1 divergence between attribute histograms (attribute SAD). Paper: Ramesh et al. Repo: WorldFoundry (SadPaD reimplementation)
attribute_pad: Attribute-aware Padé approximant divergence for text attributes. Paper: Ramesh et al. Repo: WorldFoundry (SadPaD reimplementation)

Assets:

attribute_sad: No model checkpoint — pass precomputed HCS histograms (hcs_real, hcs_gen, text_list).
attribute_pad: No model checkpoint — pass precomputed HCS histograms (hcs_real, hcs_gen, text_list).

from worldfoundry.evaluation.tasks.metrics import compute_attribute_sad, compute_attribute_pad

text_list = ["smiling", "young", "eyeglasses"]
sad = compute_attribute_sad(hcs_real, hcs_gen, text_list)
pad = compute_attribute_pad(hcs_real, hcs_gen, text_list)
print(sad, pad)

Returns: Dict with per-attribute divergences; lower = closer to real distribution.

Video distribution metrics

`fvd` (↓)

Principle: Fréchet Video Distance extends the FID idea to videos. It extracts spatiotemporal video features, commonly from an I3D network, fits Gaussian statistics to reference and generated video features, and computes the same Fréchet Gaussian distance used by FID. Lower values indicate closer reference and generated video distributions. FVD captures both appearance and temporal information better than frame-only metrics, but later work shows that it can still be biased toward frame quality and may be insensitive to some temporal corruptions.

When: Fréchet Video Distance between real and generated video sets.

Reference: Fréchet distance between I3D video feature distributions. Paper: Unterthiner et al. Repo: WorldFoundry (adapted from StyleGAN-V / MiraBench paths)

Assets: Inception I3D i3d_pretrained_400.pt via explicit arg → WORLDFOUNDRY_FVD_I3D_CKPT → WORLDFOUNDRY_MIRABENCH_FVD_I3D_CKPT → WORLDFOUNDRY_CKPT_DIR.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import (
    compute_fvd_from_numpy,
    compute_fvd_from_frame_dirs,
)

# uint8 arrays shaped (N, T, H, W, C)
real = np.load("real_videos.npy")
gen = np.load("gen_videos.npy")
print(compute_fvd_from_numpy(real, gen, device="cuda"))

print(
    compute_fvd_from_frame_dirs(
        reference_frame_dirs=["/data/ref_frames/vid0"],
        generated_frame_dirs=["/data/gen_frames/vid0"],
        device="cuda",
    )
)

Returns: Float FVD; set WORLDFOUNDRY_FVD_I3D_CKPT if needed.

`fvmd` (↓)

Principle: Fréchet Video Motion Distance focuses on motion consistency. It extracts explicit motion features, for example through keypoint tracking, fits reference and generated motion-feature distributions, and computes a Fréchet distance between them. Lower values indicate closer motion distributions. FVMD is useful when a video model produces sharp frames but unrealistic temporal dynamics.

When: Motion-feature Fréchet distance between video directories.

Reference: Fréchet distance on PIPs++ keypoint velocity/acceleration motion features. Paper: Liu et al. Repo: ljh0v0/FVMD-frechet-video-motion-distance

Assets: PIPs2 tracker pips2_weights.pth from FVMD release via torch hub cache.

from worldfoundry.evaluation.tasks.metrics import compute_fvmd

score = compute_fvmd("/data/ref_videos", "/data/gen_videos")
print(score)

Returns: Float FVMD; lower = closer motion distribution.

`jedi` (↓)

Principle: JEDi, or JEPA Embedding Distance, evaluates video generation using V-JEPA-style video representations and an MMD distance, commonly with a polynomial kernel. The computation extracts video embeddings, then compares generated and reference embedding sets through MMD:

\mathrm{JEDi}(X, Y) = \mathrm{MMD}^2\!\left(\mathrm{VJEPA}(X), \mathrm{VJEPA}(Y)\right)

Lower values indicate closer video distributions. JEDi was proposed to address limitations of FVD, including non-Gaussian feature assumptions, sample inefficiency, and weak temporal sensitivity.

When: Video distribution quality via V-JEPA feature MMD.

Reference: V-JEPA embedding MMD with polynomial kernel (video distribution quality). Paper: Luo et al. Repo: oooolga/JEDi

Assets: Live path: V-JEPA dir (WORLDFOUNDRY_JEDI_MODEL_DIR / WORLDFOUNDRY_JEDI_VJEPA_DIR / WORLDFOUNDRY_VJEPA_MODEL_DIR, optional WORLDFOUNDRY_JEDI_CONFIG_PATH). Feature mode: precomputed train/test arrays only (WORLDFOUNDRY_JEDI_FEATURE_PATH).

import numpy as np
from worldfoundry.evaluation.tasks.metrics import (
    compute_jedi_from_features,
    compute_mock_jedi,
)

train_feats = np.load("jedi_train.npy")
test_feats = np.load("jedi_test.npy")
print(compute_jedi_from_features(train_feats, test_feats))

# smoke test without checkpoints
print(compute_mock_jedi())

Returns: Float JEDi MMD; configure WORLDFOUNDRY_JEDI_* for live backend.

Run multiple torch-fidelity metrics at once

from worldfoundry.evaluation.tasks.metrics import compute_distribution_metrics

scores = compute_distribution_metrics(
    "/data/ref", "/data/gen", metrics=("fid", "kid", "prc")
)
print(scores)

Distribution metrics

Metric principles and computation

Image distribution metrics

fid (↓)

scene_fid (↓)

inception_score (↑)

kid (↓)

precision_recall (↑)

improved_precision_recall (↑)

fwd / cmmd / clean_fid / mind / trend (↓)

fwd

cmmd

clean_fid

mind

trend

ppl (↓)

vendi_score / rke / rnd (↑)

vendi_score

rke

rnd

rarity_score (↑)

fld (↓)

multimodal_mid (↑)

fjd (↓)

crosslid (↓)

cfid (↓)

ssd (↓)

linear_separability (↑)

fdd (↓)

cis (↑)

attribute_sad / attribute_pad (↓)

attribute_sad

attribute_pad

Video distribution metrics

fvd (↓)

fvmd (↓)

jedi (↓)

Run multiple torch-fidelity metrics at once

`fid` (↓)

`scene_fid` (↓)

`inception_score` (↑)

`kid` (↓)

`precision_recall` (↑)

`improved_precision_recall` (↑)

`fwd / cmmd / clean_fid / mind / trend` (↓)

`fwd`

`cmmd`

`clean_fid`

`mind`

`trend`

`ppl` (↓)

`vendi_score / rke / rnd` (↑)

`vendi_score`

`rke`

`rnd`

`rarity_score` (↑)

`fld` (↓)

`multimodal_mid` (↑)

`fjd` (↓)

`crosslid` (↓)

`cfid` (↓)

`ssd` (↓)

`linear_separability` (↑)

`fdd` (↓)

`cis` (↑)

`attribute_sad / attribute_pad` (↓)

`attribute_sad`

`attribute_pad`

`fvd` (↓)

`fvmd` (↓)

`jedi` (↓)