Perceptual pairwise

Pairwise similarity between reference and generated images (condition consistency, reconstruction quality).

Jump to a metric5 ids · click to expand

Metric principles and computation

This section describes the principle and computation contract of each registry metric. The goal is to make every metric understandable without requiring users to inspect the implementation first.

Notation used below:

• $X = \{x_i\}$ denotes reference samples.
• $Y = \{y_i\}$ denotes generated samples.
• $f(\cdot)$ denotes a feature extractor.
• $\mu_X, \Sigma_X$ and $\mu_Y, \Sigma_Y$ denote empirical mean and covariance in feature space.
• $\uparrow$ means higher is better.
• $\downarrow$ means lower is better.

For benchmark-specific aggregation, filtering, or official leaderboard normalization, document the protocol on the corresponding benchmark page instead of redefining the reusable metric here.

Pairwise perceptual metrics

lpips / ssim / ms_ssim / psnr (mixed)

lpips

Principle: LPIPS measures learned perceptual distance between two images. The method feeds both images through a pretrained deep network, extracts intermediate features, normalizes them, computes layer-wise feature distances, and combines them with learned weights. Lower values indicate stronger perceptual similarity. LPIPS is appropriate for paired image comparison, reconstruction, editing, and condition-preservation tasks; it is not a set-level distribution metric.

ssim

Principle: SSIM measures structural similarity between two aligned images. It compares local luminance, contrast, and structure statistics. A common simplified form is:

SSIM(x, y) =
((2 μ_x μ_y + C1)(2 σ_xy + C2)) /
((μ_x² + μ_y² + C1)(σ_x² + σ_y² + C2))

Higher values indicate stronger structural similarity. SSIM assumes a meaningful paired reference and can be less appropriate when many visually valid outputs are possible.

ms_ssim

Principle: MS-SSIM extends SSIM across multiple image scales. It repeatedly downsamples the images, computes contrast and structure comparisons at multiple resolutions, and combines them with scale weights. Higher values indicate stronger multi-scale structural similarity. This is useful for restoration and compression tasks where both fine and coarse structure matter.

psnr

Principle: PSNR is a pixel-error reconstruction metric derived from mean squared error. Given image dynamic range MAX_I and mean squared error MSE, it computes:

PSNR = 10 log10(MAX_I² / MSE)

Higher values indicate lower pixel-level error. PSNR is fast and standard for compression or deterministic reconstruction, but it often correlates poorly with human perceptual quality on generative tasks.

When: Pairwise reconstruction / consistency on aligned image pairs.

Reference:

• lpips: Learned Perceptual Image Patch Similarity (LPIPS). Paper: Zhang et al. Repo: richzhang/PerceptualSimilarity
• ssim: Structural Similarity Index. Paper: Wang et al. Repo: WorldFoundry (PIQ/torchmetrics)
• ms_ssim: Multi-scale SSIM. Paper: Wang et al. Repo: WorldFoundry (PIQ/torchmetrics)
• psnr: Peak Signal-to-Noise Ratio. Repo: WorldFoundry (PIQ/torchmetrics)

Assets:

• lpips: VGG16 and LPIPS / DISTS / PieAPP auxiliary weights via torchvision and torch hub release URLs.
• ssim: No model checkpoint — aligned image arrays only.
• ms_ssim: No model checkpoint — aligned image arrays only.

import numpy as np
from PIL import Image
from worldfoundry.evaluation.tasks.metrics import (
    compute_lpips,
    compute_ssim,
    compute_ms_ssim,
    compute_psnr,
    compute_perceptual_bundle,
)

ref = np.array(Image.open("ref.png").convert("RGB"))
gen = np.array(Image.open("gen.png").convert("RGB"))

print("lpips", compute_lpips(ref, gen))          # lower better
print("ssim", compute_ssim(ref, gen))            # higher better
print("ms_ssim", compute_ms_ssim(ref, gen))
print("psnr", compute_psnr(ref, gen))
print("bundle", compute_perceptual_bundle(ref, gen))

Returns: Float per metric; LPIPS lower, SSIM/MS-SSIM/PSNR higher is better.

dino_similarity / dreamsim / fsim / cpbd (mixed)

dino_similarity

Principle: DINO similarity compares paired images in a self-supervised DINO or DINOv2 feature space. The usual computation extracts image embeddings and computes cosine similarity or another normalized feature similarity:

DINO-Sim(ref, gen) = cos(DINO(ref), DINO(gen))

Higher values indicate stronger representation-level similarity. Because DINO features often capture semantic and object-centric structure, this metric can be more meaningful than raw pixel metrics for semantic consistency.

dreamsim

Principle: DreamSim is a learned perceptual similarity metric trained to align with human similarity judgments over image triplets. It combines or builds on strong visual representations and fine-tunes them for holistic perceptual similarity. The computation embeds both images and returns a learned distance. Lower values indicate stronger perceived similarity. It is useful when differences in object identity, layout, pose, and semantic content matter.

fsim

Principle: FSIM compares two aligned images using low-level feature similarity, especially phase congruency and gradient magnitude. Phase congruency captures perceptually significant local structure, and gradient magnitude captures contrast-related detail. The final score pools local similarity maps, often weighted by feature saliency. Higher values indicate stronger full-reference feature similarity.

cpbd

Principle: CPBD is a no-reference blur or sharpness metric based on the cumulative probability of blur detection. It detects edges, estimates blur width and local contrast, models the probability that blur is perceptually detectable, and pools these probabilities into a scalar sharpness score. Higher values indicate sharper images. CPBD does not require a reference image, but it only measures blur/sharpness, not realism or semantic correctness.

When: Perceptual similarity or no-reference sharpness on numpy images.

Reference:

• dino_similarity: Cosine similarity of DINO/DINOv2 features. Paper: Caron et al. Repo: facebookresearch/dino
• dreamsim: Perceptual distance from DreamSim ensemble. Paper: Fu et al. Repo: ssundaram21/dreamsim
• fsim: Feature Similarity Index for color images. Paper: Zhang et al. Repo: WorldFoundry (PIQ/torchmetrics)
• cpbd: Cumulative Probability of Blur Detection sharpness. Paper: Narvekar & Karam Repo: WorldFoundry (PIQ/torchmetrics)

Assets:

• dino_similarity: facebook/dinov2-base or facebook/dino-vitb16 via base-model capability, then HF from_pretrained. Override with WORLDFOUNDRY_DINOV2_BASE_MODEL_DIR / WORLDFOUNDRY_DINO_VITB16_MODEL_DIR.
• dreamsim: DreamSim release ZIPs from ssundaram21/dreamsim; pass cache_dir=... to compute_dreamsim(...).
• fsim: No model checkpoint — aligned image arrays only.

import numpy as np
from PIL import Image
from worldfoundry.evaluation.tasks.metrics import (
    compute_dino_similarity,
    compute_dreamsim,
    compute_fsim,
    compute_cpbd,
)

ref = np.array(Image.open("ref.png").convert("RGB"))
gen = np.array(Image.open("gen.png").convert("RGB"))

print("dino", compute_dino_similarity(ref, gen))
print("dreamsim", compute_dreamsim(ref, gen))
print("fsim", compute_fsim(ref, gen))
print("cpbd", compute_cpbd(gen))  # single image, no reference

Returns: Float scores; DreamSim lower = more similar.

mask_accuracy (↑)

Principle: mask_accuracy compares predicted masks against reference masks. Pixel accuracy computes the fraction of correctly matched mask pixels, while mask IoU computes overlap divided by union:

IoU(mask_ref, mask_pred) =
|mask_ref ∩ mask_pred| / |mask_ref ∪ mask_pred|

Higher values indicate better mask agreement. The benchmark protocol should specify thresholding, label mapping, ignored regions, and whether it reports pixel accuracy, IoU, or both.

When: Segmentation mask accuracy / IoU between prediction and GT.

Reference: Pixel-wise mask accuracy and IoU between predicted and GT masks. Repo: z-x-yang/Segment-and-Track-Anything (protocol)

Assets: No model checkpoint — predicted and ground-truth masks only.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import compute_mask_accuracy, compute_mask_iou

pred = np.load("pred_mask.npy")  # HxW binary or label map
gt = np.load("gt_mask.npy")
print(compute_mask_accuracy(pred, gt))
print(compute_mask_iou(pred, gt))

Returns: Accuracy and IoU floats in [0, 1].

object_detection (↑)

Principle: object_detection measures whether expected objects are detected in generated outputs. The typical computation runs an object detector, maps expected classes to detector classes, applies confidence and IoU thresholds, and reports a success rate:

success_rate = matched_expected_objects / expected_objects

Higher values indicate better object presence or localization. The result depends heavily on detector choice, class vocabulary, confidence thresholds, and prompt-to-object parsing.

When: WorldScore-style phrase-matching detection success rate.

Reference: Phrase-matching object detection success rate (WorldScore-style). Paper: WorldScore (ICCV 2025) Repo: haoyi-duan/WorldScore

Assets: No model checkpoint — phrase/box inputs per WorldScore-style protocol.

from worldfoundry.evaluation.tasks.metrics import compute_object_detection_success_rate

result = compute_object_detection_success_rate(
    generated_dir="/data/generated",
    prompts_path="/data/prompts.json",
)
print(result)

Returns: Success rate dict; see object_detection/ wrapper for input schema.

lqs (↑)

Principle: Layout Quality Score evaluates generated layouts against ground-truth or expected layouts. The original layout-to-image formulation considers both absolute bounding-box distribution errors and mutual spatial relationships between boxes. Higher values in WorldFoundry should indicate better layout agreement if the wrapper exposes a quality score rather than an error score. The protocol must define the layout schema, coordinate system, object matching rule, and handling of missing or extra boxes.

When: Layout Quality Score over predicted vs GT bounding-box layouts.

Reference: Layout Quality Score: LR + LP + LC + AC over bounding-box layouts. Paper: Yang et al. Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint — predicted and ground-truth layout boxes only.

from worldfoundry.evaluation.tasks.metrics import compute_lqs

gt_layout = [{"label": "person", "bbox": [10, 20, 110, 220]}]
pred_layout = [{"label": "person", "bbox": [12, 18, 108, 218]}]
print(compute_lqs(gt_layout, pred_layout))

Returns: Dict with LR/LP/LC/AC components and combined LQS.