Scorers & quality

Multimodal scorers and quality/reward metrics. Invoke through Python get_score_model(...) or the compute_* helpers below.

Jump to a metric7 ids · click to expand

Metric principles and computation

This section describes the principle and computation contract of each registry metric. The goal is to make every metric understandable without requiring users to inspect the implementation first.

Notation used below:

• $X = \{x_i\}$ denotes reference samples.
• $Y = \{y_i\}$ denotes generated samples.
• $f(\cdot)$ denotes a feature extractor.
• $\mu_X, \Sigma_X$ and $\mu_Y, \Sigma_Y$ denote empirical mean and covariance in feature space.
• $\uparrow$ means higher is better.
• $\downarrow$ means lower is better.

For benchmark-specific aggregation, filtering, or official leaderboard normalization, document the protocol on the corresponding benchmark page instead of redefining the reusable metric here.

Multimodal scorer metrics

clip_score (↑)

Principle: CLIPScore is a reference-free image-text compatibility metric. It embeds the image and text using CLIP and computes a scaled cosine similarity between the two embeddings. Higher values indicate stronger image-text alignment. It is useful for captioning and text-conditioned generation, but it can miss fine-grained counting, negation, spatial relations, and factual context.

When: Score image–text alignment with CLIPScore.

Reference: Cosine similarity between CLIP image and text embeddings. Paper: CLIPScore Repo: linzhiqiu/t2v_metrics

Assets: OpenCLIP / BLIP2 / PickScore / InternVideo2 / UMT weights via WORLDFOUNDRY_T2V_METRICS_CACHE_DIR → WORLDFOUNDRY_HFD_ROOT → package hf_cache.

from worldfoundry.evaluation.tasks.metrics import CLIPScore, list_all_clipscore_models

print(list_all_clipscore_models()[:5])

clip = CLIPScore(model="openai:ViT-L/14", device="cuda")
score = clip.forward(images="sample.png", texts="a red sports car on a wet street")
print(float(score.item()))

Returns: Float tensor; higher = better alignment.

vqa_score (↑)

Principle: VQAScore evaluates text-to-visual alignment by converting a prompt into a yes/no visual question and scoring the probability that a VQA model answers "yes." A common template is:

Question: Does this image show "{text}"?
Score: P(answer = "yes" | image, question)

Higher values indicate stronger prompt satisfaction. This is useful for compositional prompts because a VQA model can check object, attribute, relation, counting, and logic requirements more directly than a single embedding similarity.

When: Probability a VQA model answers yes to a text probe about the image.

Reference: Probability a VQA model answers "yes" to "Does this figure show {text}?" Paper: VQAScore (ECCV 2024) Repo: linzhiqiu/t2v_metrics

Assets: Hugging Face / LAVIS / WorldFoundry base-model caches for local VQA models; API models need provider credentials (OPENAI_API_KEY, Gemini key, etc.).

from worldfoundry.evaluation.tasks.metrics import VQAScore

vqa = VQAScore(model="clip-flant5-xxl", device="cuda")
score = vqa.forward(
    images="sample.png",
    texts="Is there a dog wearing sunglasses in this image?",
)
print(float(score.item()))

Returns: Probability in [0, 1]; higher = better.

itm_score (↑)

Principle: ITMScore uses an image-text matching model to estimate whether an image and text belong together. The computation usually feeds the image-text pair into a multimodal transformer and reads the matching probability or logit. Higher values indicate stronger image-text consistency. ITMScore is model-dependent, so benchmark reports should document the underlying VLM, checkpoint, preprocessing, and whether the score is a probability, logit, or normalized scalar.

When: Image–text matching (ITM) for images or videos.

Reference: Image-text matching score from BLIP-2 / InternVideo2 ITM backbones. Paper: BLIP-2 Repo: linzhiqiu/t2v_metrics

Assets: Same cache order as CLIPScore. BLIP2/LAVIS URLs, InternVideo2 internvl_c_13b_224px, UMT b16_ptk710_f8_res224 or l16_ptk710_f8_res224.

from worldfoundry.evaluation.tasks.metrics import ITMScore

itm = ITMScore(model="blip2-itm", device="cuda")
score = itm.forward(images="sample.png", texts="a cat sitting on a windowsill")
print(float(score.item()))

Returns: Matching score tensor; higher = better match.

Quality & reward metrics

facescore (↑)

Principle: FaceScore is a learned face-quality metric for human or face generation. It was introduced to address the mismatch between generic image metrics and human judgments of face realism and quality. The metric fine-tunes a reward/scoring model on face-preference pairs, then outputs a scalar face-quality score for generated images. Higher values indicate better face quality according to the learned preference model. Use it only when face content is expected and the benchmark protocol allows face-specific scoring.

When: Face quality reward on a single portrait image.

Reference: Reward model scoring facial realism and aesthetics in generated humans. Paper: FaceScore Repo: OPPO-Mente-Lab/FaceScore

Assets: Hugging Face repo AIGCer-OPPO/FaceScore (default ~/.cache/FaceScore); pass model_name as a local checkpoint path for offline runs.

from worldfoundry.evaluation.tasks.metrics import compute_facescore

score = compute_facescore("outputs/face.png", device="cuda")
print(score)

Returns: Float reward; higher = better face quality.

artscore (↑)

Principle: ArtScore estimates how strongly an image resembles authentic artwork rather than a photograph or non-artistic image. The proposed method trains a neural scorer using pseudo-annotated images with varying degrees of "artness," then predicts an artness level for arbitrary images. Higher values indicate stronger artistic style or art-likeness. This metric is useful for art-generation benchmarks, but it should not be interpreted as a universal aesthetic-quality score.

When: Estimate artness (fine-art vs photorealistic).

Reference: Neural network estimating artness (fine-art vs photorealistic appearance). Paper: ArtScore Repo: jchen175/ArtScore

Assets: Explicit ArtScore .pt checkpoint plus torchvision ResNet/VGG backbones. Stage under WORLDFOUNDRY_CKPT_DIR and pass checkpoint_path=....

from PIL import Image
from worldfoundry.evaluation.tasks.metrics import load_artscore_model

model = load_artscore_model(device="cuda")
image = Image.open("sample.png").convert("RGB")
score = model(image)
print(float(score))

Returns: Artness score; higher = more art-like.

facesim_cur (↑)

Principle: facesim_cur measures identity similarity between a reference face and a generated face. The usual principle is to detect and align faces, extract face-recognition embeddings, and compute cosine similarity:

FaceSim(ref, gen) = cos(e_ref, e_gen)

WorldFoundry's facesim_cur path uses the OpenS2V face-similarity stack with InsightFace and CurricularFace-style weights. Higher values indicate stronger identity preservation. The metric is undefined or unreliable when faces cannot be detected, are heavily occluded, or are outside the supported face-recognition domain.

When: Face cosine similarity between reference and generated portraits (OpenS2V).

Reference: Face cosine similarity via InsightFace detection + CurricularFace embeddings. Paper: OpenS2V-Nexus Repo: PKU-YuanGroup/OpenS2V-Nexus

Assets: WORLDFOUNDRY_OPENS2V_WEIGHT_DIR or BestWishYsh/OpenS2V-Weight with face_extractor/ and glint360k_curricular_face_r101_backbone.bin.

from worldfoundry.evaluation.tasks.metrics import compute_facesim_cur

score = compute_facesim_cur(
    ref_paths=["ref/face_a.png", "ref/face_b.png"],
    gen_paths=["gen/face_a.png", "gen/face_b.png"],
    device="cuda",
)
print(score)

Returns: Mean face similarity; set WORLDFOUNDRY_OPENS2V_WEIGHT_DIR first.

gme_score / nexus_score / natural_score (↑)

gme_score

Principle: GmeScore is an OpenS2V-style semantic relevance metric. It uses a vision-language model, such as GME-Qwen2-VL in the WorldFoundry implementation, to score how well generated visual content matches the text or subject condition. The computation is model-dependent, but conceptually it embeds or judges the generated artifact against the conditioning text and returns a scalar alignment score. Higher values indicate stronger text relevance or semantic consistency.

nexus_score

Principle: NexusScore is an OpenS2V metric designed for subject consistency. It combines subject-aware visual matching with object/region-aware components, such as detector features and multimodal embeddings. Conceptually, it checks whether the generated output preserves the reference subject rather than merely matching a broad prompt. Higher scores indicate better subject preservation. Because it is a composite metric, the exact detector, backbone, thresholds, and aggregation rule should be documented in the benchmark protocol.

natural_score

Principle: NaturalScore is an OpenS2V metric for perceived naturalness. The metric uses a learned or API-backed judge to estimate whether the generated output looks natural and plausible. Higher scores indicate better naturalness. Because the WorldFoundry implementation may depend on an external GPT API, reproducible benchmark reports should document the model version, prompt template, temperature or decoding settings, and any retry policy.

When: OpenS2V subject-to-video evaluation with folder layout + JSON manifest.

Reference:

• gme_score: GME-Qwen2-VL image-text alignment for subject-to-video evaluation. Paper: OpenS2V-Nexus Repo: PKU-YuanGroup/OpenS2V-Nexus
• nexus_score: YOLO-World + GME subject phrase-matching success score. Paper: OpenS2V-Nexus Repo: PKU-YuanGroup/OpenS2V-Nexus
• natural_score: GPT API naturalness rating for generated subject-to-video clips. Paper: OpenS2V-Nexus Repo: PKU-YuanGroup/OpenS2V-Nexus

Assets:

• gme_score: WORLDFOUNDRY_GME_MODEL_PATH or Alibaba-NLP/gme-Qwen2-VL-7B-Instruct/ Hugging Face snapshot.
• nexus_score: WORLDFOUNDRY_YOLO_WORLD_CKPT, WORLDFOUNDRY_YOLO_CLIP_MODEL (openai/clip-vit-base-patch32), and WORLDFOUNDRY_GME_MODEL_PATH for offline runs.
• natural_score: No local checkpoint — OpenAI-compatible judge API (OPENAI_API_KEY or api_key=...).

from worldfoundry.evaluation.tasks.metrics import (
    compute_gme_score,
    compute_nexus_score,
    compute_natural_score,
)

manifest = "opens2v/eval.json"  # keyed by video stem
videos = "opens2v/videos"
refs = "opens2v/ref_images"

gme = compute_gme_score(
    input_video_folder=videos,
    input_image_folder=refs,
    input_json_file=manifest,
    device="cuda",
)
nexus = compute_nexus_score(
    input_video_folder=videos,
    input_image_folder=refs,
    input_json_file=manifest,
    device="cuda",
)
natural = compute_natural_score(
    input_video_folder=videos,
    input_json_file=manifest,
)  # needs OPENAI_API_KEY
print(gme, nexus, natural)

Returns: Dict payloads per video; see OpenS2V eval JSON schema.

Manifest entry example: {"video_001": {"img_paths": [...], "class_label": [...]}}.