Editing & layout

Editing-specific metrics for semantic preservation, instruction following, and layout consistency.

Jump to a metric7 ids · click to expand

semsr (↑)

When: Semantic Shift Rate for trigger/origin/target image triplets.

Reference: Semantic Shift Rate: normalized CLIP semantic shift for trigger vs origin images. Paper: arXiv:2402.07562 Repo: WorldFoundry (paper reimplementation)

Assets: CLIP backbone weights via scorer / HF cache when computing from images.

from worldfoundry.evaluation.tasks.metrics import compute_semsr_from_images

result = compute_semsr_from_images(
    image_ust="trigger.png",
    image_ori="origin.png",
    image_tar="target.png",
    semantic_text="a photo of a cat",
)
print(result["semsr"])

Returns: Dict with semsr, sem_shift, and CLIP similarities.

irs (↑)

When: Image Realism Score calibrated against real-image statistics.

Reference: Image Realism Score: pentagon area over five calibrated image statistics. Paper: arXiv:2309.14756 Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint — image statistics computed from inputs.

from PIL import Image
from worldfoundry.evaluation.tasks.metrics import compute_irs_with_reference

real_paths = ["real/1.jpg", "real/2.jpg"]
test_paths = ["gen/1.jpg", "gen/2.jpg"]
real_images = [Image.open(p) for p in real_paths]
test_images = [Image.open(p) for p in test_paths]

out = compute_irs_with_reference(test_images, real_images)
print(out["irs_mean"], out["irs_scores"])

Returns: Dict with per-image IRS and fitted reference means.

cas (↑)

When: Train classifier on synthetic images; evaluate Top-k on real labels.

Reference: Classification Accuracy Score: train on synthetic, Top-k accuracy on real labels. Paper: arXiv:1905.10887 Repo: WorldFoundry (paper reimplementation)

Assets: No fixed checkpoint — trains a lightweight classifier on synthetic features supplied by the API.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import (
    train_classifier_and_compute_cas,
    compute_cas_from_predictions,
)

out = train_classifier_and_compute_cas(
    synthetic_images=syn_x,
    synthetic_labels=syn_y,
    real_images=real_x,
    real_labels=real_y,
    num_classes=10,
    device="cuda",
)
print(out["cas_top1"], out.get("cas_top5"))

# or with existing predictions
print(compute_cas_from_predictions(real_y, pred_topk, topk=(1, 5)))

Returns: Dict with cas, cas_top1, optional cas_top5.

manipulation_direction (↑)

When: CLIP change-vector alignment for image editing evaluation.

Reference: Manipulation Direction (MD): cosine similarity of CLIP image/text change vectors. Paper: Sensors 2023 Repo: WorldFoundry (paper reimplementation)

Assets: CLIP image/text encoders via HF or scorer cache.

from worldfoundry.evaluation.tasks.metrics import compute_manipulation_direction_from_pairs

md = compute_manipulation_direction_from_pairs(
    image_input="before.png",
    image_manipulated="after.png",
    text_original="a red car",
    text_replaced="a blue car",
)
print(md)

Returns: Cosine similarity in [-1, 1]; higher = edit follows text direction.

vs_similarity (↑)

When: HDGAN visual–semantic similarity from paired embeddings.

Reference: HDGAN visual-semantic similarity between paired image/text embeddings. Paper: HDGAN Repo: WorldFoundry (paper reimplementation)

Assets: HDGAN embedding inputs supplied by the API; no bundled standalone checkpoint.

import numpy as np
from worldfoundry.evaluation.tasks.metrics import compute_vs_similarity

image_embeds = np.load("img_emb.npy")  # (N, D)
text_embeds = np.load("txt_emb.npy")   # (N, D)
print(compute_vs_similarity(image_embeds, text_embeds, paired=True))

Returns: Float VS similarity for paired batches.

quality_loss (↓)

When: CLIPScore × text-presence probability (prompt optimization metric).

Reference: Quality Loss: CLIPScore multiplied by text/character presence probability (PC). Paper: IEEE Access 2023.3348778 Repo: WorldFoundry (paper reimplementation)

Assets: CLIPScore-style CLIP weights via WORLDFOUNDRY_T2V_METRICS_CACHE_DIR / HF cache.

from worldfoundry.evaluation.tasks.metrics import compute_quality_loss_for_pair

out = compute_quality_loss_for_pair(
    image="sample.png",
    prompt="HELLO WORLD",
    has_text=True,
)
print(out)  # clip_score, text_presence_probability, quality_loss

Returns: Dict; lower quality_loss with missing text is expected behavior.

object_wise_consistency (↑)

When: Location-aware T2I: match guidance boxes to detector outputs.

Reference: Object-wise consistency: max IoU + success rate R_suc vs guidance boxes (IoU>0.5). Paper: arXiv:2304.13427 Repo: WorldFoundry (paper reimplementation)

Assets: No model checkpoint — guidance boxes and detector outputs only.

from worldfoundry.evaluation.tasks.metrics import compute_object_wise_consistency

guidance = [([10, 20, 100, 200], "cat"), ([120, 30, 220, 180], "dog")]
detections = [([12, 22, 98, 198], "cat"), ([125, 35, 215, 175], "dog")]

print(compute_object_wise_consistency(guidance, detections, iou_threshold=0.5))

Returns: Dict with mean IoU and success rate object_wise_success_rate.