GenAI-Bench

Integrated

Pairwise human-preference GenAI-Bench evaluation with WorldFoundry's checked-in runner, assets, commands, metrics, and current gaps.

Markdown Source

Last updated: Jul 11, 2026

On this page

What It Measures Prepare Data And Assets Input And Output Layout Public CLI Direct Runner Metrics Limitations And Gaps

What It Measures

GenAI-Bench evaluates whether a multimodal judge agrees with human pairwise preferences for generated content. The local GenAI-Bench README describes three preference tracks: image generation, image editing, and video generation. Each example asks the judge to choose between two generated artifacts with labels such as A>B, B>A, A=B=Good, and A=B=Bad.

WorldFoundry uses the checked-in runner at worldfoundry/evaluation/tasks/execution/runners/genai_bench/run_genai_bench_official_runner.py. Do not make a separate copy of the official repo for normal WorldFoundry use. Treat the upstream paper and official README as protocol references; the runnable WorldFoundry path is already in this tree.

Prepare Data And Assets

Use one of these input styles:

Imported preference results: JSONL, JSON, CSV, or TSV rows with human_label or human_preference, prediction or judge_prediction, and task.
In-tree scorer rows: a preference-pair JSONL file with prompt, human_label, optional left_artifact / right_artifact, and task.
Generated artifacts: a directory containing the candidate images or videos referenced by the preference pairs.

Checked-in assets:

worldfoundry/data/benchmarks/assets/genai-bench/metadata.json
worldfoundry/data/benchmarks/assets/genai-bench/preference_pairs.fixture.jsonl
worldfoundry/data/benchmarks/assets/genai-bench/sample_results.jsonl

Optional downloads are fine when you want the HF data or scorer weights for your own run:

hf download TIGER-Lab/GenAI-Bench \
  --repo-type dataset \
  --local-dir /path/to/datasets/GenAI-Bench

hf download zhiqiulin/clip-flant5-xxl \
  --local-dir /path/to/checkpoints/clip-flant5-xxl

For VQAScore-style scoring, set the scorer backend and checkpoint cache:

export WORLDFOUNDRY_GENAI_BENCH_SCORER_BACKEND=vqascore
export WORLDFOUNDRY_T2V_METRICS_MODEL=clip-flant5-xxl
export WORLDFOUNDRY_T2V_METRICS_CACHE_DIR=/path/to/checkpoints
export WORLDFOUNDRY_T2V_METRICS_DEVICE=cuda

Input And Output Layout

Example generated artifact layout:

/path/to/genai/generated_artifacts/
  video_left_001.mp4
  video_right_001.mp4
  image_left_002.png
  image_right_002.png

Example preference-result JSONL:

{"task":"video_generation","human_label":"A>B","prediction":"A>B","pair_id":"video-001"}
{"task":"image_generation","human_label":"B>A","judge_prediction":"A>B","pair_id":"image-002"}

WorldFoundry output layout:

tmp/genai-bench/official-validation/
  scorecard.json
  raw_metric_table.jsonl
  per_sample_scores.jsonl
  runner_runtime_report.json
  specialized_normalizer_stdout.log
  specialized_normalizer_stderr.log

Direct runner output also includes scorecard.json, raw_metric_table.jsonl, and per_sample_scores.jsonl. When the scorer is launched, generated upstream rows are written under upstream/.

Public CLI

The catalog-supported public command for this page is result import with official-validation:

cd /path/to/WorldFoundry

worldfoundry-eval zoo benchmark-run \
  --benchmark-id genai-bench \
  --mode official-validation \
  --official-results-path /path/to/genai_preference_results.jsonl \
  --generated-artifact-dir /path/to/genai/generated_artifacts \
  --output-dir tmp/genai-bench/official-validation \
  --json

The public catalog entry does not publish a full official-run command for GenAI-Bench. Use the direct runner below when you need the checked-in scorer path.

Direct Runner

Import an existing preference-result file:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/genai_bench/run_genai_bench_official_runner.py \
  --official-results-path /path/to/genai_preference_results.jsonl \
  --generated-artifact-dir /path/to/genai/generated_artifacts \
  --output-dir tmp/genai-bench/direct-import \
  --json

Run the checked-in scorer over a preference-pair file:

cd /path/to/WorldFoundry

export WORLDFOUNDRY_GENAI_BENCH_SCORER_BACKEND=vqascore
export WORLDFOUNDRY_T2V_METRICS_MODEL=clip-flant5-xxl
export WORLDFOUNDRY_T2V_METRICS_CACHE_DIR=/path/to/checkpoints

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/genai_bench/run_genai_bench_official_runner.py \
  --run-official \
  --preference-pairs-path /path/to/preference_pairs.jsonl \
  --generated-artifact-dir /path/to/genai/generated_artifacts \
  --output-dir tmp/genai-bench/direct-run \
  --json

For local wiring without GPU scoring, set WORLDFOUNDRY_GENAI_BENCH_SCORER_BACKEND=mock. Those scores are deterministic placeholders and are not leaderboard evidence.

Metrics

Metric ID	Meaning
`pairwise_accuracy`	Fraction of all imported pair rows where the judge prediction equals the human preference label.
`image_generation_preference_accuracy`	Pairwise accuracy on rows whose task resolves to image generation.
`image_editing_preference_accuracy`	Pairwise accuracy on rows whose task resolves to image editing.
`video_preference_accuracy`	Pairwise accuracy on rows whose task resolves to video generation.
`genai_bench_average`	Mean of available task-level preference accuracies; primary metric.

All metrics are normalized to [0, 1], and higher is better.

Limitations And Gaps

WorldFoundry's GenAI-Bench page is wired for pairwise preference rows. It does not reproduce the full upstream model-submission workflow.
Full leaderboard parity still requires the official human-preference split, real generated artifacts, and a real judge or VQAScore backend.
The mock backend is only for local wiring.
The VQAScore backend requires working video decoding, the chosen checkpoint, and enough GPU memory for the selected model.

Back to Benchmark Hub