# GenAI-Bench (/docs/evaluation/benchmark-hub/genai-bench) ## What It Measures [#what-it-measures] GenAI-Bench evaluates whether a multimodal judge agrees with human pairwise preferences for generated content. The local GenAI-Bench README describes three preference tracks: image generation, image editing, and video generation. Each example asks the judge to choose between two generated artifacts with labels such as `A>B`, `B>A`, `A=B=Good`, and `A=B=Bad`. WorldFoundry uses the checked-in runner at `worldfoundry/evaluation/tasks/execution/runners/genai_bench/run_genai_bench_official_runner.py`. Do not make a separate copy of the official repo for normal WorldFoundry use. Treat the upstream paper and official README as protocol references; the runnable WorldFoundry path is already in this tree. ## Prepare Data And Assets [#prepare-data-and-assets] Use one of these input styles: * Imported preference results: JSONL, JSON, CSV, or TSV rows with `human_label` or `human_preference`, `prediction` or `judge_prediction`, and `task`. * In-tree scorer rows: a preference-pair JSONL file with `prompt`, `human_label`, optional `left_artifact` / `right_artifact`, and `task`. * Generated artifacts: a directory containing the candidate images or videos referenced by the preference pairs. Checked-in assets: * `worldfoundry/data/benchmarks/assets/genai-bench/metadata.json` * `worldfoundry/data/benchmarks/assets/genai-bench/preference_pairs.fixture.jsonl` * `worldfoundry/data/benchmarks/assets/genai-bench/sample_results.jsonl` Optional downloads are fine when you want the HF data or scorer weights for your own run: ```bash hf download TIGER-Lab/GenAI-Bench \ --repo-type dataset \ --local-dir /path/to/datasets/GenAI-Bench hf download zhiqiulin/clip-flant5-xxl \ --local-dir /path/to/checkpoints/clip-flant5-xxl ``` For VQAScore-style scoring, set the scorer backend and checkpoint cache: ```bash export WORLDFOUNDRY_GENAI_BENCH_SCORER_BACKEND=vqascore export WORLDFOUNDRY_T2V_METRICS_MODEL=clip-flant5-xxl export WORLDFOUNDRY_T2V_METRICS_CACHE_DIR=/path/to/checkpoints export WORLDFOUNDRY_T2V_METRICS_DEVICE=cuda ``` ## Input And Output Layout [#input-and-output-layout] Example generated artifact layout: ```text /path/to/genai/generated_artifacts/ video_left_001.mp4 video_right_001.mp4 image_left_002.png image_right_002.png ``` Example preference-result JSONL: ```jsonl {"task":"video_generation","human_label":"A>B","prediction":"A>B","pair_id":"video-001"} {"task":"image_generation","human_label":"B>A","judge_prediction":"A>B","pair_id":"image-002"} ``` WorldFoundry output layout: ```text tmp/genai-bench/official-validation/ scorecard.json raw_metric_table.jsonl per_sample_scores.jsonl runner_runtime_report.json specialized_normalizer_stdout.log specialized_normalizer_stderr.log ``` Direct runner output also includes `scorecard.json`, `raw_metric_table.jsonl`, and `per_sample_scores.jsonl`. When the scorer is launched, generated upstream rows are written under `upstream/`. ## Public CLI [#public-cli] The catalog-supported public command for this page is result import with `official-validation`: ```bash cd /path/to/WorldFoundry worldfoundry-eval zoo benchmark-run \ --benchmark-id genai-bench \ --mode official-validation \ --official-results-path /path/to/genai_preference_results.jsonl \ --generated-artifact-dir /path/to/genai/generated_artifacts \ --output-dir tmp/genai-bench/official-validation \ --json ``` The public catalog entry does not publish a full `official-run` command for GenAI-Bench. Use the direct runner below when you need the checked-in scorer path. ## Direct Runner [#direct-runner] Import an existing preference-result file: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/genai_bench/run_genai_bench_official_runner.py \ --official-results-path /path/to/genai_preference_results.jsonl \ --generated-artifact-dir /path/to/genai/generated_artifacts \ --output-dir tmp/genai-bench/direct-import \ --json ``` Run the checked-in scorer over a preference-pair file: ```bash cd /path/to/WorldFoundry export WORLDFOUNDRY_GENAI_BENCH_SCORER_BACKEND=vqascore export WORLDFOUNDRY_T2V_METRICS_MODEL=clip-flant5-xxl export WORLDFOUNDRY_T2V_METRICS_CACHE_DIR=/path/to/checkpoints PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/genai_bench/run_genai_bench_official_runner.py \ --run-official \ --preference-pairs-path /path/to/preference_pairs.jsonl \ --generated-artifact-dir /path/to/genai/generated_artifacts \ --output-dir tmp/genai-bench/direct-run \ --json ``` For local wiring without GPU scoring, set `WORLDFOUNDRY_GENAI_BENCH_SCORER_BACKEND=mock`. Those scores are deterministic placeholders and are not leaderboard evidence. ## Metrics [#metrics] | Metric ID | Meaning | | -------------------------------------- | ------------------------------------------------------------------------------------------------ | | `pairwise_accuracy` | Fraction of all imported pair rows where the judge prediction equals the human preference label. | | `image_generation_preference_accuracy` | Pairwise accuracy on rows whose task resolves to image generation. | | `image_editing_preference_accuracy` | Pairwise accuracy on rows whose task resolves to image editing. | | `video_preference_accuracy` | Pairwise accuracy on rows whose task resolves to video generation. | | `genai_bench_average` | Mean of available task-level preference accuracies; primary metric. | All metrics are normalized to `[0, 1]`, and higher is better. ## Limitations And Gaps [#limitations-and-gaps] * WorldFoundry's GenAI-Bench page is wired for pairwise preference rows. It does not reproduce the full upstream model-submission workflow. * Full leaderboard parity still requires the official human-preference split, real generated artifacts, and a real judge or VQAScore backend. * The `mock` backend is only for local wiring. * The VQAScore backend requires working video decoding, the chosen checkpoint, and enough GPU memory for the selected model. [Back to Benchmark Hub](/docs/evaluation/benchmark-hub)