AIGCBench

What It Measures

AIGCBench evaluates image-to-video generation quality across four families from the official paper and project assets: control-video alignment, motion effects, temporal consistency, and video quality. The official dataset combines WebVid validation videos, LAION-aesthetics image-text pairs, and an "Ours" generated image-text set. WorldFoundry uses the in-tree runner to materialize prompt records and to import metric rows produced by the AIGCBench scoring pipeline.

The in-tree surface is not a standalone reimplementation of every CLIP, DOVER, optical-flow, and reference-video calculation. It expects either a caller-provided metric result file or a WorldFoundry scorer output produced by the local evaluation stack.

Data And Artifacts

WorldFoundry looks for the AIGCBench prompt suite in this order:

• --prompt-manifest <path>
• WORLDFOUNDRY_AIGCBENCH_PROMPT_MANIFEST
• bundled worldfoundry/data/benchmarks/assets/aigcbench/prompt_suite.json
• a downloaded HF dataset root from WORLDFOUNDRY_AIGCBENCH_DATASET_ROOT
• the default local HF cache entry for stevenfan/AIGCBench_v1.0

Useful dataset files from the HF dataset are:

• webvid_eval_1000.txt for WebVid prompt/reference-video ids.
• Laion-aesthetics_select_samples.txt for LAION image prompts.
• t2i_625/, t2i_aspect_ratio_625/, or AIGCBench/t2i_625/ for image-conditioned prompt assets.

Put candidate videos in one flat directory and name each file by prompt id:

<generated_videos>/
  <prompt_id>.mp4
  <prompt_id>.webm

Supported video suffixes are .mp4, .mov, .mkv, .webm, and .avi. Coverage is computed by matching each video stem to the prompt ids in the resolved manifest.

Dependencies

For full metric production, the scoring side needs the same assets implied by AIGCBench: reference images/videos where applicable, CLIP ViT-L/14, DOVER, and optical-flow tooling. The WorldFoundry runner itself only needs Python package access to the repository and a metric result file when using result import.

HF dataset downloads are supported. External repository checkouts are not required for these docs because the WorldFoundry integration lives in-tree.

Supported Commands

Set the generated artifact directory once:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/aigcbench/generated_videos

Import an existing AIGCBench metric report through the public CLI:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id aigcbench \
  --mode official-validation \
  --official-results-path /path/to/aigcbench_results.csv \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --prompt-manifest /path/to/prompt_suite.json \
  --output-dir tmp/aigcbench/import \
  --json

Use the direct in-tree runner when you need its native arguments:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/aigcbench/run_aigcbench_official_runner.py \
  --official-results-path /path/to/aigcbench_results.csv \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --prompt-manifest /path/to/prompt_suite.json \
  --output-dir tmp/aigcbench/import \
  --json

If your local WorldFoundry metric stack is configured to create AIGCBench results from generated videos, call the same runner with --run-official:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/aigcbench/run_aigcbench_official_runner.py \
  --run-official \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --prompt-manifest /path/to/prompt_suite.json \
  --output-dir tmp/aigcbench/run \
  --json

Import Paths

• Runner: worldfoundry.evaluation.tasks.execution.runners.aigcbench.run_aigcbench_official_runner
• Metrics: worldfoundry.evaluation.tasks.execution.runners.aigcbench.aigcbench_metrics
• Prompts and artifact layout: worldfoundry.evaluation.tasks.execution.runners.aigcbench.aigcbench_prompts
• Runtime adapter: worldfoundry.evaluation.tasks.execution.runners.aigcbench.aigcbench_runtime

Result File Shape

The importer accepts JSON or CSV-style result rows. It can read summary rows with metric_id and score or per-sample rows with metric columns. Metric aliases such as mse, ssim, clip_per, and AIGCBench display names are normalized to the canonical ids below.

Outputs written under --output-dir include:

• scorecard.json: benchmark status, coverage, metric table, and output paths.
• raw_metric_table.jsonl: one row per declared metric.
• per_sample_scores.jsonl: per-sample subset when the input result file contains sample ids.

Metrics

Primary metric: aigcbench_average.

← Benchmark Hub