EvalCrafter

What It Measures

EvalCrafter is a 700-prompt text-to-video benchmark covering visual quality, content alignment, motion quality, and temporal consistency. The original benchmark reports 17 objective metrics plus human feedback; WorldFoundry exposes those objective scores through the in-tree EvalCrafter runner and final_result.txt parser.

Use EvalCrafter when you need a broad diagnostic stack for generated videos rather than a physics-only or world-model-only benchmark.

Assets And Data

WorldFoundry keeps the prompt suite and runtime in-tree:

The optional public dataset asset is RaphaelLiu/EvalCrafter_T2V_Dataset, tracked in WorldFoundry as evalcrafter_dataset_assets.

Generated Artifacts

The generated-video directory must be the inner directory containing exactly the videos for the 700 prompts:

/path/to/evalcrafter/videos/
  0000.mp4
  0001.mp4
  0002.mp4
  ...
  0699.mp4

Do not pass a parent experiment directory unless the 0000.mp4 through 0699.mp4 files are directly inside it. The runner checks the prompt ids from prompt700.txt against the direct .mp4 filenames.

Dependencies

The WorldFoundry runner no longer launches EvalCrafter's benchmark-local shell pipeline. It consumes final_result.txt produced by the WorldFoundry metric stack or by an existing official EvalCrafter run, then writes WorldFoundry scorecards and metric rows.

For full metric production, use the dedicated environment:

bash scripts/setup/model_env_install.sh --model evalcrafter

Place the metric assets under the EvalCrafter checkpoint root or set WORLDFOUNDRY_EVALCRAFTER_CHECKPOINTS_DIR. The full metric stack can need DOVER, BLIP2, SDXL, Inception, VideoMAE, FlowNet2, VGG-Face, DeAOT/AOT, OCR, detection, segmentation, and action-recognition dependencies. Existing final_result.txt import does not need all heavyweight metric packages.

Useful variables:

Commands

Set the generated video directory:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/evalcrafter/videos
export WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR=tmp/evalcrafter/run

Score a generated-video directory when final_result.txt is already available through WORLDFOUNDRY_EVALCRAFTER_RESULTS_PATH or inside the generated directory:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/evalcrafter/run_evalcrafter_official_runner.py \
  --evalcrafter-root worldfoundry/evaluation/tasks/execution/runners/evalcrafter/runtime/evalcrafter \
  --videos-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" \
  --run-official \
  --json

Import an existing result directory or result file:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/evalcrafter/run_evalcrafter_official_runner.py \
  --evalcrafter-root worldfoundry/evaluation/tasks/execution/runners/evalcrafter/runtime/evalcrafter \
  --results-dir /path/to/final_result.txt \
  --videos-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/evalcrafter/import \
  --json

The public wrapper can also launch the integrated runner when inputs are passed through the benchmark manifest:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id evalcrafter \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --official-results-path /path/to/final_result.txt \
  --output-dir tmp/evalcrafter/zoo-run \
  --json

Import Paths

For Python use:

from worldfoundry.evaluation.tasks.execution.runners.evalcrafter.evalcrafter_metrics import (
    load_upstream_results,
    normalized_score,
)
from worldfoundry.evaluation.tasks.execution.runners.evalcrafter.evalcrafter_official_impl import (
    normalize_evalcrafter_results,
    run_official_evalcrafter,
)

Outputs

The output directory contains:

Primary metrics are visual_quality, text_video_alignment, motion_quality, temporal_consistency, and evalcrafter_total. Component metrics include vqa_aesthetic, vqa_technical, inception_score, clip_temp_score, warping_error, face_consistency_score, action_score, motion_ac_score, flow_score, clip_score, blip_bleu, sd_score, detection_score, color_score, count_score, ocr_error, and celebrity_id_error.

← Benchmark Hub