# EvalCrafter (/docs/evaluation/benchmark-hub/evalcrafter)



## What It Measures [#what-it-measures]

EvalCrafter is a 700-prompt text-to-video benchmark covering visual quality, content alignment, motion quality, and temporal consistency. The original benchmark reports 17 objective metrics plus human feedback; WorldFoundry exposes those objective scores through the in-tree EvalCrafter runner and `final_result.txt` parser.

Use EvalCrafter when you need a broad diagnostic stack for generated videos rather than a physics-only or world-model-only benchmark.

## Assets And Data [#assets-and-data]

WorldFoundry keeps the prompt suite and runtime in-tree:

| Asset        | Path                                                                                             |
| ------------ | ------------------------------------------------------------------------------------------------ |
| Task YAML    | `worldfoundry/data/benchmarks/tasks/external/evalcrafter.yaml`                                   |
| Prompt suite | `worldfoundry/data/benchmarks/assets/evalcrafter/prompt700.txt`                                  |
| Runner       | `worldfoundry/evaluation/tasks/execution/runners/evalcrafter/run_evalcrafter_official_runner.py` |
| Runtime root | `worldfoundry/evaluation/tasks/execution/runners/evalcrafter/runtime/evalcrafter`                |

The optional public dataset asset is `RaphaelLiu/EvalCrafter_T2V_Dataset`, tracked in WorldFoundry as `evalcrafter_dataset_assets`.

## Generated Artifacts [#generated-artifacts]

The generated-video directory must be the inner directory containing exactly the videos for the 700 prompts:

```text
/path/to/evalcrafter/videos/
  0000.mp4
  0001.mp4
  0002.mp4
  ...
  0699.mp4
```

Do not pass a parent experiment directory unless the `0000.mp4` through `0699.mp4` files are directly inside it. The runner checks the prompt ids from `prompt700.txt` against the direct `.mp4` filenames.

## Dependencies [#dependencies]

The WorldFoundry runner no longer launches EvalCrafter's benchmark-local shell pipeline. It consumes `final_result.txt` produced by the WorldFoundry metric stack or by an existing official EvalCrafter run, then writes WorldFoundry scorecards and metric rows.

For full metric production, use the dedicated environment:

```bash
bash scripts/setup/model_env_install.sh --model evalcrafter
```

Place the metric assets under the EvalCrafter checkpoint root or set `WORLDFOUNDRY_EVALCRAFTER_CHECKPOINTS_DIR`. The full metric stack can need DOVER, BLIP2, SDXL, Inception, VideoMAE, FlowNet2, VGG-Face, DeAOT/AOT, OCR, detection, segmentation, and action-recognition dependencies. Existing `final_result.txt` import does not need all heavyweight metric packages.

Useful variables:

| Variable                                   | Use                                                          |
| ------------------------------------------ | ------------------------------------------------------------ |
| `WORLDFOUNDRY_EVALCRAFTER_ROOT`            | Override the in-tree runtime root.                           |
| `WORLDFOUNDRY_EVALCRAFTER_RESULTS_PATH`    | Point to `final_result.txt` or a directory containing it.    |
| `WORLDFOUNDRY_EVALCRAFTER_PROMPT_MANIFEST` | Override `prompt700.txt`.                                    |
| `WORLDFOUNDRY_EVALCRAFTER_SCORER_BACKEND`  | `artifact` by default; reads an existing `final_result.txt`. |

## Commands [#commands]

Set the generated video directory:

```bash
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/evalcrafter/videos
export WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR=tmp/evalcrafter/run
```

Score a generated-video directory when `final_result.txt` is already available through `WORLDFOUNDRY_EVALCRAFTER_RESULTS_PATH` or inside the generated directory:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/evalcrafter/run_evalcrafter_official_runner.py \
  --evalcrafter-root worldfoundry/evaluation/tasks/execution/runners/evalcrafter/runtime/evalcrafter \
  --videos-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" \
  --run-official \
  --json
```

Import an existing result directory or result file:

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/evalcrafter/run_evalcrafter_official_runner.py \
  --evalcrafter-root worldfoundry/evaluation/tasks/execution/runners/evalcrafter/runtime/evalcrafter \
  --results-dir /path/to/final_result.txt \
  --videos-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/evalcrafter/import \
  --json
```

The public wrapper can also launch the integrated runner when inputs are passed through the benchmark manifest:

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id evalcrafter \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --official-results-path /path/to/final_result.txt \
  --output-dir tmp/evalcrafter/zoo-run \
  --json
```

## Import Paths [#import-paths]

For Python use:

```python
from worldfoundry.evaluation.tasks.execution.runners.evalcrafter.evalcrafter_metrics import (
    load_upstream_results,
    normalized_score,
)
from worldfoundry.evaluation.tasks.execution.runners.evalcrafter.evalcrafter_official_impl import (
    normalize_evalcrafter_results,
    run_official_evalcrafter,
)
```

## Outputs [#outputs]

The output directory contains:

| File                       | Meaning                                                                              |
| -------------------------- | ------------------------------------------------------------------------------------ |
| `scorecard.json`           | Coverage, eligibility, and all available EvalCrafter metrics.                        |
| `raw_metric_table.jsonl`   | One row per metric.                                                                  |
| `per_sample_metrics.jsonl` | Reserved for per-sample rows; currently empty for result-file import.                |
| `input_validation.json`    | Written by `--run-official` before scoring.                                          |
| `final_result.txt`         | Copied into the output directory when the artifact backend reads an external result. |

Primary metrics are `visual_quality`, `text_video_alignment`, `motion_quality`, `temporal_consistency`, and `evalcrafter_total`. Component metrics include `vqa_aesthetic`, `vqa_technical`, `inception_score`, `clip_temp_score`, `warping_error`, `face_consistency_score`, `action_score`, `motion_ac_score`, `flow_score`, `clip_score`, `blip_bleu`, `sd_score`, `detection_score`, `color_score`, `count_score`, `ocr_error`, and `celebrity_id_error`.

[← Benchmark Hub](/docs/evaluation/benchmark-hub)