FETV

Integrated

Fine-grained text-to-video evaluation with in-tree FETV-EVAL runtime, prompt assets, and runnable scoring commands.

Last updated: Jul 11, 2026

On this page

About Evaluation Protocol Data And Checkpoint Preparation Run Automatic Scoring Import Existing Results Outputs

About

FETV is a NeurIPS 2023 fine-grained open-domain benchmark for text-to-video generation. It evaluates models across prompt content, controllable attributes, prompt complexity, and temporal categories. The official evaluation combines human ratings for quality and alignment with automatic CLIPScore, BLIPScore, FID, and FVD metrics.

WorldFoundry integrates the runnable FETV-EVAL code in tree. The official runtime adapter is worldfoundry/evaluation/tasks/execution/runners/fetv/fetv_official_runtime.py, the runner is worldfoundry/evaluation/tasks/execution/runners/fetv/run_fetv_official_runner.py, and the reusable BLIP / StyleGAN-V code is under worldfoundry/base_models.

Official references:

FETV paper: NeurIPS 2023 proceedings
FETV benchmark data: lyx97/FETV
Generated videos and frames: lyx97/FETV_gen_videos
Official source references: llyx97/FETV, llyx97/FETV-EVAL

Evaluation Protocol

WorldFoundry bundles the official prompt assets:

worldfoundry/data/benchmarks/assets/fetv/fetv_data.json
worldfoundry/data/benchmarks/assets/fetv/sampled_prompts_for_fid_fvd/prompts_gen.json
worldfoundry/data/benchmarks/assets/fetv/sampled_prompts_for_fid_fvd/prompts_real.json

The core FETV prompt file contains 619 prompts. For CLIPScore and BLIPScore, the official evaluator expects generated videos converted into uniformly sampled frame folders:

/path/to/fetv/<model>/16frames_uniform/
  sent0/
    frame0.jpg
    ...
    frame15.jpg
  sent1/
  ...
  sent618/

For FID and FVD, prepare the larger generated/reference sets from sampled_prompts_for_fid_fvd using the same official FETV directory convention.

Primary metric: fetv_average.

Metric	Meaning
`static_quality`	Human static visual quality score.
`temporal_quality`	Human temporal quality score.
`overall_alignment`	Human overall text-video alignment.
`fine_grained_alignment`	Human prompt-aspect alignment.
`clip_score`	Automatic CLIPScore over sampled frames.
`blip_score`	Automatic BLIPScore over sampled frames.
`fid`	FID distribution distance; lower is better.
`fvd`	FVD distribution distance; lower is better.
`fetv_average`	Mean over available normalized FETV metrics.

Data And Checkpoint Preparation

Start from the WorldFoundry repository root:

cd /path/to/WorldFoundry
export WORLDFOUNDRY_FETV_PROMPT_FILE="$PWD/worldfoundry/data/benchmarks/assets/fetv/fetv_data.json"
export WORLDFOUNDRY_FETV_MODEL_NAME=my_t2v_model

Download the official released generated videos and frame assets if you need to reproduce published model cells:

hf download lyx97/FETV_gen_videos \
  --repo-type dataset \
  --local-dir /path/to/datasets/FETV_gen_videos

For a new model, generate videos for the 619 prompts in fetv_data.json, then convert each video into 16 uniformly sampled frames. The repository includes the FETV prompt file; you do not need any external source tree for the scoring code.

Set the CLIP/BLIP frame directory:

export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/fetv/my_t2v_model/16frames_uniform

For full automatic metrics, also prepare:

export WORLDFOUNDRY_FETV_FID_GENERATED_VIDEO_DIR=/path/to/fetv/my_t2v_model_fid
export WORLDFOUNDRY_FETV_FID_REAL_VIDEO_DIR=/path/to/fetv/real_videos_fid
export WORLDFOUNDRY_FETV_FVD_GENERATED_VIDEO_DIR=/path/to/fetv/my_t2v_model_fvd
export WORLDFOUNDRY_FETV_FVD_REAL_VIDEO_DIR=/path/to/fetv/real_videos_fvd

Run Automatic Scoring

Run CLIPScore only:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/fetv/run_fetv_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --fetv-model-name "${WORLDFOUNDRY_FETV_MODEL_NAME}" \
  --fetv-prompt-file "${WORLDFOUNDRY_FETV_PROMPT_FILE}" \
  --fetv-metrics clip_score \
  --output-dir tmp/fetv/clip-score \
  --json

Run CLIPScore and BLIPScore:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/fetv/run_fetv_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --fetv-model-name "${WORLDFOUNDRY_FETV_MODEL_NAME}" \
  --fetv-prompt-file "${WORLDFOUNDRY_FETV_PROMPT_FILE}" \
  --fetv-metrics clip_score blip_score \
  --output-dir tmp/fetv/clip-blip \
  --json

Run all integrated automatic metrics after staging the FID/FVD directories:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/fetv/run_fetv_official_runner.py \
  --run-official \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --fetv-model-name "${WORLDFOUNDRY_FETV_MODEL_NAME}" \
  --fetv-prompt-file "${WORLDFOUNDRY_FETV_PROMPT_FILE}" \
  --fetv-metrics clip_score blip_score fid fvd \
  --fetv-fid-generated-video-dir "${WORLDFOUNDRY_FETV_FID_GENERATED_VIDEO_DIR}" \
  --fetv-fid-real-video-dir "${WORLDFOUNDRY_FETV_FID_REAL_VIDEO_DIR}" \
  --fetv-fvd-generated-video-dir "${WORLDFOUNDRY_FETV_FVD_GENERATED_VIDEO_DIR}" \
  --fetv-fvd-real-video-dir "${WORLDFOUNDRY_FETV_FVD_REAL_VIDEO_DIR}" \
  --output-dir tmp/fetv/official-run \
  --json

Import Existing Results

If you already have a FETV-EVAL output directory or a CSV/JSON result export, import it directly:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/fetv/run_fetv_official_runner.py \
  --official-results-path /path/to/FETV-EVAL/results-or-fetv_eval_results.csv \
  --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --fetv-model-name "${WORLDFOUNDRY_FETV_MODEL_NAME}" \
  --output-dir tmp/fetv/imported \
  --json

The unified benchmark entry can import the same result:

worldfoundry-eval zoo benchmark-run \
  --benchmark-id fetv \
  --mode official-validation \
  --official-results-path /path/to/FETV-EVAL/results-or-fetv_eval_results.csv \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/fetv/official-validation \
  --json

Outputs

The run writes:

scorecard.json: WorldFoundry scorecard with available FETV metrics.
raw_metric_table.jsonl: normalized metric rows.
fetv_eval_results_<model>.csv: metric summary produced by the in-tree runtime.
auto_eval_results/: raw CLIP/BLIP/FID/FVD outputs when --run-official is used.
fetv_official_runtime_summary.json: runtime commands and produced result paths.

Back to Benchmark Hub