WorldFoundry
EnglishδΈ­ζ–‡

VideoScore

Integrated

Human-feedback-trained video reward metric with in-tree runtime, frame-tree requirements, datasets, and run commands.

MarkdownSource

On this page
AboutEvaluation ProtocolData And Frames PreparationCheckpoint PreparationRun A Full Frame-Tree EvaluationRun A Bounded ValidationImport Existing ResultsTrainingOutputs

About

VideoScore is an EMNLP 2024 automatic video evaluation model trained to simulate fine-grained human feedback for video generation. It is trained on VideoFeedback, a dataset of 37.6K generated videos from 11 video models with human ratings across multiple aspects.

WorldFoundry integrates the VideoScore inference and benchmark evaluation runtime in tree at worldfoundry/evaluation/tasks/execution/runners/videoscore. The official repository is a protocol reference; WorldFoundry runs the checked-in VideoScore runtime and loads datasets/checkpoints from local assets or Hugging Face.

Official references:

Evaluation Protocol

VideoScore predicts five aspect scores on a 1-4 scale:

MetricMeaning
visual_qualityOverall visual quality, fidelity, and artifact level.
temporal_consistencyTemporal stability and frame-to-frame coherence.
dynamic_degreeAmount and quality of motion.
text_to_video_alignmentAlignment between the prompt and generated video.
factual_consistencyWhether generated content is factually consistent with the described scene.
videoscore_averageMean of the five aspect scores.

WorldFoundry normalizes 1-4 raw scores to 0-1 in the scorecard.

Data And Frames Preparation

VideoScore benchmark evaluation consumes extracted frame directories, not a flat directory of mp4 files. Prepare either:

  • --frames-dir: a directory already containing frames in the layout expected by the official eval_videoscore.py, or
  • --bench-data-root plus --bounded-sample-count: WorldFoundry materializes a bounded set of frames from the local benchmark dataset for a small validation run.

Download benchmark data and optional training/evidence data:

export WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT=/path/to/datasets/VideoScore-Bench
export WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT=/path/to/datasets/VideoFeedback

hf download TIGER-Lab/VideoScore-Bench \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT}"

hf download TIGER-Lab/VideoFeedback \
  --repo-type dataset \
  --local-dir "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}"

If you prepare a full frame tree yourself:

export WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR=/path/to/videoscore/frames

--bench-name selects the split inside VideoScore-Bench. The default is video_feedback; other official splits come from the VideoScore-Bench release.

Checkpoint Preparation

Use the official VideoScore-v1.1 reward model or a local mirror:

export WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO=/path/to/checkpoints/VideoScore-v1.1
hf download TIGER-Lab/VideoScore-v1.1 \
  --local-dir "${WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO}"

If WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO is not set, the runner falls back to the Hugging Face id TIGER-Lab/VideoScore-v1.1.

Run A Full Frame-Tree Evaluation

Run VideoScore on a prepared frame tree:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \
  --model-repo-name "${WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO:-TIGER-Lab/VideoScore-v1.1}" \
  --bench-data-root "${WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT}" \
  --bench-name "${WORLDFOUNDRY_VIDEOSCORE_BENCH_NAME:-video_feedback}" \
  --frames-dir "${WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR}" \
  --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \
  --output-dir tmp/videoscore/official-run \
  --timeout 7200 \
  --json

This path calls the in-tree benchmark/eval_videoscore.py runtime and writes the official result JSON under upstream/.

Run A Bounded Validation

For a small local check using the downloaded benchmark data:

cd /path/to/WorldFoundry

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \
  --model-repo-name "${WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO:-TIGER-Lab/VideoScore-v1.1}" \
  --bench-data-root "${WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT}" \
  --bench-name video_feedback \
  --bounded-sample-count 8 \
  --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \
  --output-dir tmp/videoscore/bounded8 \
  --json

The bounded path materializes frames for a small number of benchmark rows and runs the same VideoScore normalization pipeline. Use full frame-tree evaluation for benchmark-scale results.

Import Existing Results

Import an existing official-compatible VideoScore result JSON:

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \
  --official-results-path /path/to/eval_video_feedback_videoscore.json \
  --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \
  --frames-dir "${WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR}" \
  --output-dir tmp/videoscore/imported \
  --json

Training

The official VideoScore paper trains the reward model using the Mantis training stack and VideoFeedback data. WorldFoundry vendors the inference/evaluation runtime needed for benchmark reproduction. To reproduce VideoScore model training itself, use the official training protocol and then pass your trained model directory with --model-repo-name.

Outputs

Each run writes:

  • scorecard.json: normalized aspect scores and videoscore_average.
  • raw_metric_table.jsonl: metric rows used by the scorecard.
  • per_sample_scores.jsonl: per-sample extracted scores when available.
  • generated_video_manifest.json: frame/video manifest for the evaluated artifacts.
  • dataset_manifest.json: discovered dataset metadata.
  • upstream/eval_<bench_name>_videoscore.json: raw official result JSON.
  • upstream_stdout.log and upstream_stderr.log: runtime logs.

Back to Benchmark Hub

Related

OverviewBenchmark HubMetricsScorers & qualityEnvironmentCLILocal assets