# VideoScore (/docs/evaluation/benchmark-hub/videoscore) ## About [#about] VideoScore is an EMNLP 2024 automatic video evaluation model trained to simulate fine-grained human feedback for video generation. It is trained on VideoFeedback, a dataset of 37.6K generated videos from 11 video models with human ratings across multiple aspects. WorldFoundry integrates the VideoScore inference and benchmark evaluation runtime in tree at `worldfoundry/evaluation/tasks/execution/runners/videoscore`. The official repository is a protocol reference; WorldFoundry runs the checked-in VideoScore runtime and loads datasets/checkpoints from local assets or Hugging Face. Official references: * Project page: [tiger-ai-lab.github.io/VideoScore](https://tiger-ai-lab.github.io/VideoScore/) * Paper: [arXiv:2406.15252](https://arxiv.org/abs/2406.15252) * Dataset: [TIGER-Lab/VideoFeedback](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback) * Benchmark dataset: [TIGER-Lab/VideoScore-Bench](https://huggingface.co/datasets/TIGER-Lab/VideoScore-Bench) * Model: [TIGER-Lab/VideoScore-v1.1](https://huggingface.co/TIGER-Lab/VideoScore-v1.1) * In-tree runner: `worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py` ## Evaluation Protocol [#evaluation-protocol] VideoScore predicts five aspect scores on a 1-4 scale: | Metric | Meaning | | ------------------------- | --------------------------------------------------------------------------- | | `visual_quality` | Overall visual quality, fidelity, and artifact level. | | `temporal_consistency` | Temporal stability and frame-to-frame coherence. | | `dynamic_degree` | Amount and quality of motion. | | `text_to_video_alignment` | Alignment between the prompt and generated video. | | `factual_consistency` | Whether generated content is factually consistent with the described scene. | | `videoscore_average` | Mean of the five aspect scores. | WorldFoundry normalizes 1-4 raw scores to 0-1 in the scorecard. ## Data And Frames Preparation [#data-and-frames-preparation] VideoScore benchmark evaluation consumes extracted frame directories, not a flat directory of mp4 files. Prepare either: * `--frames-dir`: a directory already containing frames in the layout expected by the official `eval_videoscore.py`, or * `--bench-data-root` plus `--bounded-sample-count`: WorldFoundry materializes a bounded set of frames from the local benchmark dataset for a small validation run. Download benchmark data and optional training/evidence data: ```bash export WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT=/path/to/datasets/VideoScore-Bench export WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT=/path/to/datasets/VideoFeedback hf download TIGER-Lab/VideoScore-Bench \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT}" hf download TIGER-Lab/VideoFeedback \ --repo-type dataset \ --local-dir "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" ``` If you prepare a full frame tree yourself: ```bash export WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR=/path/to/videoscore/frames ``` `--bench-name` selects the split inside VideoScore-Bench. The default is `video_feedback`; other official splits come from the VideoScore-Bench release. ## Checkpoint Preparation [#checkpoint-preparation] Use the official VideoScore-v1.1 reward model or a local mirror: ```bash export WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO=/path/to/checkpoints/VideoScore-v1.1 hf download TIGER-Lab/VideoScore-v1.1 \ --local-dir "${WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO}" ``` If `WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO` is not set, the runner falls back to the Hugging Face id `TIGER-Lab/VideoScore-v1.1`. ## Run A Full Frame-Tree Evaluation [#run-a-full-frame-tree-evaluation] Run VideoScore on a prepared frame tree: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \ --model-repo-name "${WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO:-TIGER-Lab/VideoScore-v1.1}" \ --bench-data-root "${WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT}" \ --bench-name "${WORLDFOUNDRY_VIDEOSCORE_BENCH_NAME:-video_feedback}" \ --frames-dir "${WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR}" \ --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \ --output-dir tmp/videoscore/official-run \ --timeout 7200 \ --json ``` This path calls the in-tree `benchmark/eval_videoscore.py` runtime and writes the official result JSON under `upstream/`. ## Run A Bounded Validation [#run-a-bounded-validation] For a small local check using the downloaded benchmark data: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \ --model-repo-name "${WORLDFOUNDRY_VIDEOSCORE_MODEL_REPO:-TIGER-Lab/VideoScore-v1.1}" \ --bench-data-root "${WORLDFOUNDRY_VIDEOSCORE_BENCH_ROOT}" \ --bench-name video_feedback \ --bounded-sample-count 8 \ --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \ --output-dir tmp/videoscore/bounded8 \ --json ``` The bounded path materializes frames for a small number of benchmark rows and runs the same VideoScore normalization pipeline. Use full frame-tree evaluation for benchmark-scale results. ## Import Existing Results [#import-existing-results] Import an existing official-compatible VideoScore result JSON: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/videoscore/run_videoscore_official_runner.py \ --official-results-path /path/to/eval_video_feedback_videoscore.json \ --dataset-root "${WORLDFOUNDRY_VIDEOFEEDBACK_DATASET_ROOT}" \ --frames-dir "${WORLDFOUNDRY_VIDEOSCORE_FRAMES_DIR}" \ --output-dir tmp/videoscore/imported \ --json ``` ## Training [#training] The official VideoScore paper trains the reward model using the Mantis training stack and VideoFeedback data. WorldFoundry vendors the inference/evaluation runtime needed for benchmark reproduction. To reproduce VideoScore model training itself, use the official training protocol and then pass your trained model directory with `--model-repo-name`. ## Outputs [#outputs] Each run writes: * `scorecard.json`: normalized aspect scores and `videoscore_average`. * `raw_metric_table.jsonl`: metric rows used by the scorecard. * `per_sample_scores.jsonl`: per-sample extracted scores when available. * `generated_video_manifest.json`: frame/video manifest for the evaluated artifacts. * `dataset_manifest.json`: discovered dataset metadata. * `upstream/eval__videoscore.json`: raw official result JSON. * `upstream_stdout.log` and `upstream_stderr.log`: runtime logs. [Back to Benchmark Hub](/docs/evaluation/benchmark-hub)