# MiraBench (/docs/evaluation/benchmark-hub/mirabench) ## About [#about] MiraBench is the evaluation component introduced with MiraData, a large-scale long-video dataset with structured captions. It evaluates long video generation from six perspectives: temporal consistency, temporal motion strength, 3D consistency, visual quality, text-video alignment, and distribution consistency. The official scorer exposes 17 metrics, including DINO/CLIP temporal consistency, RAFT-style motion strength, 3D reconstruction errors, aesthetic and imaging quality, ViCLIP alignment, FVD, FID, and KID. WorldFoundry integrates the MiraBench runtime in tree under `worldfoundry/evaluation/tasks/execution/runners/mirabench/runtime/mirabench`, with the public runner at `worldfoundry/evaluation/tasks/execution/runners/mirabench/run_mirabench_official_runner.py`. Official references: * Project page: [mira-space.github.io](https://mira-space.github.io/) * Paper: [arXiv:2407.06358](https://arxiv.org/abs/2407.06358) * Dataset: [TencentARC/MiraData](https://huggingface.co/datasets/TencentARC/MiraData) * Official source reference: [mira-space/MiraData](https://github.com/mira-space/MiraData) ## Evaluation Protocol [#evaluation-protocol] MiraBench uses metadata CSV files with structured captions. WorldFoundry bundles the official example files: ```text worldfoundry/data/benchmarks/assets/mirabench/data/evaluation_example/meta_generated.csv worldfoundry/data/benchmarks/assets/mirabench/data/evaluation_example/meta_gt.csv ``` For a full run, prepare a `meta_generated.csv` with these columns: ```text video_idx,video_path,short_caption,dense_caption,main_object_caption,background_caption,style_caption,camera_caption ``` The runner materializes the generated-video paths from `video_idx`, so stage generated videos as: ```text /path/to/mirabench/generated_videos/ 1.mp4 2.mp4 ... 150.mp4 ``` The canonical benchmark prompt set contains 150 prompts. Use `--strict` when you want the run to fail on missing prompt videos. Primary metric: `mirabench_average`. | Metric | Meaning | | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | `dynamic_degree`, `tracking_strength` | Motion intensity and long-range tracking strength. | | `dino_temporal_consistency`, `clip_temporal_consistency`, `temporal_motion_smoothness` | Temporal structure, semantic continuity, and motion smoothness. | | `mean_absolute_error`, `root_mean_square_error` | 3D consistency errors; lower is better. | | `aesthetic_quality`, `imaging_quality` | Visual quality metrics. | | `camera_alignment`, `main_object_alignment`, `background_alignment`, `style_alignment`, `overall_alignment` | Structured caption alignment metrics. | | `fvd`, `fid`, `kid` | Distribution metrics; lower is better. | | `mirabench_average` | Official aggregate when present, or mean over available metrics. | ## Data And Checkpoint Preparation [#data-and-checkpoint-preparation] Start from the WorldFoundry repository root: ```bash cd /path/to/WorldFoundry export WORLDFOUNDRY_MIRABENCH_META_CSV=/path/to/mirabench/meta_generated.csv export WORLDFOUNDRY_MIRABENCH_GT_META_CSV=/path/to/mirabench/meta_gt.csv export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/mirabench/generated_videos ``` Download MiraData metadata from Hugging Face if you need the released metadata: ```bash hf download TencentARC/MiraData \ --repo-type dataset \ --local-dir /path/to/datasets/MiraData ``` Stage the scorer checkpoints used by `calculate_score.py`: ```bash export WORLDFOUNDRY_MIRABENCH_CKPT_PATH=/path/to/mirabench/data/ckpt export WORLDFOUNDRY_MIRABENCH_SCORER_BACKEND=official export WORLDFOUNDRY_MIRABENCH_DEVICE=cuda ``` For quick local wiring runs, `WORLDFOUNDRY_MIRABENCH_SCORER_BACKEND=mock` writes deterministic placeholder metric values. Do not use that backend for reported benchmark numbers. ## Generate Candidate Videos [#generate-candidate-videos] Use the `dense_caption` or `short_caption` fields from `meta_generated.csv` as generation prompts. For WorldFoundry-integrated models, run the model's synthesis workflow over those prompts, then export one video per `video_idx`: ```text /path/to/mirabench/generated_videos/ 1.mp4 2.mp4 ... ``` ## Run Official Scoring [#run-official-scoring] Run the in-tree MiraBench official scorer: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/mirabench/run_mirabench_official_runner.py \ --run-official \ --meta-csv "${WORLDFOUNDRY_MIRABENCH_META_CSV}" \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/mirabench/official-run \ --strict \ --json ``` The official backend dispatches the in-tree `calculate_score.py`, materializes an updated generated meta CSV with absolute video paths, and converts the produced `average_score.csv` into a WorldFoundry scorecard. ## Import Existing Results [#import-existing-results] If you already have a MiraBench `average_score.csv` or equivalent JSON/CSV result export, import it directly: ```bash cd /path/to/WorldFoundry PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/mirabench/run_mirabench_official_runner.py \ --official-results-path /path/to/mirabench/average_score.csv \ --meta-csv "${WORLDFOUNDRY_MIRABENCH_META_CSV}" \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/mirabench/imported \ --json ``` The unified benchmark entry can import the same result: ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id mirabench \ --mode official-validation \ --official-results-path /path/to/mirabench/average_score.csv \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/mirabench/official-validation \ --json ``` ## Outputs [#outputs] The run writes: * `scorecard.json`: WorldFoundry scorecard with available MiraBench metrics. * `mirabench_average_score.json`: normalized copy of the official average score output. * `raw_metric_table.jsonl`: metric rows used by the scorecard. * `per_sample_scores.jsonl`: per-video rows when the result file exposes sample-level values. * `upstream_scorer/`: materialized meta CSV, extracted frames, and official scorer outputs for `--run-official`. [Back to Benchmark Hub](/docs/evaluation/benchmark-hub)