# DEVIL Dynamics (/docs/evaluation/benchmark-hub/devil-dynamics) ## About [#about] DEVIL Dynamics evaluates text-to-video generation from a dynamics perspective. It complements temporal-consistency suites by checking whether generated videos actually move, whether the motion follows the requested dynamics grade, and whether that motion remains visually plausible. WorldFoundry keeps the DEVIL evaluation code in-tree at `worldfoundry/evaluation/tasks/execution/runners/devil_dynamics/`. No official source checkout is needed to run this benchmark. ## Prompt Suite [#prompt-suite] The official prompt text files are bundled in WorldFoundry: ```text worldfoundry/data/benchmarks/assets/devil-dynamics/prompts/ static/ low/ medium/ high/ very_high/ ``` Generate one video per prompt and keep DEVIL's dynamics-grade filename convention. Examples: ```text static_1.mp4 low_12.mp4 medium_37.mp4 high_88.mp4 very_high_123.mp4 ``` The generated video directory can be flat or recursively nested; the runner searches for `.mp4` files. ## Required Assets [#required-assets] Set `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR` to the directory containing the generated videos. For a full metric recomputation, prepare these checkpoints and credentials: | Asset | Env or flag | Notes | | ----------------------- | ------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | DEVIL regression model | `WORLDFOUNDRY_DEVIL_REGRESSION_CKPT` or `--regression-ckpt` | Usually `linear_regress_model.pth` from DEVIL `model_weights`. | | RAFT Things | `WORLDFOUNDRY_DEVIL_RAFT_CKPT` / `WORLDFOUNDRY_RAFT_THINGS_CKPT` or `--raft-ckpt` | Reuses WorldFoundry RAFT base model assets when staged. | | OpenAI CLIP ViT-L/14 | `WORLDFOUNDRY_DEVIL_CLIP_VIT_L14_PT` / `WORLDFOUNDRY_VBENCH_CLIP_VIT_L14_PT` or `--clip-vit-l14` | Used by dynamics scoring. | | OpenAI CLIP ViT-B/32 | `WORLDFOUNDRY_DEVIL_CLIP_VIT_B32_PT` / `WORLDFOUNDRY_VBENCH_CLIP_VIT_B32_PT` or `--clip-vit-b32` | Used by quality scoring. | | ViCLIP | `WORLDFOUNDRY_DEVIL_VICLIP_CKPT` / `WORLDFOUNDRY_VBENCH_VICLIP_CKPT` or `--viclip-ckpt` | Reuses the VBench ViCLIP asset. | | DINOv2 ViT-L/14 | `WORLDFOUNDRY_DEVIL_DINOV2_VITL14_CKPT` / `WORLDFOUNDRY_WBENCH_MEGASAM_DINOV2_CKPT` or `--dinov2-vitl14-ckpt` | Used for segment-level dynamics. | | timm DINO ViT-S/16 | `WORLDFOUNDRY_DEVIL_TIMM_DINO_CKPT` or `--timm-dino-ckpt` | Optional when timm can resolve pretrained weights from cache/download; set it for offline reproducibility. | | DINO ViT-B/16 | `WORLDFOUNDRY_DEVIL_DINO_VITB16_CKPT` / `WORLDFOUNDRY_VBENCH_DINO_VITB16_CKPT` or `--dino-vitb16-ckpt` | Uses WorldFoundry's in-tree DINO shim. | | AMT-S config/checkpoint | `WORLDFOUNDRY_DEVIL_AMT_CONFIG`, `WORLDFOUNDRY_DEVIL_AMT_CKPT`, or flags | Config defaults to WorldFoundry's in-tree AMT-S config when available. | | Naturalness judge | `WORLDFOUNDRY_DEVIL_GEMINI_API_KEY` / `GEMINI_API_KEY` or `--gemini-api-key` | Required unless you pass `--naturalness-path`. | If you have the official DEVIL `model_weights` directory, you can also point `WORLDFOUNDRY_DEVIL_MODEL_WEIGHTS_DIR` or `--model-weights-dir` at it. The runner looks there for common filenames such as `linear_regress_model.pth`, `raft-things.pth`, `ViT-L-14.pt`, `ViT-B-32.pt`, `ViClip-InternVid-10M-FLT.pth`, `dinov2_vitl14_pretrain.pth`, `dino_vitbase16_pretrain.pth`, `AMT-S.yaml`, and `amt-s.pth`. The Python environment must include the video metric packages used by DEVIL, including `torch`, `torchvision`, `timm`, `decord`, `opencv-python`, `clip`, `easydict`, `scikit-image`, `pytorch-msssim`, `piqa`, `imagehash`, `pandas`, `openpyxl`, `prettytable`, `omegaconf`, and `google-generativeai`. The temporal-information component also expects system tools `ffmpeg`, `ffprobe`, and `bc`. ## Full Evaluation [#full-evaluation] Run the in-tree official runtime through the WorldFoundry runner: ```bash export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/devil/generated-videos export WORLDFOUNDRY_DEVIL_MODEL_WEIGHTS_DIR=/path/to/devil/model_weights export GEMINI_API_KEY=... PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/devil_dynamics/run_devil_dynamics_official_runner.py \ --run-official \ --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --model-weights-dir "${WORLDFOUNDRY_DEVIL_MODEL_WEIGHTS_DIR}" \ --num-gpus 8 \ --output-dir tmp/devil-dynamics/full-run \ --json ``` Use fewer GPUs by changing `--num-gpus`. For a single-GPU local run, set `--num-gpus 1`. If Gemini naturalness has already been computed, skip API calls by passing the naturalness xlsx: ```bash PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \ worldfoundry/evaluation/tasks/execution/runners/devil_dynamics/run_devil_dynamics_official_runner.py \ --run-official \ --generated-video-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --model-weights-dir "${WORLDFOUNDRY_DEVIL_MODEL_WEIGHTS_DIR}" \ --naturalness-path /path/to/naturalness_results.xlsx \ --num-gpus 8 \ --output-dir tmp/devil-dynamics/full-run-reuse-naturalness \ --json ``` ## Import Existing Results [#import-existing-results] Use this when you already have an official DEVIL result export such as `dynamics_quality_results.xlsx` or `devil_dynamics_results.json`: ```bash export WORLDFOUNDRY_DEVIL_DYNAMICS_RESULTS_PATH=/path/to/dynamics_quality_results.xlsx worldfoundry-eval zoo benchmark-run \ --benchmark-id devil-dynamics \ --mode official-validation \ --official-results-path "${WORLDFOUNDRY_DEVIL_DYNAMICS_RESULTS_PATH}" \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/devil-dynamics/import-results \ --json ``` ## Output Layout [#output-layout] A full run writes: ```text / scorecard.json raw_metric_table.jsonl per_sample_scores.jsonl upstream/ dynamics_results.xlsx quality_results.xlsx naturalness_results.xlsx dynamics_quality_results.xlsx devil_dynamics_results.json upstream_stdout.log upstream_stderr.log ``` Use `scorecard.json` for WorldFoundry's normalized aggregate and `upstream/devil_dynamics_results.json` for the DEVIL summary emitted by the in-tree runtime. ## Metrics [#metrics] | Metric ID | Meaning | Better | | -------------------------- | ----------------------------------------------------------------------------------- | ------ | | `dynamics_range` | Whether generated videos span the expected static-to-high-motion range. | Higher | | `dynamics_controllability` | Whether motion follows the prompt's requested dynamics grade ordering. | Higher | | `dynamics_quality` | Dynamics-aware quality and naturalness, including Gemini naturalness when computed. | Higher | | `devil_dynamics_average` | Mean aggregate over the DEVIL components available in the run. | Higher | ## Leaderboard Notes [#leaderboard-notes] Leaderboard-equivalent numbers require the complete DEVIL prompt suite, filenames that preserve dynamics-grade prefixes, the same checkpoint assets, and Gemini naturalness outputs produced with the same judge settings. Partial imports remain useful for regression testing, but should not be reported as full DEVIL leaderboard results. [← Benchmark Hub](/docs/evaluation/benchmark-hub)