# Visual Chronometer (/docs/evaluation/benchmark-hub/visual-chronometer)



## About [#about]

Visual Chronometer external benchmark task manifest for in-tree PhyFPS prediction.

**WorldFoundry evaluation path:** Runs the in-tree official metric runtime on caller-supplied generated artifacts using local checkpoints and evaluators. Candidate outputs are read from the generated-artifact directory.

## Integration status [#integration-status]

| Item                | Current state                           |
| ------------------- | --------------------------------------- |
| Catalog integration | `integrated`                            |
| Runner verification | `official_runtime_ready`                |
| Leaderboard         | not yet eligible for leaderboard claims |

## Metrics [#metrics]

### Task YAML [#task-yaml]

`worldfoundry/data/benchmarks/tasks/external/visual-chronometer.yaml`

### Primary [#primary]

* `visual_chronometer_average`

| Metric                       | Focus                                                                                                          |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `mean_phyfps`                | Mean average PhyFPS across evaluated videos.                                                                   |
| `inter_video_cv`             | Cross-video coefficient of variation over average PhyFPS.                                                      |
| `intra_video_cv`             | Mean within-video coefficient of variation over sliding-window clips.                                          |
| `visual_chronometer_average` | Mean over Visual Chronometer temporal-consistency score families for time-perception probes (lower is better). |

## Environment setup [#environment-setup]

### Runtime environment [#runtime-environment]

* **Conda env:** `worldfoundry-unified-cu128`.
* **Separate setup:** defaults to the unified WorldFoundry env (`needs_new_env: false`); a benchmark-only conda env is usually not required.
* WorldFoundry runs Visual Chronometer PhyFPS prediction in-process and normalizes results.csv into the Visual Chronometer metric surface.
* Use WORLDFOUNDRY\_VISUAL\_CHRONOMETER\_PREDICT\_BACKEND to select the local prediction backend.

### Evaluation data [#evaluation-data]

* Candidate model outputs: set `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR` to the generated-video or rollout artifact root.
* `worldfoundry/evaluation/tasks/execution/runners/phyfps_bench_gen/runtime/visual_chronometer`
* `worldfoundry/data/benchmarks/tasks/external/visual-chronometer.yaml`

### Checkpoints & assets [#checkpoints--assets]

* Base-model / metric dependencies: `visual_chronometer_fps_predictor`.
* Default metric-checkpoint paths and override env vars are listed in the [local assets guide](/docs/guides/local-assets).

### Key environment variables [#key-environment-variables]

* `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR`
* `WORLDFOUNDRY_VISUAL_CHRONOMETER_ROOT`
* `WORLDFOUNDRY_VISUAL_CHRONOMETER_PREDICT_BACKEND`
* `WORLDFOUNDRY_VISUAL_CHRONOMETER_RESULTS_PATH`
* `WORLDFOUNDRY_VISUAL_CHRONOMETER_STRICT`

### Verify setup [#verify-setup]

```bash
PYTHONPATH=. ${WORLDFOUNDRY_UNIFIED_PYTHON:-python} -m worldfoundry.evaluation.tasks.execution.orchestration.runtime_preflight --profile visual-chronometer --manifest worldfoundry/data/benchmarks/runtime_profiles/official/visual-chronometer.yaml --output-dir tmp/worldfoundry_preflight/visual-chronometer --json
```

## Run Evaluation [#run-evaluation]

After preparing the assets and candidate artifacts listed on this page, use the public run entry points below.

### Set Candidate Artifacts [#set-candidate-artifacts]

```bash
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/artifacts
```

### Import Official Results [#import-official-results]

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id visual-chronometer \
  --mode normalizer \
  --official-results-path /path/to/official/results-or-report \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/visual-chronometer/normalizer \
  --json
```

### Score Generated Artifacts [#score-generated-artifacts]

```bash
worldfoundry-eval zoo benchmark-run \
  --benchmark-id visual-chronometer \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/visual-chronometer/official-run \
  --json
```

### Direct In-Tree Runner [#direct-in-tree-runner]

```bash
PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" worldfoundry/evaluation/tasks/execution/runners/phyfps_bench_gen/run_visual_chronometer_official_runner.py --run-official --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" --output-dir "${WORLDFOUNDRY_BENCHMARK_OUTPUT_DIR}" --json
```

## Requirements [#requirements]

### Inputs [#inputs]

* WORLDFOUNDRY\_GENERATED\_ARTIFACT\_DIR
* Visual Chronometer checkpoint assets for full official prediction

### Environment [#environment]

* `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR`

### Outputs [#outputs]

* `scorecard.json`
* `benchmark_contract.json`
* `raw_metric_table.jsonl`
* `results.csv`
* `per_sample_scores.jsonl`

## Official references [#official-references]

* [GitHub](https://github.com/taco-group/Visual_Chronometer)

[← Benchmark Hub](/docs/evaluation/benchmark-hub)