# WorldReasonBench (/docs/evaluation/benchmark-hub/worldreasonbench) ## About [#about] World-state reasoning QA and reward-model evaluation for generated videos. **WorldFoundry evaluation path:** Imports existing official or upstream results into a WorldFoundry scorecard; this integration cannot yet replay the full official protocol from raw generated videos. ## Integration status [#integration-status] | Item | Current state | | ------------------- | --------------------------------------- | | Catalog integration | `integrated` | | Runner verification | `normalizer_only` | | Leaderboard | not yet eligible for leaderboard claims | ## Metrics [#metrics] ### Task YAML [#task-yaml] `worldfoundry/data/benchmarks/tasks/external/worldreasonbench.yaml` ### Primary [#primary] * `score_pr` | Metric | Focus | | -------------------------------- | ----------------------------------------------------------------------------------------------- | | `score_pr` | Geometric combination of QA accuracy and dynamic reasoning score. | | `qa_accuracy` | Correctness of predicted actions, states, or outcomes against the benchmark success criterion. | | `state_score` | Value reported for State Score under the official benchmark protocol. | | `process_score` | Value reported for Process Score under the official benchmark protocol. | | `fidelity_score` | Value reported for Fidelity Score under the official benchmark protocol. | | `mechanism_score` | Value reported for Mechanism Score under the official benchmark protocol. | | `static_outcome_score` | Value reported for Static Outcome Score under the official benchmark protocol. | | `dynamic_reasoning_score` | Value reported for Dynamic Reasoning Score under the official benchmark protocol. | | `reasoning_gap` | Static outcome score minus dynamic reasoning score; lower values indicate less outcome hacking. | | `pointwise_score` | Value reported for Pointwise S(v) under the official benchmark protocol. | | `reasoning_correctness` | Value reported for Reasoning Correctness under the official benchmark protocol. | | `content_fidelity` | Value reported for Content Fidelity under the official benchmark protocol. | | `visual_aesthetics` | Value reported for Visual Aesthetics under the official benchmark protocol. | | `pointwise_spearman` | Value reported for Pointwise Spearman Correlation under the official benchmark protocol. | | `induced_pairwise_accuracy` | Correctness of predicted actions, states, or outcomes against the benchmark success criterion. | | `pairwise_accuracy_with_ties` | Correctness of predicted actions, states, or outcomes against the benchmark success criterion. | | `pairwise_accuracy_without_ties` | Correctness of predicted actions, states, or outcomes against the benchmark success criterion. | ## Environment setup [#environment-setup] ### Runtime environment [#runtime-environment] * **Conda env:** `worldfoundry-unified-cu128`. * **Separate setup:** defaults to the unified WorldFoundry env (`needs_new_env: false`); a benchmark-only conda env is usually not required. * Result normalization uses the unified WorldFoundry environment and has no GPU dependency. * Real evaluation reuses the unified environment when it provides openai, opencv-python, numpy, Pillow, and tqdm. * The judge model server is an external OpenAI-compatible service and may use its own serving environment. ### Evaluation data [#evaluation-data] * Candidate model outputs: set `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR` to the generated-video or rollout artifact root. * `worldfoundry/evaluation/tasks/execution/runners/worldreasonbench` * `worldfoundry/data/benchmarks/tasks/external/worldreasonbench.yaml` ### Checkpoints & assets [#checkpoints--assets] * This benchmark is primarily rollout- or official-result-driven; see catalog `requires` and the task YAML for policy/simulator checkpoints. ### Key environment variables [#key-environment-variables] * `WORLDFOUNDRY_WORLDREASONBENCH_ROOT` * `WORLDFOUNDRY_WORLDREASONBENCH_RESULTS_PATH` * `WORLDFOUNDRY_WORLDREASONBENCH_PROTOCOL` * `WORLDFOUNDRY_WORLDREASONBENCH_QA_JSON` * `WORLDFOUNDRY_WORLDREASONBENCH_PAIRS_JSON` * `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR` * `OPENAI_BASE_URL` * `OPENAI_API_KEY` * `OPENAI_MODEL` * `HF_ENDPOINT` ## Run Evaluation [#run-evaluation] After preparing the assets and candidate artifacts listed on this page, use the public run entry points below. ### Set Candidate Artifacts [#set-candidate-artifacts] ```bash export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/generated/artifacts ``` ### Import Official Results [#import-official-results] ```bash worldfoundry-eval zoo benchmark-run \ --benchmark-id worldreasonbench \ --mode normalizer \ --official-results-path /path/to/official/results-or-report \ --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \ --output-dir tmp/worldreasonbench/normalizer \ --json ``` ### Validate or Normalize Existing Results [#validate-or-normalize-existing-results] ```bash worldfoundry-eval zoo benchmark-run --benchmark-id worldreasonbench --mode official-validation --official-results-path --output-dir --json ``` ## Requirements [#requirements] ### Inputs [#inputs] * WORLDFOUNDRY\_WORLDREASONBENCH\_ROOT for official execution * OPENAI\_BASE\_URL and a compatible multimodal judge ### Environment [#environment] * `WORLDFOUNDRY_GENERATED_ARTIFACT_DIR` ### Outputs [#outputs] * `scorecard.json` * `raw_metric_table.jsonl` * `protocol_results.json` * `benchmark_contract.json` ## Official references [#official-references] * [GitHub](https://github.com/UniX-AI-Lab/WorldReasonBench) * [Paper](https://arxiv.org/abs/2605.10434) * [Project page](https://unix-ai-lab.github.io/WorldReasonBench/) * [Hugging Face dataset](https://huggingface.co/datasets/WorldReasonBench) [← Benchmark Hub](/docs/evaluation/benchmark-hub)