Run 与 benchmark

Existing artifact、模型推理、单个 model × benchmark cell 和 suite 的统一进程内调度。

最后更新: 2026年7月22日

本页内容

一个完整的无 GPU run WorldFoundryRunRequestWorldFoundryRunResultrun_worldfoundrylist_video_benchmarksrun_benchmarknormalize_upstream_resultsbenchmark_integration_spec

run_worldfoundry 是宽入口。它检查一个 typed request，再分发到 existing-results、模型生成、单个 model × benchmark 或 suite runner。若生成 artifact 已经存在，只需要运行 official evaluator 或 normalizer，则应使用更窄的 benchmark facade。

本页 symbol 均从 worldfoundry.evaluation.public 导入。

一个完整的无 GPU run

这个例子创建一份很小的 trajectory artifact，写出 request/result ledger，再用内置 artifact_count metric 评测已有结果。它会经过真实 run 与 reporting 路径，但不会加载模型。

from pathlib import Path

from worldfoundry.evaluation.api import ArtifactRef, GenerationRequest, GenerationResult
from worldfoundry.evaluation.public import WorldFoundryRunRequest, run_worldfoundry

root = Path("tmp/python_api_example")
trace_path = root / "artifacts" / "trajectory.json"
trace_path.parent.mkdir(parents=True, exist_ok=True)
trace_path.write_text('{"actions":["forward","left"]}\n', encoding="utf-8")

request = GenerationRequest(sample_id="nav-0001", task_name="navigation-trace")
result = GenerationResult(
    sample_id=request.sample_id,
    model_id="existing-trace",
    artifacts={
        "trajectory": ArtifactRef.from_path(trace_path, kind="trajectory"),
    },
)

requests_path = root / "requests.jsonl"
results_path = root / "results.jsonl"
requests_path.write_text(request.to_json() + "\n", encoding="utf-8")
results_path.write_text(result.to_json() + "\n", encoding="utf-8")

outcome = run_worldfoundry(
    WorldFoundryRunRequest(
        output_dir=root / "evaluation",
        requests_path=requests_path,
        results_path=results_path,
        metrics=("artifact_count",),
    )
)

assert outcome.ok
print(outcome.to_dict()["scorecard_path"])

输出目录包含对齐后的 ledger、execution plan、metric row、run_manifest.json、summary.json、report.md 和 scorecard.json。这个 run 可以证明 existing-results artifact check 成功，但不会因此变成 official benchmark 或 leaderboard 结果。

`WorldFoundryRunRequest`

这个 request 刻意覆盖多个 mode。results_path 选择 existing-results；只有 model ID 时选择模型执行；model 与 benchmark ID 同时存在时选择一个 benchmark cell；多个选择或 suite ID 会进入矩阵 runner。execute=False 可以先规划兼容 cell，不立即消耗计算资源。

class WorldFoundryRunRequest(output_dir: str | Path,model_ids: Sequence[str] = (),benchmark_ids: Sequence[str] = (),suite_ids: Sequence[str] = (),all_benchmarks: bool = False,benchmark_id: str | None = None,benchmark_manifest_dir: str | Path = BENCHMARK_ZOO_DIR,model_manifest_dir: str | Path | None = MODEL_ZOO_DIR,suite_preset_path: str | Path | None = None,engine: str = 'auto',benchmark_mode: str = 'official-run',execute: bool = True,resume: bool = False,skip_incompatible: bool = True,fail_on_skipped: bool = False,model_runner: str | None = None,model_variant_id: str | None = None,model_parameters: Mapping[str, Any] | None = None,model_runtime: Mapping[str, Any] | None = None,model_config: Mapping[str, Any] | Any | None = None,requests_path: str | Path | None = None,results_path: str | Path | None = None,task_name: str | None = None,task_roots: Sequence[str | Path] | None = None,task_benchmark: str | None = None,task_recursive: bool = False,task_root_dir: str | Path | None = None,dataset_root: str | Path | None = None,dataset_id: str | None = None,split: str = 'default',num_samples: int | None = None,generated_artifact_dir: str | Path | None = None,output_artifact: str | None = None,required_artifacts: Sequence[str] | None = None,metrics: Sequence[str] = ('artifact_count', 'required_artifacts_present'),generation_cache_dir: str | Path | None = None,generation_cache_mode: str = 'off',generation_cache_namespace: str = 'worldfoundry_run',benchmark_timeout_seconds: float | None = None,benchmark_workdir: str | Path | None = None,benchmark_env: Mapping[str, Any] | None = None,benchmark_parameters: Mapping[str, Any] | None = None,materialize_placeholders: bool | None = None,contract_fixture: bool = False,fail_on_generation_error: bool = False,run_id: str | None = None,fail_on_sample_error: bool = False,write_artifacts_index: bool = True)

clsworldfoundry.evaluation.public.WorldFoundryRunRequestfrom worldfoundry.evaluation.public import WorldFoundryRunRequest

源码

简介

在进程内启动 WorldFoundry 评测的顶层请求（模型、任务、限制与输出目录）。

属性

output_dirstr | Path
model_idsSequence[str]: 默认值: ()
benchmark_idsSequence[str]: 默认值: ()
suite_idsSequence[str]: 默认值: ()
all_benchmarksbool: 默认值: False
benchmark_idstr | None: 默认值: None
benchmark_manifest_dirstr | Path: 默认值: BENCHMARK_ZOO_DIR
model_manifest_dirstr | Path | None: 默认值: MODEL_ZOO_DIR
suite_preset_pathstr | Path | None: 默认值: None
enginestr: 默认值: 'auto'
benchmark_modestr: 默认值: 'official-run'
executebool: 默认值: True
resumebool: 默认值: False
skip_incompatiblebool: 默认值: True
fail_on_skippedbool: 默认值: False
model_runnerstr | None: 默认值: None
model_variant_idstr | None: 默认值: None
model_parametersMapping[str, Any] | None: 默认值: None
model_runtimeMapping[str, Any] | None: 默认值: None
model_configMapping[str, Any] | Any | None: 默认值: None
requests_pathstr | Path | None: 默认值: None
results_pathstr | Path | None: 默认值: None
task_namestr | None: 默认值: None
task_rootsSequence[str | Path] | None: 默认值: None
task_benchmarkstr | None: 默认值: None
task_recursivebool: 默认值: False
task_root_dirstr | Path | None: 默认值: None
dataset_rootstr | Path | None: 默认值: None
dataset_idstr | None: 默认值: None
splitstr: 默认值: 'default'
num_samplesint | None: 默认值: None
generated_artifact_dirstr | Path | None: 默认值: None
output_artifactstr | None: 默认值: None
required_artifactsSequence[str] | None: 默认值: None
metricsSequence[str]: 默认值: ('artifact_count', 'required_artifacts_present')
generation_cache_dirstr | Path | None: 默认值: None
generation_cache_modestr: 默认值: 'off'
generation_cache_namespacestr: 默认值: 'worldfoundry_run'
benchmark_timeout_secondsfloat | None: 默认值: None
benchmark_workdirstr | Path | None: 默认值: None
benchmark_envMapping[str, Any] | None: 默认值: None
benchmark_parametersMapping[str, Any] | None: 默认值: None
materialize_placeholdersbool | None: 默认值: None
contract_fixturebool: 默认值: False
fail_on_generation_errorbool: 默认值: False
run_idstr | None: 默认值: None
fail_on_sample_errorbool: 默认值: False
write_artifacts_indexbool: 默认值: True

`WorldFoundryRunResult`

Wrapper 提供共同的 status、exit code、输出目录与 mode-specific delegate。to_dict() 会把常用 manifest 和 scorecard 路径提升到顶层，因此自动化代码不必为每种 run kind 单独分支。

class WorldFoundryRunResult(schema_version: str,kind: str,status: str,exit_code: int,output_dir: Path,delegate: Any)

clsworldfoundry.evaluation.public.WorldFoundryRunResultfrom worldfoundry.evaluation.public import WorldFoundryRunResult

源码

简介

run_worldfoundry 的结构化结果：manifest、scorecard 与逐样本 ledger 路径。

属性

schema_versionstr
kindstr
statusstr
exit_codeint
output_dirPath
delegateAny

方法

propok -> bool源码

简介

ok — Return True if the run completed successfully with exit code 0.

参数

self

返回值: bool

methto_dict() -> dict[str, Any]源码

简介

to_dict — Convert the run result and its delegate details into a serializable dictionary.

返回值: dict[str, Any]

`run_worldfoundry`

可以传 typed request、mapping 或 keyword arguments。推荐 typed request，因为编辑器能补全字段，拼错字段也能在执行前暴露。

def run_worldfoundry(request: WorldFoundryRunRequest | Mapping[str, Any] | None = None,**kwargs: Any) -> WorldFoundryRunResult

funcworldfoundry.evaluation.public.run_worldfoundryfrom worldfoundry.evaluation.public import run_worldfoundry

源码

简介

规范的进程内入口：执行 WorldFoundryRunRequest 并返回证据产物。

参数

requestWorldFoundryRunRequest | Mapping[str, Any] | None: 默认值: None
kwargsAny

返回值: WorldFoundryRunResult

`list_video_benchmarks`

这个发现 helper 返回仓库 video benchmark catalog 中的 ID。返回列表并不表示所有 benchmark 都已在本机可运行；readiness 与资产需要单独检查。

def list_video_benchmarks(, catalog_dir: str | Path | None = None) -> list[str]

funcworldfoundry.evaluation.public.list_video_benchmarksfrom worldfoundry.evaluation.public import list_video_benchmarks

源码

简介

列出公开评测 facade 可用的已注册视频 benchmark。

参数

catalog_dirstr | Path | None: 默认值: None

返回值: list[str]

`run_benchmark`

Artifact 已经物化、需要进入 benchmark 专属路径时使用这个 facade。official-run 调用配置的官方 runtime，official-validation 执行 bounded validation，normalizer 导入调用者提供的 official-shaped result。

result = run_benchmark(
    "vbench",
    output_dir="tmp/vbench_run",
    generated_artifact_dir="runs/generated_videos",
    mode="official-run",
)

上面是实际 API 形态，但运行需要满足当前 VBench manifest 报告的资产、依赖、prompt 覆盖率与环境。Runner 可用不代表 leaderboard ready。

def run_benchmark(benchmark_id: str,output_dir: str | Path,mode: str = 'official-run',generated_artifact_dir: str | Path | None = None,manifest_path: str | Path = BENCHMARK_ZOO_DIR,**kwargs: Any) -> Any

funcworldfoundry.evaluation.public.run_benchmarkfrom worldfoundry.evaluation.public import run_benchmark

源码

简介

通过公开 facade 运行指定 benchmark，输入输出均已规范化。

源码 docstring

Run a benchmark through the unified official runner stack.

Modes mirror `worldfoundry-eval zoo benchmark-run`:

`normalizer`: normalize caller-provided official results
`official-validation`: run the benchmark's bounded validation command
`official-run`: invoke upstream official runtime when assets are available

参数

benchmark_idstr
output_dirstr | Path
modestr: 默认值: 'official-run'
generated_artifact_dirstr | Path | None: 默认值: None
manifest_pathstr | Path: 默认值: BENCHMARK_ZOO_DIR
kwargsAny

返回值: Any

`normalize_upstream_results`

上游 evaluator 已经生成结果文件时使用这个函数。它会围绕该文件创建 WorldFoundry 证据，但不会反过来证明 WorldFoundry 曾执行官方 evaluator。

def normalize_upstream_results(benchmark_id: str,results_path: str | Path,output_dir: str | Path,generated_artifact_dir: str | Path | None = None,manifest_path: str | Path = BENCHMARK_ZOO_DIR,**kwargs: Any) -> Any

funcworldfoundry.evaluation.public.normalize_upstream_resultsfrom worldfoundry.evaluation.public import normalize_upstream_results

源码

简介

把上游/厂商结果载荷适配为 WorldFoundry 的 GenerationResult 记录。

参数

benchmark_idstr
results_pathstr | Path
output_dirstr | Path
generated_artifact_dirstr | Path | None: 默认值: None
manifest_pathstr | Path: 默认值: BENCHMARK_ZOO_DIR
kwargsAny

返回值: Any

`benchmark_integration_spec`

当 in-tree integration specification 存在时，这个 lookup 会返回它。Catalog 条目可以先于 integration spec 存在，所以返回 None 是正常的发现结果。

def benchmark_integration_spec(benchmark_id: str) -> BenchmarkIntegrationSpec | None

funcworldfoundry.evaluation.public.benchmark_integration_specfrom worldfoundry.evaluation.public import benchmark_integration_spec

源码

简介

描述上游 benchmark 的接入方式（入口、产物与环境需求）。

参数

benchmark_idstr

返回值: BenchmarkIntegrationSpec | None

调用 official runtime 前，请先在 Benchmark Hub查看 protocol 专属输入与 blocker。