Metric 与 task
Metric 输出与实现、task 契约、评测 protocol 和 benchmark specification。
Task 契约描述应该生成什么，metric 契约描述如何给兼容 result 打分。二者分开后，同一套生成 artifact 可以交给多个 metric，而不必重新运行模型。
本页 symbol 均从 worldfoundry.evaluation.api 导入。
MetricSpec
MetricSpec 是声明式 metadata，记录身份、alias、所需 artifact kind、输出单位、聚合策略和分数方向。implementation 可以指向代码，但 spec 本身不会执行代码。
class MetricSpec(id: str | None = None,metric_id: str | None = None,aliases: Sequence[str] = (),display_name: str = '',description: str = '',version: str = '1.0',family: str = '',capability: str = '',requires_reference: bool = False,required_artifacts: Sequence[str] = (),output_unit: str = '',higher_is_better: bool | None = None,normalizer: str | None = None,aggregator: str = 'mean',statistics: Sequence[str] = ('mean',),primary: bool = False,weight: float = 1.0,implementation: str | None = None,tags: Sequence[str] = (),metadata: Mapping[str, Any] | None = None,schema_version: str = METRIC_SPEC_SCHEMA_VERSION)cls
源码worldfoundry.evaluation.api.MetricSpecfrom worldfoundry.evaluation.api import MetricSpec简介
度量指标的声明式描述：身份、输入与聚合期望，供 registry 与 scorecard 使用。
属性
idstraliasestuple[str, ...]- 默认值:
() display_namestr- 默认值:
'' descriptionstr- 默认值:
'' versionstr- 默认值:
'1.0' familystr- 默认值:
'' capabilitystr- 默认值:
'' requires_referencebool- 默认值:
False required_artifactstuple[str, ...]- 默认值:
() output_unitstr- 默认值:
'' higher_is_betterbool | None- 默认值:
None normalizerstr | None- 默认值:
None aggregatorstr- 默认值:
'mean' statisticstuple[str, ...]- 默认值:
('mean',) primarybool- 默认值:
False weightfloat- 默认值:
1.0 implementationstr | None- 默认值:
None tagstuple[str, ...]- 默认值:
() metadataMapping[str, Any]- 默认值:
<dict factory> schema_versionstr- 默认值:
METRIC_SPEC_SCHEMA_VERSION
方法
MetricResult
一个 MetricResult 对应一个 sample 与一个 metric。Metric 无法给该 sample 打分时，应使用 valid=False 和 skip_reason。coverage 让部分评测保持可见，避免只平均成功子集却不说明缺失范围。
class MetricResult(sample_id: str,metric_id: str,raw_value: Any = None,normalized_value: float | None = None,components: Mapping[str, Any] = <dict factory>,valid: bool = True,coverage: float = 1.0,skip_reason: str | None = None,diagnostics: Mapping[str, Any] = <dict factory>,artifact_refs: Mapping[str, ArtifactRef] = <dict factory>,judge_trace: Mapping[str, Any] = <dict factory>,schema_version: str = METRIC_RESULT_SCHEMA_VERSION)cls
源码worldfoundry.evaluation.api.MetricResultfrom worldfoundry.evaluation.api import MetricResult简介
单样本 metric 输出，含分数与可选诊断信息；AggregateResult 负责多样本汇总。
属性
sample_idstrmetric_idstrraw_valueAny- 默认值:
None normalized_valuefloat | None- 默认值:
None componentsMapping[str, Any]- 默认值:
<dict factory> validbool- 默认值:
True coveragefloat- 默认值:
1.0 skip_reasonstr | None- 默认值:
None diagnosticsMapping[str, Any]- 默认值:
<dict factory> artifact_refsMapping[str, ArtifactRef]- 默认值:
<dict factory> judge_traceMapping[str, Any]- 默认值:
<dict factory> schema_versionstr- 默认值:
METRIC_RESULT_SCHEMA_VERSION
方法
AggregateResult
Aggregate 同时记录总数、有效数、跳过数与统计量。它是 metric 级结果；是否具备 leaderboard 资格，要到 benchmark 和 scorecard 证据层再判断。
class AggregateResult(metric_id: str,n_total: int = 0,n_valid: int = 0,n_skipped: int = 0,raw_stats: Mapping[str, Any] = <dict factory>,normalized_stats: Mapping[str, Any] = <dict factory>,confidence_interval: Mapping[str, Any] = <dict factory>,stderr: float | None = None,skip_breakdown: Mapping[str, int] = <dict factory>,valid: bool = True,diagnostics: Mapping[str, Any] = <dict factory>,schema_version: str = AGGREGATE_RESULT_SCHEMA_VERSION)cls
源码worldfoundry.evaluation.api.AggregateResultfrom worldfoundry.evaluation.api import AggregateResult简介
对多条 MetricResult 的数据集/划分级汇总（均值、计数或自定义摘要）。
属性
metric_idstrn_totalint- 默认值:
0 n_validint- 默认值:
0 n_skippedint- 默认值:
0 raw_statsMapping[str, Any]- 默认值:
<dict factory> normalized_statsMapping[str, Any]- 默认值:
<dict factory> confidence_intervalMapping[str, Any]- 默认值:
<dict factory> stderrfloat | None- 默认值:
None skip_breakdownMapping[str, int]- 默认值:
<dict factory> validbool- 默认值:
True diagnosticsMapping[str, Any]- 默认值:
<dict factory> schema_versionstr- 默认值:
AGGREGATE_RESULT_SCHEMA_VERSION
方法
Metric
Metric 实现通过结构满足这个 protocol。下面故意使用一个只计算 artifact 引用数的简单 metric；它用来展示 API 形态，不是 benchmark 级视频指标。
from worldfoundry.evaluation.api import AggregateResult, MetricResult
class ArtifactCountMetric:
name = "artifact_count_example"
version = "1.0"
required_artifacts = ()
higher_is_better = None
def compute_sample(self, request, result):
value = len(result.artifacts)
return MetricResult(
sample_id=request.sample_id,
metric_id=self.name,
raw_value=value,
normalized_value=value,
)
def aggregate(self, results):
values = [item.raw_value for item in results if item.valid]
mean = sum(values) / len(values) if values else None
return AggregateResult(
metric_id=self.name,
n_total=len(results),
n_valid=len(values),
n_skipped=len(results) - len(values),
raw_stats={"mean": mean},
normalized_stats={"mean": mean},
valid=bool(values),
)class Metric(name: str,version: str,required_artifacts: tuple[str, ...],higher_is_better: bool | None)prot
源码worldfoundry.evaluation.api.Metricfrom worldfoundry.evaluation.api import Metric简介
Metric 的最小实现面：对样本打分，并可选择做聚合。重量级后端应藏在该契约之后。
属性
namestrversionstrrequired_artifactstuple[str, ...]higher_is_betterbool | None
方法
EvaluationProtocolSpec
Protocol 把 metric ID 与 metric group 组织到一种命名评测行为下。从 catalog 读取的额外 protocol 字段会保留在 metadata。
class EvaluationProtocolSpec(name: str,metric_ids: tuple[str, ...] = (),metric_groups: tuple[str, ...] = (),metadata: Mapping[str, Any] = <dict factory>,schema_version: str = EVALUATION_PROTOCOL_SCHEMA_VERSION)cls
源码worldfoundry.evaluation.api.EvaluationProtocolSpecfrom worldfoundry.evaluation.api import EvaluationProtocolSpec简介
描述某任务/协议如何评测生成结果（所需产物、指标与通过规则）。
源码 docstring
Evaluation protocol entry for a task or benchmark catalog task.
The public API keeps simple string protocols supported, while catalog surfaces can use this structured form to attach metric ids, groups, and protocol-specific metadata.
属性
namestrmetric_idstuple[str, ...]- 默认值:
() metric_groupstuple[str, ...]- 默认值:
() metadataMapping[str, Any]- 默认值:
<dict factory> schema_versionstr- 默认值:
EVALUATION_PROTOCOL_SCHEMA_VERSION
方法
WorldTaskConfig
这个对象描述一个 task 的输入 key、输出 key、生成默认值和预期 metric。对于动作条件视频，输入可以包含初始图像，controls 携带动作，输出 key 则是 generated_video。
from worldfoundry.evaluation.api import WorldTaskConfig
task = WorldTaskConfig(
name="action-conditioned-video",
protocol="open_loop",
input_keys=("image", "actions"),
output_keys=("generated_video",),
metric_ids=("artifact_count_example",),
generation_defaults={"fps": 12, "seed": 42},
)class WorldTaskConfig(name: str | None = None,task_id: str | None = None,protocol: str = 'open_loop',evaluation_protocol: str = 'reference_metrics',capability_track: str = 'core_video',schema_type: str = 'sample',input_keys: Sequence[str] = (),output_keys: Sequence[str] = ('generated_video',),metric_ids: Sequence[str] = (),metric_groups: Sequence[str] = (),tags: Sequence[str] = (),description: str = '',data: Mapping[str, Any] | None = None,generation_defaults: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = WORLD_TASK_CONFIG_SCHEMA_VERSION)cls
源码worldfoundry.evaluation.api.WorldTaskConfigfrom worldfoundry.evaluation.api import WorldTaskConfig简介
绑定协议与数据切片的任务配置，用于一次评测运行。
属性
namestrprotocolstr- 默认值:
'open_loop' evaluation_protocolstr- 默认值:
'reference_metrics' capability_trackstr- 默认值:
'core_video' schema_typestr- 默认值:
'sample' input_keystuple[str, ...]- 默认值:
() output_keystuple[str, ...]- 默认值:
('generated_video',) metric_idstuple[str, ...]- 默认值:
() metric_groupstuple[str, ...]- 默认值:
() tagstuple[str, ...]- 默认值:
() descriptionstr- 默认值:
'' dataMapping[str, Any]- 默认值:
<dict factory> generation_defaultsMapping[str, Any]- 默认值:
<dict factory> metadataMapping[str, Any]- 默认值:
<dict factory> schema_versionstr- 默认值:
WORLD_TASK_CONFIG_SCHEMA_VERSION
方法
BenchmarkSpec
BenchmarkSpec 把 task、metric、split 和 dataset metadata 组成进程内 benchmark 描述。仓库中的 benchmark catalog 条目可能更丰富；这个公开 DTO 是面向执行的形态。
from worldfoundry.evaluation.api import BenchmarkSpec, MetricSpec
benchmark = BenchmarkSpec(
name="navigation-smoke-test",
tasks=(task,),
metrics=(MetricSpec(id="artifact_count_example"),),
splits=("validation",),
)class BenchmarkSpec(name: str | None = None,benchmark_id: str | None = None,version: str = '1.0',tasks: Sequence[WorldTaskConfig | Mapping[str, Any]] = (),metrics: Sequence[MetricSpec | Mapping[str, Any]] = (),splits: Sequence[str] = ('default',),tags: Sequence[str] = (),description: str = '',dataset_root: str | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = BENCHMARK_SPEC_SCHEMA_VERSION)cls
源码worldfoundry.evaluation.api.BenchmarkSpecfrom worldfoundry.evaluation.api import BenchmarkSpec简介
公开 benchmark 身份与接线元数据，供 Hub 与 runner 使用。
属性
namestrversionstr- 默认值:
'1.0' taskstuple[WorldTaskConfig, ...]- 默认值:
() metricstuple[MetricSpec, ...]- 默认值:
() splitstuple[str, ...]- 默认值:
('default',) tagstuple[str, ...]- 默认值:
() descriptionstr- 默认值:
'' dataset_rootstr | None- 默认值:
None metadataMapping[str, Any]- 默认值:
<dict factory> schema_versionstr- 默认值:
BENCHMARK_SPEC_SCHEMA_VERSION
方法
对于 official benchmark integration，这些契约是必要条件，但还不够。添加基准指南还会处理 dataset、official runner、normalizer、runtime profile、覆盖率与证据 gate。