Metrics and tasks
Metric outputs, metric implementations, task contracts, evaluation protocols, and benchmark specifications.
On this page
Task contracts describe what should be generated. Metric contracts describe how compatible results are scored. Keeping them separate allows one generated artifact set to be evaluated by more than one metric without rerunning the model.
Import the symbols on this page from worldfoundry.evaluation.api.
MetricSpec
MetricSpec is declarative metadata: identity, aliases, required artifact kinds, output unit, aggregation policy, and score direction. The implementation field may point to code, but the spec itself does not execute it.
class MetricSpec(id: str | None = None,metric_id: str | None = None,aliases: Sequence[str] = (),display_name: str = '',description: str = '',version: str = '1.0',family: str = '',capability: str = '',requires_reference: bool = False,required_artifacts: Sequence[str] = (),output_unit: str = '',higher_is_better: bool | None = None,normalizer: str | None = None,aggregator: str = 'mean',statistics: Sequence[str] = ('mean',),primary: bool = False,weight: float = 1.0,implementation: str | None = None,tags: Sequence[str] = (),metadata: Mapping[str, Any] | None = None,schema_version: str = METRIC_SPEC_SCHEMA_VERSION)worldfoundry.evaluation.api.MetricSpecfrom worldfoundry.evaluation.api import MetricSpecOverview
Declarative description of a metric’s identity, inputs, and aggregation expectations used by registries and scorecards.
Attributes
idstraliasestuple[str, ...]- default:
() display_namestr- default:
'' descriptionstr- default:
'' versionstr- default:
'1.0' familystr- default:
'' capabilitystr- default:
'' requires_referencebool- default:
False required_artifactstuple[str, ...]- default:
() output_unitstr- default:
'' higher_is_betterbool | None- default:
None normalizerstr | None- default:
None aggregatorstr- default:
'mean' statisticstuple[str, ...]- default:
('mean',) primarybool- default:
False weightfloat- default:
1.0 implementationstr | None- default:
None tagstuple[str, ...]- default:
() metadataMapping[str, Any]- default:
<dict factory> schema_versionstr- default:
METRIC_SPEC_SCHEMA_VERSION
Methods
MetricResult
One MetricResult belongs to one sample and one metric. Use valid=False with skip_reason when the metric cannot score the sample. coverage makes partial evaluation visible instead of silently averaging only the successful subset.
class MetricResult(sample_id: str,metric_id: str,raw_value: Any = None,normalized_value: float | None = None,components: Mapping[str, Any] = <dict factory>,valid: bool = True,coverage: float = 1.0,skip_reason: str | None = None,diagnostics: Mapping[str, Any] = <dict factory>,artifact_refs: Mapping[str, ArtifactRef] = <dict factory>,judge_trace: Mapping[str, Any] = <dict factory>,schema_version: str = METRIC_RESULT_SCHEMA_VERSION)worldfoundry.evaluation.api.MetricResultfrom worldfoundry.evaluation.api import MetricResultOverview
Per-sample metric output with score payload and optional diagnostics. AggregateResult rolls many of these up.
Attributes
sample_idstrmetric_idstrraw_valueAny- default:
None normalized_valuefloat | None- default:
None componentsMapping[str, Any]- default:
<dict factory> validbool- default:
True coveragefloat- default:
1.0 skip_reasonstr | None- default:
None diagnosticsMapping[str, Any]- default:
<dict factory> artifact_refsMapping[str, ArtifactRef]- default:
<dict factory> judge_traceMapping[str, Any]- default:
<dict factory> schema_versionstr- default:
METRIC_RESULT_SCHEMA_VERSION
Methods
AggregateResult
An aggregate records total, valid, and skipped counts together with summary statistics. It is the metric-level result; leaderboard eligibility is decided later by benchmark and scorecard evidence.
class AggregateResult(metric_id: str,n_total: int = 0,n_valid: int = 0,n_skipped: int = 0,raw_stats: Mapping[str, Any] = <dict factory>,normalized_stats: Mapping[str, Any] = <dict factory>,confidence_interval: Mapping[str, Any] = <dict factory>,stderr: float | None = None,skip_breakdown: Mapping[str, int] = <dict factory>,valid: bool = True,diagnostics: Mapping[str, Any] = <dict factory>,schema_version: str = AGGREGATE_RESULT_SCHEMA_VERSION)worldfoundry.evaluation.api.AggregateResultfrom worldfoundry.evaluation.api import AggregateResultOverview
Dataset- or split-level aggregation over MetricResult rows (means, counts, custom summaries).
Attributes
metric_idstrn_totalint- default:
0 n_validint- default:
0 n_skippedint- default:
0 raw_statsMapping[str, Any]- default:
<dict factory> normalized_statsMapping[str, Any]- default:
<dict factory> confidence_intervalMapping[str, Any]- default:
<dict factory> stderrfloat | None- default:
None skip_breakdownMapping[str, int]- default:
<dict factory> validbool- default:
True diagnosticsMapping[str, Any]- default:
<dict factory> schema_versionstr- default:
AGGREGATE_RESULT_SCHEMA_VERSION
Methods
Metric
Metric implementations satisfy this protocol structurally. The following deliberately simple metric counts artifact references; it shows the API shape, not a benchmark-quality video metric.
from worldfoundry.evaluation.api import AggregateResult, MetricResult
class ArtifactCountMetric:
name = "artifact_count_example"
version = "1.0"
required_artifacts = ()
higher_is_better = None
def compute_sample(self, request, result):
value = len(result.artifacts)
return MetricResult(
sample_id=request.sample_id,
metric_id=self.name,
raw_value=value,
normalized_value=value,
)
def aggregate(self, results):
values = [item.raw_value for item in results if item.valid]
mean = sum(values) / len(values) if values else None
return AggregateResult(
metric_id=self.name,
n_total=len(results),
n_valid=len(values),
n_skipped=len(results) - len(values),
raw_stats={"mean": mean},
normalized_stats={"mean": mean},
valid=bool(values),
)class Metric(name: str,version: str,required_artifacts: tuple[str, ...],higher_is_better: bool | None)worldfoundry.evaluation.api.Metricfrom worldfoundry.evaluation.api import MetricOverview
Minimum metric implementation surface: score samples and optionally aggregate. Keep heavyweight backends behind this contract.
Attributes
namestrversionstrrequired_artifactstuple[str, ...]higher_is_betterbool | None
Methods
Overview
Public method on this type.
Parameters
requestGenerationRequestresultGenerationResult
Returns: MetricResult
EvaluationProtocolSpec
A protocol groups metric IDs and metric groups under a named evaluation behavior. Extra protocol-specific keys loaded from a catalog are preserved in metadata.
class EvaluationProtocolSpec(name: str,metric_ids: tuple[str, ...] = (),metric_groups: tuple[str, ...] = (),metadata: Mapping[str, Any] = <dict factory>,schema_version: str = EVALUATION_PROTOCOL_SCHEMA_VERSION)worldfoundry.evaluation.api.EvaluationProtocolSpecfrom worldfoundry.evaluation.api import EvaluationProtocolSpecOverview
Describes how a task/protocol evaluates generations (required artifacts, metrics, pass rules).
Source docstring
Evaluation protocol entry for a task or benchmark catalog task.
The public API keeps simple string protocols supported, while catalog surfaces can use this structured form to attach metric ids, groups, and protocol-specific metadata.
Attributes
namestrmetric_idstuple[str, ...]- default:
() metric_groupstuple[str, ...]- default:
() metadataMapping[str, Any]- default:
<dict factory> schema_versionstr- default:
EVALUATION_PROTOCOL_SCHEMA_VERSION
Methods
Overview
Public classmethod on this type.
Parameters
dataMapping[str, Any]
Returns: 'EvaluationProtocolSpec'
Overview
Public classmethod on this type.
Parameters
valueAny
Returns: tuple['EvaluationProtocolSpec', ...]
WorldTaskConfig
This object describes one task's input keys, output keys, generation defaults, and intended metric set. For an action-conditioned video task, inputs might contain an initial image, controls might carry actions, and the output key would be generated_video.
from worldfoundry.evaluation.api import WorldTaskConfig
task = WorldTaskConfig(
name="action-conditioned-video",
protocol="open_loop",
input_keys=("image", "actions"),
output_keys=("generated_video",),
metric_ids=("artifact_count_example",),
generation_defaults={"fps": 12, "seed": 42},
)class WorldTaskConfig(name: str | None = None,task_id: str | None = None,protocol: str = 'open_loop',evaluation_protocol: str = 'reference_metrics',capability_track: str = 'core_video',schema_type: str = 'sample',input_keys: Sequence[str] = (),output_keys: Sequence[str] = ('generated_video',),metric_ids: Sequence[str] = (),metric_groups: Sequence[str] = (),tags: Sequence[str] = (),description: str = '',data: Mapping[str, Any] | None = None,generation_defaults: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = WORLD_TASK_CONFIG_SCHEMA_VERSION)worldfoundry.evaluation.api.WorldTaskConfigfrom worldfoundry.evaluation.api import WorldTaskConfigOverview
Task configuration bound to a protocol and dataset slice for one evaluation run.
Attributes
namestrprotocolstr- default:
'open_loop' evaluation_protocolstr- default:
'reference_metrics' capability_trackstr- default:
'core_video' schema_typestr- default:
'sample' input_keystuple[str, ...]- default:
() output_keystuple[str, ...]- default:
('generated_video',) metric_idstuple[str, ...]- default:
() metric_groupstuple[str, ...]- default:
() tagstuple[str, ...]- default:
() descriptionstr- default:
'' dataMapping[str, Any]- default:
<dict factory> generation_defaultsMapping[str, Any]- default:
<dict factory> metadataMapping[str, Any]- default:
<dict factory> schema_versionstr- default:
WORLD_TASK_CONFIG_SCHEMA_VERSION
Methods
BenchmarkSpec
BenchmarkSpec collects tasks, metrics, splits, and dataset metadata into an in-process benchmark description. A checked-in benchmark catalog entry can be richer; this public DTO is the execution-facing form.
from worldfoundry.evaluation.api import BenchmarkSpec, MetricSpec
benchmark = BenchmarkSpec(
name="navigation-smoke-test",
tasks=(task,),
metrics=(MetricSpec(id="artifact_count_example"),),
splits=("validation",),
)class BenchmarkSpec(name: str | None = None,benchmark_id: str | None = None,version: str = '1.0',tasks: Sequence[WorldTaskConfig | Mapping[str, Any]] = (),metrics: Sequence[MetricSpec | Mapping[str, Any]] = (),splits: Sequence[str] = ('default',),tags: Sequence[str] = (),description: str = '',dataset_root: str | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = BENCHMARK_SPEC_SCHEMA_VERSION)worldfoundry.evaluation.api.BenchmarkSpecfrom worldfoundry.evaluation.api import BenchmarkSpecOverview
Public benchmark identity and wiring metadata used by the hub and runners.
Attributes
namestrversionstr- default:
'1.0' taskstuple[WorldTaskConfig, ...]- default:
() metricstuple[MetricSpec, ...]- default:
() splitstuple[str, ...]- default:
('default',) tagstuple[str, ...]- default:
() descriptionstr- default:
'' dataset_rootstr | None- default:
None metadataMapping[str, Any]- default:
<dict factory> schema_versionstr- default:
BENCHMARK_SPEC_SCHEMA_VERSION
Methods
For an official benchmark integration, these contracts are necessary but not sufficient. The add a benchmark guide covers datasets, official runners, normalizers, runtime profiles, coverage, and evidence gates.