WorldFoundry
English中文

Contracts and artifacts

Serializable request, result, and artifact types used across model execution and evaluation.

Edit this pageGitHubMarkdown

On this page
ArtifactReflocal_path_for_urienrich_artifact_refGenerationRequestGenerationResultnormalize_generation_statusis_generation_result_successfulSerialization round trip

These contracts are stdlib-only dataclasses. They can cross process and environment boundaries as JSON, which is why evaluation can consume outputs without importing the model runtime that produced them.

Import the symbols on this page from worldfoundry.evaluation.api.

ArtifactRef

ArtifactRef points to an output without embedding its bytes. kind describes the semantic role, while sha256, size_bytes, MIME information, and media metadata make a local artifact inspectable and auditable.

artifact = ArtifactRef.from_uri(
    "runs/matrix-game-2/generated.mp4",
    kind="generated_video",
    media_metadata={"fps": 12, "width": 640, "height": 352},
)
class ArtifactRef(uri: str,kind: str,sha256: str | None = None,size_bytes: int | None = None,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None,schema_version: str = ARTIFACT_REF_SCHEMA_VERSION)
clsworldfoundry.evaluation.api.ArtifactReffrom worldfoundry.evaluation.api import ArtifactRef
source

Overview

A portable pointer to an artifact (path or URI) with optional size, hash, and media metadata. Use it to pass outputs across process boundaries without embedding file bytes.

Attributes

uristr
kindstr
sha256str | None
default: None
size_bytesint | None
default: None
mime_typestr | None
default: None
media_metadataMapping[str, Any]
default: <dict factory>
metadataMapping[str, Any]
default: <dict factory>
schema_versionstr
default: ARTIFACT_REF_SCHEMA_VERSION

Methods

cmethfrom_path(path: str | Path,kind: str,uri: str | None = None,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None) -> 'ArtifactRef'source

Overview

Public classmethod on this type.

Parameters

pathstr | Path
kindstr
uristr | None
default: None
mime_typestr | None
default: None
media_metadataMapping[str, Any] | None
default: None
metadataMapping[str, Any] | None
default: None

Returns: 'ArtifactRef'

cmethfrom_bytes(data: bytes,uri: str,kind: str,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None) -> 'ArtifactRef'source

Overview

Public classmethod on this type.

Parameters

databytes
uristr
kindstr
mime_typestr | None
default: None
media_metadataMapping[str, Any] | None
default: None
metadataMapping[str, Any] | None
default: None

Returns: 'ArtifactRef'

cmethfrom_uri(uri: str | Path,kind: str,base_dir: str | Path | None = None,mime_type: str | None = None,media_metadata: Mapping[str, Any] | None = None,metadata: Mapping[str, Any] | None = None) -> 'ArtifactRef'source

Overview

Public classmethod on this type.

Parameters

uristr | Path
kindstr
base_dirstr | Path | None
default: None
mime_typestr | None
default: None
media_metadataMapping[str, Any] | None
default: None
metadataMapping[str, Any] | None
default: None

Returns: 'ArtifactRef'

cmethfrom_dict(data: Mapping[str, Any]) -> 'ArtifactRef'source

Overview

Public classmethod on this type.

Parameters

dataMapping[str, Any]

Returns: 'ArtifactRef'

local_path_for_uri

Use this helper before treating a URI as a local file. HTTP, object-store, Hugging Face, and in-memory URIs return None; relative local paths can be resolved against base_dir.

def local_path_for_uri(uri: str | Path,base_dir: str | Path | None = None) -> Path | None
funcworldfoundry.evaluation.api.local_path_for_urifrom worldfoundry.evaluation.api import local_path_for_uri
source

Overview

Resolve a URI to a local filesystem path when possible. Returns None for remote or non-file schemes so callers do not treat HTTP/object-store URIs as local files.

Parameters

uristr | Path
base_dirstr | Path | None
default: None

Returns: Path | None

enrich_artifact_ref

This function adds hash and size evidence when an artifact reference resolves to an existing local file. Remote or missing artifacts remain unchanged.

def enrich_artifact_ref(artifact: ArtifactRef,base_dir: str | Path | None = None) -> ArtifactRef
funcworldfoundry.evaluation.api.enrich_artifact_reffrom worldfoundry.evaluation.api import enrich_artifact_ref
source

Overview

Fill size and SHA-256 on an ArtifactRef when the URI resolves to an existing local file. Missing or remote refs are left unchanged.

Parameters

artifactArtifactRef
base_dirstr | Path | None
default: None

Returns: ArtifactRef

GenerationRequest

A request represents one sample, not an entire batch. Put source media or text in inputs, actions or camera paths in controls, native sampling settings such as seed and FPS in generation_kwargs, and expected artifact shape in output_schema.

For example, two models can receive the same sample_id and action sequence even if their operators translate those controls into different native argument names.

class GenerationRequest(sample_id: str,task_name: str | None = None,task_id: str | None = None,split: str = 'default',request_id: str | None = None,inputs: Mapping[str, Any] | None = None,controls: Mapping[str, Any] | None = None,generation_kwargs: Mapping[str, Any] | None = None,output_schema: Mapping[str, Any] | None = None,cache_policy: Mapping[str, Any] | None = None,schema_version: str = GENERATION_REQUEST_SCHEMA_VERSION)
clsworldfoundry.evaluation.api.GenerationRequestfrom worldfoundry.evaluation.api import GenerationRequest
source

Overview

One evaluation sample’s generation request: inputs, controls, sampling kwargs, and expected outputs. Prefer one request per sample rather than packing a whole batch.

Attributes

sample_idstr
task_namestr
splitstr
default: 'default'
request_idstr | None
default: None
inputsMapping[str, Any]
default: <dict factory>
controlsMapping[str, Any]
default: <dict factory>
generation_kwargsMapping[str, Any]
default: <dict factory>
output_schemaMapping[str, Any]
default: <dict factory>
cache_policyMapping[str, Any]
default: <dict factory>
schema_versionstr
default: GENERATION_REQUEST_SCHEMA_VERSION

Methods

proptask_id -> strsource

Overview

Public property on this type.

Parameters

self

Returns: str

cmethfrom_dict(data: Mapping[str, Any]) -> 'GenerationRequest'source

Overview

Public classmethod on this type.

Parameters

dataMapping[str, Any]

Returns: 'GenerationRequest'

GenerationResult

A result records what happened for the corresponding sample. Successful output belongs in artifacts; execution details belong in timings and metadata; a failure keeps its status and error instead of creating a placeholder artifact.

result = GenerationResult(
    sample_id=request.sample_id,
    model_id="matrix-game-2",
    artifacts={"generated_video": artifact},
    timings={"generation_seconds": 49.75},
)
class GenerationResult(sample_id: str,request_id: str | None = None,model_id: str = '',artifacts: Mapping[str, ArtifactRef] = <dict factory>,status: str = 'succeeded',error: str | None = None,timings: Mapping[str, Any] = <dict factory>,metadata: Mapping[str, Any] = <dict factory>,schema_version: str = GENERATION_RESULT_SCHEMA_VERSION)
clsworldfoundry.evaluation.api.GenerationResultfrom worldfoundry.evaluation.api import GenerationResult
source

Overview

Normalized generation outcome for one sample: artifacts on success, or status/error on failure. Benchmarks and metrics consume this shape, not the model runtime.

Attributes

sample_idstr
request_idstr | None
default: None
model_idstr
default: ''
artifactsMapping[str, ArtifactRef]
default: <dict factory>
statusstr
default: 'succeeded'
errorstr | None
default: None
timingsMapping[str, Any]
default: <dict factory>
metadataMapping[str, Any]
default: <dict factory>
schema_versionstr
default: GENERATION_RESULT_SCHEMA_VERSION

Methods

cmethfrom_dict(data: Mapping[str, Any]) -> 'GenerationResult'source

Overview

Public classmethod on this type.

Parameters

dataMapping[str, Any]

Returns: 'GenerationResult'

normalize_generation_status

Runners may use nearby status words such as success, completed, or done. This helper normalizes text before the success predicate is applied; it does not inspect whether a referenced file exists.

def normalize_generation_status(status: Any) -> str
funcworldfoundry.evaluation.api.normalize_generation_statusfrom worldfoundry.evaluation.api import normalize_generation_status
source

Overview

Map nearby status strings (success/completed/done, …) onto a canonical status before success checks. Does not verify that artifact files exist.

Parameters

statusAny
Raw status value from a generation runner or result row.

Returns: str

is_generation_result_successful

A result is considered successful when its normalized status is in the accepted success set and error is empty. Artifact existence and benchmark eligibility are separate checks.

def is_generation_result_successful(result: 'GenerationResult') -> bool
funcworldfoundry.evaluation.api.is_generation_result_successfulfrom worldfoundry.evaluation.api import is_generation_result_successful
source

Overview

True when the normalized status is an accepted success value and error is empty. Artifact presence and leaderboard eligibility are separate checks.

Parameters

result'GenerationResult'
Generation result emitted by an runner or loaded from disk.

Returns: bool

Serialization round trip

All three contract classes inherit the shared JSON contract methods. Use to_json() for a ledger row and from_json() or from_dict() when loading it again.

payload = result.to_json()
restored = GenerationResult.from_json(payload)

assert restored.sample_id == result.sample_id
assert restored.artifacts["generated_video"].kind == "generated_video"

Serialization proves that the record is well formed. It does not prove that the model ran, the artifact is visually correct, or the result is leaderboard eligible; those stronger claims come from runtime validation and scorecard evidence.