MemoBench

About

MemoBench evaluates visual memory in world generation models. Its official protocol is Visible -> Disappeared -> Reappear (V-D-R): a target object is visible, leaves the camera field of view, and later re-enters the scene. The model is judged on whether it preserves object identity, scene memory, camera behavior, geometric consistency, visual quality, and plausible physical state across the disappearance interval.

The benchmark contains 360 clips across synthetic and real-world scenes and evaluates 14 metrics covering visual quality, temporal consistency, geometric fidelity, object permanence, camera controllability, and VQA-based reasoning.

The in-tree MemoBench runtime and source docs live under:

worldfoundry/evaluation/tasks/execution/runners/memobench/runtime/memobench/

WorldFoundry includes the runtime code, docs, result-import surface, and small static evaluation assets under worldfoundry/data/benchmarks/assets/memobench/data. Large dataset assets, model checkpoints, SAM-3, and VLM/API credentials remain explicit setup inputs for full-suite execution.

What To Prepare

Official Data Layout

Prepare the MemoBench data assets in the layout used by the official runtime:

data/
|-- Synthetic_processed/
|   `-- {Scene}/{Scene}_{NNN}/
|       |-- image.jpg
|       |-- intrinsics.npy
|       `-- poses.npy
|-- Real_Raw/
|   `-- {NNN}/
|       |-- {NNN}.mp4
|       |-- {NNN}-intrinsics.json
|       `-- timestamps.txt
|-- mapanything/outputs/real/*.npz
|-- sam3_metadata/*.csv
`-- vqa_questions/
    |-- *.csv
    `-- failure-cases.csv

The release data can be stored outside the repository. When running the MemoBench scripts from the in-tree runtime, point their data arguments or working directory at the prepared asset root. The full suite needs SAM-3-compatible segmentation assets for Object Revisit Score and a Gemini or compatible VLM setup for VQA.

Generated Output Layout

MemoBench expects frame directories rather than a single video file per clip:

output/{model_name}/
|-- Synthetic/
|   `-- {Scene}/
|       `-- {Scene}_{NNN}/
|           `-- frames/
|               |-- 00000.png
|               `-- ...
`-- Real/
    `-- {NNN}/
        `-- frames/
            |-- 00001.png
            `-- ...

Frames should be PNG files with zero-padded numeric filenames. The upstream evaluation sorts frames lexicographically, so inconsistent padding can change temporal order.

Metrics

Automated Metrics

Object Revisit

Pixel Fidelity

VQA Metrics

memobench_average is the WorldFoundry aggregate over available normalized component metrics. It is useful for local validation, but leaderboard claims still require full MemoBench evidence coverage.

Run Evaluation

Step 1: Automated Metrics Through WorldFoundry

export WORLDFOUNDRY_MEMOBENCH_MODEL_NAME=MyModel
export WORLDFOUNDRY_GENERATED_ARTIFACT_DIR=/path/to/output/MyModel

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/memobench/run_memobench_official_runner.py \
  --run-official \
  --mode both \
  --generated-synthetic-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}/Synthetic" \
  --generated-real-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}/Real" \
  --model-name "${WORLDFOUNDRY_MEMOBENCH_MODEL_NAME}" \
  --output-dir tmp/memobench/step1 \
  --json

This launches the in-tree evaluation/run_eval.py stage and writes a WorldFoundry scorecard from the resulting eval_both.csv.

Step 1 Through The Public CLI

worldfoundry-eval zoo benchmark-run \
  --benchmark-id memobench \
  --mode official-run \
  --generated-artifact-dir "${WORLDFOUNDRY_GENERATED_ARTIFACT_DIR}" \
  --output-dir tmp/memobench/step1-cli \
  --json

Use the direct runner when you want to pass separate synthetic and real roots, --device, --max-side, or --sample-step.

Import Existing MemoBench Outputs

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/memobench/run_memobench_official_runner.py \
  --official-results-path /path/to/eval_both.csv \
  --model-name "${WORLDFOUNDRY_MEMOBENCH_MODEL_NAME}" \
  --output-dir tmp/memobench/imported \
  --json

You can also combine stage directories:

export WORLDFOUNDRY_MEMOBENCH_EVAL_DIR=/path/to/step1_outputs
export WORLDFOUNDRY_MEMOBENCH_ORS_DIR=/path/to/ors_results
export WORLDFOUNDRY_MEMOBENCH_VQA_DIR=/path/to/vqa_results
export WORLDFOUNDRY_MEMOBENCH_LEADERBOARD_PATH=/path/to/leaderboard.csv

PYTHONPATH=. "${WORLDFOUNDRY_UNIFIED_PYTHON:-python}" \
  worldfoundry/evaluation/tasks/execution/runners/memobench/run_memobench_official_runner.py \
  --model-name "${WORLDFOUNDRY_MEMOBENCH_MODEL_NAME}" \
  --output-dir tmp/memobench/combined \
  --json

Official Stage Details

Step 1: Automated Metrics

From the in-tree runtime directory:

cd worldfoundry/evaluation/tasks/execution/runners/memobench/runtime/memobench

python evaluation/run_eval.py \
  --mode both \
  --gen_root_syn /path/to/output/MyModel/Synthetic \
  --gen_root_real /path/to/output/MyModel/Real \
  --out_csv outputs/eval_both.csv

Useful options:

Step 1 writes one row per clip and includes composite metrics, diagnostics, and per-phase pixel fidelity.

Step 2: Object Revisit Score

From the same in-tree runtime directory:

python evaluation/compute_ors.py \
  --model-name MyModel \
  --output-dir ors_results

The script uses object descriptions from worldfoundry/data/benchmarks/assets/memobench/data/sam3_metadata/ by default, identifies Reappear-phase frames from MemoBench phase-boundary files, and runs SAM-3 text-prompted segmentation. Override the metadata directory with WORLDFOUNDRY_MEMOBENCH_SAM3_METADATA_DIR when needed. The resulting ors_scores.csv includes the clip id, data type, scene, Reappear start index, ORS, detection rate, mean confidence, and evaluated frame counts.

Step 3: VQA Evaluation

From the same in-tree runtime directory:

python evaluation/vqa/llm-vqa.py \
  --model-name MyModel \
  --output-dir vqa_results \
  --questions-dir worldfoundry/data/benchmarks/assets/memobench/data/vqa_questions \
  --cases-csv worldfoundry/data/benchmarks/assets/memobench/data/vqa_questions/failure-cases.csv

The VQA step writes per-clip CSV files under vqa_results/{ModelName}/. Each row includes the scene, video id, full question-level evaluation JSON, and a score dictionary for instruction following, object/background consistency, continuity of memory, and physics adherence.

Leaderboard

After Step 1, ORS, and VQA outputs are available, run the leaderboard script from the in-tree runtime:

python leaderboard/leaderboard.py \
  --model-name MyModel \
  --output-dir leaderboard_results

WorldFoundry can import the generated leaderboard CSV, but local comparisons are only meaningful when the full evidence package is present.

Environment

Requirements

Inputs

• MemoBench generated frame layout for synthetic and real clips
• Official dataset metadata and phase-boundary files for full Step 1 execution
• Step 1, ORS, VQA, or leaderboard output files for import-only runs
• SAM-3 setup for Object Revisit Score
• Gemini or compatible VLM configuration for VQA

Outputs

• scorecard.json
• raw_metric_table.jsonl
• per_sample_scores.jsonl
• Upstream Step 1 eval_*.csv when --run-official is used

Back to Benchmark Hub