BENCHMARK.md

# LeWorld Memory Architecture vs. LeCun's World Models — Benchmark Comparison

## ⚠️ Honest Status: This Is a Theoretical Comparison

**Our architecture has NOT been benchmarked on the standard world-model evaluation suites yet.** What follows is:
1. A factual catalog of every LeCun-family world model, their exact published numbers, and benchmarks
2. An architectural comparison showing where our design is fundamentally different
3. A concrete plan for which benchmarks to run and what we'd need to demonstrate

We are not claiming results we don't have. Here's what exists and what's needed.

---

## 1. The LeCun World Model Family — Published Numbers

### Model Lineup (by publication date)

| Model | Paper | Params | Training | Key Innovation |
|-------|-------|--------|----------|----------------|
| **I-JEPA** | arxiv:2301.08243 | 632M (ViT-H/14) | ImageNet, 16×A100 | Predicts masked image patch representations |
| **V-JEPA** | arxiv:2404.08471 | 632M (ViT-H/16) | VideoMix2M (2M videos) | Predicts masked video region features |
| **DINO-WM** | arxiv:2411.04983 | ~300M (frozen DINOv2 + predictor) | Offline trajectories | Plans in DINOv2 patch-feature space |
| **LeWM** | arxiv:2603.19312 | **~15M** | Single GPU, few hours | End-to-end from pixels, 2 loss terms only |
| **V-JEPA 2.1** | arxiv:2603.14482 | 1.1–1.9B (ViT-g/G) | Massive video corpus | Dense features, multi-layer loss |

### LeWM is our primary comparison target (both ~15M params).

---

## 2. Published Benchmark Results — Robotic Planning

### Push-T (Tabletop Block Pushing → Success Rate ↑)

| Model | Params | Success Rate | Planning Time | Notes |
|-------|--------|-------------|---------------|-------|
| **DINO-WM** | ~300M+ (frozen DINOv2) | **0.90** | ~48 seconds | Uses pretrained DINOv2 encoder |
| **LeWM** | ~15M | **0.88** (beats DINO-WM pixel-only) | **<1 second** | End-to-end from pixels, 48× faster |
| **PLDM** | ~15M | 0.70 | <1 second | End-to-end, 7 loss terms (VICReg) |
| DreamerV3 | 12-400M | 0.30 | — | Model-based RL, needs rewards |
| IRIS | — | 0.32 | — | — |
| TD-MPC2 | — | 0.00 | — | Fails without reward signal |
| **Ours (LeWorld Memory)** | **13.5M** | **❌ Not yet tested** | — | — |

### PointMaze / Wall Navigation → Success Rate ↑

| Model | Maze SR | Wall SR | Notes |
|-------|---------|---------|-------|
| **DINO-WM** | 0.98 | 0.96 | Near-perfect |
| DreamerV3 | **1.00** | **1.00** | Perfect on simple navigation |
| LeWM | Lower than DINO-WM | Lower | Struggles on very simple envs (SIGReg limitation) |
| **Ours** | **❌ Not yet tested** | **❌ Not yet tested** | — |

### Reach (Robotic Arm) → Success Rate ↑

| Model | Success Rate |
|-------|-------------|
| **DINO-WM** | **0.92** |
| DreamerV3 | 0.64 |
| IRIS | 0.18 |
| **Ours** | **❌ Not yet tested** |

### Rope & Granular Manipulation → Chamfer Distance ↓

| Model | Rope CD ↓ | Granular CD ↓ |
|-------|-----------|---------------|
| **DINO-WM** | **0.41** | **0.26** |
| DreamerV3 | 2.49 | 1.05 |
| IRIS | 1.11 | 0.37 |
| **Ours** | **❌ Not yet tested** | **❌ Not yet tested** |

---

## 3. Published Benchmark Results — Physical Understanding

### Physical Latent Probing on Push-T (Pearson r ↑, higher = better)

| Property | DINO-WM (MLP) | LeWM (MLP) | PLDM (MLP) |
|----------|--------------|------------|------------|
| Agent Location | **r = 0.999** | r = 0.998 | r = 0.993 |
| Block Location | **r = 0.999** | **r = 0.999** | r = 0.994 |
| Block Angle | **r = 0.995** | r = 0.990 | r = 0.972 |

LeWM at 15M achieves near-parity with DINO-WM (300M+ pretrained) on physical probing.

### Violation-of-Expectation (VoE) — Physics Anomaly Detection

| Perturbation | LeWM | PLDM |
|-------------|------|------|
| Teleportation (physically impossible) | **Detects (p<0.01)** | Detects |
| Color change (visual only) | Does NOT flag | Does NOT flag |
| Correct distinction? | ✅ Yes | ✅ Yes |

### IntPhys 2 (Intuitive Physics) — Accuracy ↑

| Model | Accuracy | Notes |
|-------|----------|-------|
| Human | **96.4%** | Ceiling |
| V-JEPA 2 (1B+ params) | 57.5% | Best JEPA model |
| Gemini 2.5 Flash | 55.6% | Best commercial LLM |
| GPT-4o | ~50% | Near random |
| **Ours** | **❌ Not yet tested** | **Potential differentiator** (see below) |

**The IntPhys gap (57.5% vs 96.4% human) is the biggest open problem in world models.**

---

## 4. Published Benchmark Results — Video Understanding

### Kinetics-400 (Video Action Recognition) → Top-1 Accuracy ↑

| Model | Params | Accuracy | Probe Type |
|-------|--------|----------|------------|
| V-JEPA 2.1 ViT-G | 1.9B | **~88%** | Attentive |
| V-JEPA ViT-H/16 | 632M | 81.9% | Frozen, attentive |
| VideoMAEv2 ViT-H | ~632M | 87% | Fine-tuned |
| **Ours** | 13.5M | **N/A** | Different modality (state vectors, not video) |

### Something-Something-v2 (Temporal Reasoning) → Top-1 Accuracy ↑

| Model | Accuracy |
|-------|----------|
| V-JEPA 2.1 ViT-G | **77.7%** |
| V-JEPA ViT-H/16 | 72.2% |
| **Ours** | **N/A** |

> **Note**: Our architecture operates on state vectors + bit-level memory, not pixels/video. The video benchmarks above are not directly applicable without adding a visual encoder front-end.

---

## 5. Architectural Comparison — Where We're Different

| Dimension | LeWM / JEPA Family | Our LeWorld Memory Architecture |
|-----------|-------------------|--------------------------------|
| **Memory** | Implicit (in network weights + latent state) | **Explicit bit-level RAM** (64K × 32-bit words, address-range access) |
| **State Prediction** | Single model predicts next embedding | **Hierarchical**: 3 SLMs find memory, 1 BLM aggregates + predicts |
| **Information Retrieval** | All in one forward pass | **Active retrieval**: BLM asks "what do I need?", SLMs fetch from memory |
| **Model Selection** | N/A (single model) | **Binary routing** [1,0,1]: BLM selects which SLMs to trust |
| **Collapse Prevention** | SIGReg (LeWM), EMA (V-JEPA), frozen encoder (DINO-WM) | **Diversity loss** + load-balance loss + temperature annealing |
| **Training** | Single loss (LeWM: 2 terms) | **3-phase**: pre-train → joint → info-request refinement |
| **Params** | 15M (LeWM), 632M (V-JEPA), 1.9B (V-JEPA 2.1) | **13.5M** (3×745K SLMs + 11.2M BLM) |
| **Input Modality** | Pixels / video frames | State vectors + characteristics (extensible to pixels with encoder) |
| **Planning** | CEM in latent space | BLM next-state prediction + info-request loop |
| **Gradient through discrete** | N/A (continuous latent space) | **ST-Sigmoid** for routing, **product-key CE** for addressing |

### What Our Architecture Adds That JEPA Doesn't Have

1. **Explicit addressable memory** — JEPA has no equivalent. All "memory" in JEPA is implicit in weights. Our architecture has a literal 256KB RAM that models can read/write to by address.

2. **Multi-agent retrieval** — 3 independent SLMs each search different memory regions. This is like having 3 specialized "attention heads" that look at different parts of a knowledge base, with a gating mechanism that selects the most useful ones.

3. **Active information request** — The BLM generates "what do I need next?" queries that influence what SLMs look for. JEPA models have no equivalent — they receive all information passively.

4. **CPU-inspired structure** — Address bus → RAM → data bus → processor pipeline mirrors actual computer architecture. This structural prior could help with systematic, compositional reasoning that neural networks typically struggle with.

---

## 6. What Benchmarks We Need to Run (Roadmap)

### Tier 1 — Must Run (direct comparison with LeWM/DINO-WM)

| Benchmark | What's Needed | Expected Difficulty |
|-----------|--------------|-------------------|
| **Push-T** | Add pixel encoder to our architecture, train on Push-T trajectories | Medium — need ~18K trajectories, visual encoder front-end |
| **PointMaze/Wall** | Same as above | Easy — simple navigation |
| **OGBench-Cube** | Same + 3D rendering | Medium-Hard |
| **Physical Probing** | Train linear/MLP probes on our latent space | Easy — we already have latent representations |
| **VoE (Violation of Expectation)** | Inject anomalies, measure surprise | Easy — our architecture naturally computes prediction error |

### Tier 2 — High-Impact Differentiators

| Benchmark | Why It Matters | Our Advantage |
|-----------|---------------|---------------|
| **IntPhys 2** | ALL JEPA models fail (≤57.5% vs 96.4% human) | Our explicit memory could help with object permanence |
| **Long-horizon planning** | JEPA models degrade over long rollouts | Our info-request loop provides feedback for multi-step |
| **Memory-dependent tasks** | Tasks requiring recall of past observations | **Direct advantage** — our architecture has literal memory |

### Tier 3 — Efficiency Benchmarks

| Metric | LeWM | Our Target |
|--------|------|-----------|
| Planning time | <1 second | Should be comparable (similar param count) |
| Training time | Single GPU, few hours | Same — 13.5M params |
| Training data efficiency | Scales with dataset size | To be measured |

---

## 7. Honest Assessment — Strengths and Weaknesses

### Where Our Architecture Should Excel (Hypotheses)

1. **Memory-dependent tasks**: Any task where the agent must remember and recall past observations to make current decisions. JEPA has no explicit memory — it's all in the latent state. Our 64K-word memory is persistent.

2. **Compositional state tracking**: Tasks with multiple objects where different "aspects" of the state need different information sources. Our 3 SLMs can specialize (one tracks the agent, one tracks the object, one tracks the environment).

3. **Anomaly detection / physics violation**: Our explicit memory + multi-step prediction error should catch "impossible" events better than implicit models. The info-request loop acts as an active hypothesis tester.

4. **Interpretability**: You can literally inspect which memory addresses were read, which SLMs were selected, what the info-request query was. JEPA is a black box.

### Where Our Architecture Will Likely Struggle

1. **Raw pixel processing**: Our current architecture works on state vectors. Adding a visual encoder is engineering work — but JEPA models are built pixel-first.

2. **Large-scale visual representation**: V-JEPA 2.1 at 1.9B params has seen millions of videos. Our 13.5M model can't compete on raw representation quality for visual tasks.

3. **Simple tasks**: LeWM already struggles on trivial environments (TwoRoom). Our more complex architecture might face similar issues — the overhead of memory + routing may not help when the task is simple.

4. **Training stability**: 3-phase training is more complex than LeWM's elegant 2-loss setup. More things can go wrong.

---

## 8. Comparison Summary Table

| | I-JEPA | V-JEPA | DINO-WM | LeWM | V-JEPA 2.1 | **Ours** |
|---|---|---|---|---|---|---|
| **Params** | 632M | 632M | ~300M | 15M | 1.9B | **13.5M** |
| **Input** | Images | Video | Pixels+Act | Pixels+Act | Video | State vec |
| **Memory** | None | None | None | None | None | **Explicit 256KB** |
| **Multi-model routing** | No | No | No | No | No | **Yes** |
| **Active info request** | No | No | No | No | No | **Yes** |
| **Push-T SR** | — | — | 0.90 | 0.88 | — | ❌ TBD |
| **Maze SR** | — | — | 0.98 | — | — | ❌ TBD |
| **Reach SR** | — | — | 0.92 | — | — | ❌ TBD |
| **IntPhys 2** | — | — | — | — | 57.5% | ❌ TBD |
| **K400 Acc** | — | 81.9% | — | — | ~88% | N/A |
| **Planning Speed** | — | — | ~48s | **<1s** | — | ~<1s (est) |
| **Training** | 16×A100 | Cluster | Offline | **1 GPU** | Cluster | **1 GPU** |
| **Interpretable** | Low | Low | Low | Low | Low | **High** |

---

## 9. Next Steps to Get Real Numbers

1. **Add visual encoder** to our architecture (small CNN or ViT-Tiny) for pixel observations → enables Push-T, Maze, Reach benchmarks
2. **Integrate with `stable-worldmodel` evaluation suite** (arxiv:2602.08968) for standardized comparison
3. **Run Push-T first** — most-used benchmark, open code, our architecture could show SLM specialization
4. **Design memory-dependent benchmark** — a custom task where agents MUST recall past observations to solve current goals. This is where we should clearly beat all JEPA models.

---

## 10. References

| Paper | ArXiv ID | Key Numbers |
|-------|----------|-------------|
| I-JEPA | 2301.08243 | ImageNet linear: 80%+, 632M params |
| V-JEPA | 2404.08471 | K400: 81.9%, SSv2: 72.2%, 632M params |
| DINO-WM | 2411.04983 | Push-T: 0.90 SR, Reach: 0.92 SR |
| LeWM | 2603.19312 | Push-T: 0.88 SR, 15M params, <1s planning, 48× faster |
| V-JEPA 2.1 | 2603.14482 | Ego4D: 7.71 mAP, SSv2: 77.7%, 1.9B params |
| IntPhys 2 | 2506.09849 | V-JEPA 2: 57.5%, Human: 96.4% |
| JEPA-WMs study | 2512.24497 | CEM best planner, proprioception critical |
| DreamerV3 | 2301.04104 | Atari SOTA, Push-T: 0.30 SR |
| LeCun position paper | 2306.02572 | Theoretical H-JEPA architecture |