BENCHMARK.md

LeWorld Memory Architecture vs. LeCun's World Models — Benchmark Comparison

⚠️ Honest Status: This Is a Theoretical Comparison

Our architecture has NOT been benchmarked on the standard world-model evaluation suites yet. What follows is:

1. A factual catalog of every LeCun-family world model, their exact published numbers, and benchmarks
2. An architectural comparison showing where our design is fundamentally different
3. A concrete plan for which benchmarks to run and what we'd need to demonstrate

We are not claiming results we don't have. Here's what exists and what's needed.

---

1. The LeCun World Model Family — Published Numbers

Model Lineup (by publication date)

Model	Paper	Params	Training	Key Innovation
I-JEPA	arxiv:2301.08243	632M (ViT-H/14)	ImageNet, 16×A100	Predicts masked image patch representations
V-JEPA	arxiv:2404.08471	632M (ViT-H/16)	VideoMix2M (2M videos)	Predicts masked video region features
DINO-WM	arxiv:2411.04983	~300M (frozen DINOv2 + predictor)	Offline trajectories	Plans in DINOv2 patch-feature space
LeWM	arxiv:2603.19312	~15M	Single GPU, few hours	End-to-end from pixels, 2 loss terms only
V-JEPA 2.1	arxiv:2603.14482	1.1–1.9B (ViT-g/G)	Massive video corpus	Dense features, multi-layer loss

LeWM is our primary comparison target (both ~15M params).

---

2. Published Benchmark Results — Robotic Planning

Push-T (Tabletop Block Pushing → Success Rate ↑)

Model	Params	Success Rate	Planning Time	Notes
DINO-WM	~300M+ (frozen DINOv2)	0.90	~48 seconds	Uses pretrained DINOv2 encoder
LeWM	~15M	0.88 (beats DINO-WM pixel-only)	<1 second	End-to-end from pixels, 48× faster
PLDM	~15M	0.70	<1 second	End-to-end, 7 loss terms (VICReg)
DreamerV3	12-400M	0.30	—	Model-based RL, needs rewards
IRIS	—	0.32	—	—
TD-MPC2	—	0.00	—	Fails without reward signal
Ours (LeWorld Memory)	13.5M	❌ Not yet tested	—	—

PointMaze / Wall Navigation → Success Rate ↑

Model	Maze SR	Wall SR	Notes
DINO-WM	0.98	0.96	Near-perfect
DreamerV3	1.00	1.00	Perfect on simple navigation
LeWM	Lower than DINO-WM	Lower	Struggles on very simple envs (SIGReg limitation)
Ours	❌ Not yet tested	❌ Not yet tested	—

Reach (Robotic Arm) → Success Rate ↑

Model	Success Rate
DINO-WM	0.92
DreamerV3	0.64
IRIS	0.18
Ours	❌ Not yet tested

Rope & Granular Manipulation → Chamfer Distance ↓

Model	Rope CD ↓	Granular CD ↓
DINO-WM	0.41	0.26
DreamerV3	2.49	1.05
IRIS	1.11	0.37
Ours	❌ Not yet tested	❌ Not yet tested

---

3. Published Benchmark Results — Physical Understanding

Physical Latent Probing on Push-T (Pearson r ↑, higher = better)

Property	DINO-WM (MLP)	LeWM (MLP)	PLDM (MLP)
Agent Location	r = 0.999	r = 0.998	r = 0.993
Block Location	r = 0.999	r = 0.999	r = 0.994
Block Angle	r = 0.995	r = 0.990	r = 0.972

LeWM at 15M achieves near-parity with DINO-WM (300M+ pretrained) on physical probing.

Violation-of-Expectation (VoE) — Physics Anomaly Detection

Perturbation	LeWM	PLDM
Teleportation (physically impossible)	Detects (p<0.01)	Detects
Color change (visual only)	Does NOT flag	Does NOT flag
Correct distinction?	✅ Yes	✅ Yes

IntPhys 2 (Intuitive Physics) — Accuracy ↑

Model	Accuracy	Notes
Human	96.4%	Ceiling
V-JEPA 2 (1B+ params)	57.5%	Best JEPA model
Gemini 2.5 Flash	55.6%	Best commercial LLM
GPT-4o	~50%	Near random
Ours	❌ Not yet tested	Potential differentiator (see below)

The IntPhys gap (57.5% vs 96.4% human) is the biggest open problem in world models.

---

4. Published Benchmark Results — Video Understanding

Kinetics-400 (Video Action Recognition) → Top-1 Accuracy ↑

Model	Params	Accuracy	Probe Type
V-JEPA 2.1 ViT-G	1.9B	~88%	Attentive
V-JEPA ViT-H/16	632M	81.9%	Frozen, attentive
VideoMAEv2 ViT-H	~632M	87%	Fine-tuned
Ours	13.5M	N/A	Different modality (state vectors, not video)

Something-Something-v2 (Temporal Reasoning) → Top-1 Accuracy ↑

Model	Accuracy
V-JEPA 2.1 ViT-G	77.7%
V-JEPA ViT-H/16	72.2%
Ours	N/A

Note: Our architecture operates on state vectors + bit-level memory, not pixels/video. The video benchmarks above are not directly applicable without adding a visual encoder front-end.

---

5. Architectural Comparison — Where We're Different

Dimension	LeWM / JEPA Family	Our LeWorld Memory Architecture
Memory	Implicit (in network weights + latent state)	Explicit bit-level RAM (64K × 32-bit words, address-range access)
State Prediction	Single model predicts next embedding	Hierarchical: 3 SLMs find memory, 1 BLM aggregates + predicts
Information Retrieval	All in one forward pass	Active retrieval: BLM asks "what do I need?", SLMs fetch from memory
Model Selection	N/A (single model)	Binary routing [1,0,1]: BLM selects which SLMs to trust
Collapse Prevention	SIGReg (LeWM), EMA (V-JEPA), frozen encoder (DINO-WM)	Diversity loss + load-balance loss + temperature annealing
Training	Single loss (LeWM: 2 terms)	3-phase: pre-train → joint → info-request refinement
Params	15M (LeWM), 632M (V-JEPA), 1.9B (V-JEPA 2.1)	13.5M (3×745K SLMs + 11.2M BLM)
Input Modality	Pixels / video frames	State vectors + characteristics (extensible to pixels with encoder)
Planning	CEM in latent space	BLM next-state prediction + info-request loop
Gradient through discrete	N/A (continuous latent space)	ST-Sigmoid for routing, product-key CE for addressing

What Our Architecture Adds That JEPA Doesn't Have

1. Explicit addressable memory — JEPA has no equivalent. All "memory" in JEPA is implicit in weights. Our architecture has a literal 256KB RAM that models can read/write to by address.

2. Multi-agent retrieval — 3 independent SLMs each search different memory regions. This is like having 3 specialized "attention heads" that look at different parts of a knowledge base, with a gating mechanism that selects the most useful ones.

3. Active information request — The BLM generates "what do I need next?" queries that influence what SLMs look for. JEPA models have no equivalent — they receive all information passively.

4. CPU-inspired structure — Address bus → RAM → data bus → processor pipeline mirrors actual computer architecture. This structural prior could help with systematic, compositional reasoning that neural networks typically struggle with.

---

6. What Benchmarks We Need to Run (Roadmap)

Tier 1 — Must Run (direct comparison with LeWM/DINO-WM)

Benchmark	What's Needed	Expected Difficulty
Push-T	Add pixel encoder to our architecture, train on Push-T trajectories	Medium — need ~18K trajectories, visual encoder front-end
PointMaze/Wall	Same as above	Easy — simple navigation
OGBench-Cube	Same + 3D rendering	Medium-Hard
Physical Probing	Train linear/MLP probes on our latent space	Easy — we already have latent representations
VoE (Violation of Expectation)	Inject anomalies, measure surprise	Easy — our architecture naturally computes prediction error

Tier 2 — High-Impact Differentiators

Benchmark	Why It Matters	Our Advantage
IntPhys 2	ALL JEPA models fail (≤57.5% vs 96.4% human)	Our explicit memory could help with object permanence
Long-horizon planning	JEPA models degrade over long rollouts	Our info-request loop provides feedback for multi-step
Memory-dependent tasks	Tasks requiring recall of past observations	Direct advantage — our architecture has literal memory

Tier 3 — Efficiency Benchmarks

Metric	LeWM	Our Target
Planning time	<1 second	Should be comparable (similar param count)
Training time	Single GPU, few hours	Same — 13.5M params
Training data efficiency	Scales with dataset size	To be measured

---

7. Honest Assessment — Strengths and Weaknesses

Where Our Architecture Should Excel (Hypotheses)

1. Memory-dependent tasks: Any task where the agent must remember and recall past observations to make current decisions. JEPA has no explicit memory — it's all in the latent state. Our 64K-word memory is persistent.

2. Compositional state tracking: Tasks with multiple objects where different "aspects" of the state need different information sources. Our 3 SLMs can specialize (one tracks the agent, one tracks the object, one tracks the environment).

3. Anomaly detection / physics violation: Our explicit memory + multi-step prediction error should catch "impossible" events better than implicit models. The info-request loop acts as an active hypothesis tester.

4. Interpretability: You can literally inspect which memory addresses were read, which SLMs were selected, what the info-request query was. JEPA is a black box.

Where Our Architecture Will Likely Struggle

1. Raw pixel processing: Our current architecture works on state vectors. Adding a visual encoder is engineering work — but JEPA models are built pixel-first.

2. Large-scale visual representation: V-JEPA 2.1 at 1.9B params has seen millions of videos. Our 13.5M model can't compete on raw representation quality for visual tasks.

3. Simple tasks: LeWM already struggles on trivial environments (TwoRoom). Our more complex architecture might face similar issues — the overhead of memory + routing may not help when the task is simple.

4. Training stability: 3-phase training is more complex than LeWM's elegant 2-loss setup. More things can go wrong.

---

8. Comparison Summary Table

	I-JEPA	V-JEPA	DINO-WM	LeWM	V-JEPA 2.1	Ours
Params	632M	632M	~300M	15M	1.9B	13.5M
Input	Images	Video	Pixels+Act	Pixels+Act	Video	State vec
Memory	None	None	None	None	None	Explicit 256KB
Multi-model routing	No	No	No	No	No	Yes
Active info request	No	No	No	No	No	Yes
Push-T SR	—	—	0.90	0.88	—	❌ TBD
Maze SR	—	—	0.98	—	—	❌ TBD
Reach SR	—	—	0.92	—	—	❌ TBD
IntPhys 2	—	—	—	—	57.5%	❌ TBD
K400 Acc	—	81.9%	—	—	~88%	N/A
Planning Speed	—	—	~48s	<1s	—	~<1s (est)
Training	16×A100	Cluster	Offline	1 GPU	Cluster	1 GPU
Interpretable	Low	Low	Low	Low	Low	High

---

9. Next Steps to Get Real Numbers

1. Add visual encoder to our architecture (small CNN or ViT-Tiny) for pixel observations → enables Push-T, Maze, Reach benchmarks
2. Integrate with stable-worldmodel evaluation suite (arxiv:2602.08968) for standardized comparison
3. Run Push-T first — most-used benchmark, open code, our architecture could show SLM specialization
4. Design memory-dependent benchmark — a custom task where agents MUST recall past observations to solve current goals. This is where we should clearly beat all JEPA models.

---

10. References

Paper	ArXiv ID	Key Numbers
I-JEPA	2301.08243	ImageNet linear: 80%+, 632M params
V-JEPA	2404.08471	K400: 81.9%, SSv2: 72.2%, 632M params
DINO-WM	2411.04983	Push-T: 0.90 SR, Reach: 0.92 SR
LeWM	2603.19312	Push-T: 0.88 SR, 15M params, <1s planning, 48× faster
V-JEPA 2.1	2603.14482	Ego4D: 7.71 mAP, SSv2: 77.7%, 1.9B params
IntPhys 2	2506.09849	V-JEPA 2: 57.5%, Human: 96.4%
JEPA-WMs study	2512.24497	CEM best planner, proprioception critical
DreamerV3	2301.04104	Atari SOTA, Push-T: 0.30 SR
LeCun position paper	2306.02572	Theoretical H-JEPA architecture