# Preliminary TernaryScale Benchmark Results

**Date:** 2025-05-15  
**Hardware:** RTX 4060 8GB  
**Steps:** 2500 | **Warmup:** 250 | **Batch:** 64 | **CTX:** 66  
**Params:** 1,668,128 | **Ternary forward:** T32  
**Data:** enwik8 (train 1,003,854 bytes / val 111,540 bytes)  
**All 6 configs run in parallel per step**

## Loss Over Time

| Step | SignSGD-C | SignSGD-E | Lion bf16 | Lion FP32 | Adam bf16 | Adam FP32 |
|------|-----------|-----------|-----------|-----------|-----------|-----------|
| 0    | 6.9770    | 8.0108    | 7.0011    | 6.5737    | 6.7317    | 6.8118    |
| 250  | 2.1486    | 1.9869    | 3.0933    | 3.1620    | 2.8921    | 2.7365    |
| 500  | 1.9801    | 1.7453    | 2.8211    | 2.5914    | 2.5739    | 2.4437    |
| 750  | 1.8682    | 1.6757    | 2.6109    | 2.5917    | 2.3861    | 2.2972    |
| 1000 | 1.8499    | 1.6601    | 2.6047    | 3.0808    | 2.3597    | 2.2618    |
| 1250 | 1.7943    | 1.6378    | 2.5462    | 3.2841    | 2.3527    | 2.3462    |
| 1500 | 1.7533    | 1.6506    | 2.5942    | 3.3279    | 2.4036    | 2.4838    |
| 1750 | 1.7601    | 1.6370    | 2.5652    | 3.3534    | 2.3876    | 2.5985    |
| 2000 | 1.7102    | 1.6108    | 2.5563    | 3.3116    | 2.3305    | 2.6721    |
| 2250 | 1.7354    | 1.5892    | 2.6064    | 3.4192    | 2.3509    | 2.7199    |
| 2500 | 1.6944    | 1.5861    | 2.5486    | 3.4403    | 2.3237    | 2.6991    |

## Summary

| Config | Final Loss | Min Loss | Loss@1k | Loss@2.5k | Step(ms) | Opt Mem | Total Mem |
|--------|-----------|----------|---------|------------|----------|---------|-----------|
| **SignSGD ConfigC** | 1.7113 | 1.5921 | 1.8099 | 1.6944 | 11.2 | 6.4 MB | 12.7 MB |
| **SignSGD ConfigE** | **1.6149** | **1.5157** | **1.7146** | **1.5861** | 11.2 | 6.4 MB | 12.7 MB |
| Lion bf16 | 2.5521 | 2.4668 | 2.5683 | 2.5486 | 11.2 | 9.6 MB | 12.7 MB |
| Lion FP32 | 3.4280 | 2.4911 | 3.0947 | 3.4403 | 11.2 | 12.7 MB | 19.1 MB |
| Adam bf16 | 2.3487 | 2.2480 | 2.3016 | 2.3237 | 11.2 | 9.6 MB | 12.7 MB |
| Adam FP32 | 2.7597 | 2.2049 | 2.3037 | 2.6991 | 11.2 | 19.1 MB | 25.4 MB |

## Loss Ratio vs SignSGD ConfigC

| Config | Ratio | Verdict |
|--------|-------|---------|
| SignSGD ConfigE | **0.944x** | better |
| Lion bf16 | 1.491x | worse |
| Lion FP32 | 2.003x | worse |
| Adam bf16 | 1.373x | worse |
| Adam FP32 | 1.613x | worse |

## Peak VRAM: 388 MB (all 6 models combined)

## Key Findings

1. **SignSGD ConfigE wins** — lowest loss (1.6149), lowest memory (12.7 MB total), no optimizer state
2. **SignSGD ConfigC second** — group-averaged S blurs magnitudes vs ConfigE's per-element S
3. **Momentum optimizers diverge on ternary** — Lion FP32 and Adam FP32 loss increases after step 1000
4. **bf16 helps momentum optimizers** — Lion bf16 (2.55) and Adam bf16 (2.35) are more stable than their FP32 variants (3.43, 2.76) but still far worse than SignSGD
5. **Root cause:** Momentum amplifies gradients that get blocked by ternary STE threshold mask, causing oscillation. SignSGD's pure sign update naturally complements ternary training — no amplification, no conflict.
6. **SignSGD memory advantage:** 6.4 MB optimizer mem vs 19.1 MB (Adam FP32) — 3x savings, zero state buffers