PAPER.md

# Beyond Linear Neurons: An Empirical Study of Multiplicative Periodic Architectures at Small Scale

**Authors:** anshdadhich, with adversarial review from two LLM collaborators

**Repository:** [huggingface.co/anshdadhich/richneuron-vs-vanilla-benchmark](https://huggingface.co/anshdadhich/richneuron-vs-vanilla-benchmark)

---

## Abstract

We investigate whether replacing the standard neuron computation `y = ReLU(Wx + b)` with richer per-neuron functions can increase information storage and accuracy at fixed parameter budgets. Through 15 architecture iterations tested across 9 tasks (regression, classification, memorization, frequency generalization, and out-of-distribution), we find that **multiplicative periodic neurons** (`sin(ω·W₁x) ⊙ W₂x`, i.e., SinGLU) consistently outperform vanilla MLPs by 4-168,327× on structured tasks. We then systematically test 10 adaptive mechanisms (routing, learnable frequency, free phase, scaled phase, aligned phase, frequency modulation, dual-phase decomposition) attempting to improve upon SinGLU. All fail to consistently beat it at small scale (3K-8K parameters). We identify the root cause: every parameter spent on meta-computation (deciding *how* to compute) is stolen from actual computation, and the meta-learning signal is too weak at small scale to justify the cost. Our killer experiments reveal that fixed-frequency architectures generalize better to unseen frequencies than adaptive ones — directly contradicting the intuition that more expressive neurons generalize better.

---

## 1. Introduction

### 1.1 The Question

A standard neural network neuron computes `y = σ(Wx + b)` — a linear transformation followed by a fixed nonlinearity. Each weight parameter participates in exactly one multiply-add operation. We ask: **can replacing this with a richer computation store more information per parameter and achieve better accuracy without increasing total parameter count?**

This question is motivated by recent theoretical results showing that standard transformers store approximately 2 bits of knowledge per parameter ([Allen-Zhu & Li, 2024](https://arxiv.org/abs/2404.05405)), and by architectures like KAN ([Liu et al., 2024](https://arxiv.org/abs/2404.19756)), SIREN ([Sitzmann et al., 2020](https://arxiv.org/abs/2006.09661)), and MONet ([Chrysos et al., 2024](https://arxiv.org/abs/2401.17992)) that propose richer neuron computations.

### 1.2 Approach

We conduct a systematic architecture search, starting from simple modifications and iterating based on empirical results and adversarial critique. Each version tests a specific hypothesis:

| Version | Hypothesis | Architecture |
|---------|-----------|-------------|
| v1 | Multiplicative + periodic > linear | `(W₁x) ⊙ sin(ω·W₂x) + W₁x` |
| v4 | Width penalty can be eliminated | Low-rank, shared-weight, GLU-style variants |
| v5 | Honest multi-seed re-evaluation | 3 seeds, gradient norms, OOD |
| v6 | Adaptive routing (α) can select computation type | `α·periodic + (1-α)·linear` |
| v7 | Learnable frequency adapts per input | `sin(ω(x)·Wx)` |
| v8 | Phase + gate replaces frequency | `sin(ω·Wx + φ(x))` with sigmoid gate |
| v9 | Controlled frequency + phase + gate | Bounded ω(x) + φ(x) + α(x) |
| v10 | Minimal: SinGLU + free phase only | `sin(ω·Wg·x + π·tanh(Wφ·x))` |
| v11 | Disciplined phase (scaled down) | `sin(ω·(Wg·x + 0.1·tanh(Wφ·x)))` |
| v12 | Signal-proportional phase | `sin(ω·g·(1 + 0.2·tanh(Wφ·x)))` |
| v13 | Signal-aligned phase | `sin(ω·g + 0.1·g·tanh(Wφ·x))` |
| v15 | Dual-frequency decomposition | `sin(ωg+βφ) ⊙ (1 + α·sin(2ωg+γφ))` |

All comparisons use **strictly matched parameter budgets** via binary search over hidden dimensions.

---

## 2. Experimental Setup

### 2.1 Tasks

We use 9 tasks spanning different computational demands:

**Regression (lower MSE = better):**
- **Complex Fn (4D):** `f(x) = exp(sin(x₁²+x₂²) + sin(x₃²+x₄²))` — compositional structure ([from KAN paper](https://arxiv.org/abs/2404.19756))
- **Nested Fn (2D):** `f(x) = sin(π(x₁²+x₂²))·cos(3π·x₁x₂)` — multiplicative + periodic
- **High-Frequency Signal:** `f(x) = sin(20x) + sin(50x) + 0.5·sin(100x)` — pure frequency representation
- **Knowledge Memorization:** 200 random 8D→4D mappings — raw storage capacity test

**Classification (higher accuracy = better):**
- **Two-Spiral:** Interleaving spirals — nonlinear decision boundaries
- **Checkerboard (freq=3):** Feature interaction pattern

**Generalization:**
- **OOD:** Train on `[-1,1]`, test on `[1,2]` for `f(x₁,x₂) = sin(3πx₁)·cos(3πx₂) + x₁x₂`
- **Frequency Generalization:** Train on `sin(2πx)`, test on `sin(10πx)` — can the model represent unseen frequencies?
- **Mixed Frequency:** Train on `sin(2πx)+sin(4πx)`, test on `sin(2πx)+sin(20πx)` — can it decompose and generalize frequency components?

### 2.2 Protocol

- **Parameter matching:** Binary search finds hidden dimension giving closest match to target budget per architecture
- **3 random seeds** per experiment, reporting mean±std
- **Optimizer:** Adam with cosine annealing LR schedule
- **Gradient clipping:** max norm 1.0
- **Parameter budgets:** 3K-8K depending on task input dimensionality

### 2.3 Baselines

- **Vanilla MLP:** `Linear → ReLU → Linear → ReLU → Linear → ReLU → Linear`
- **SinGLU:** `sin(ω·Wg·x) ⊙ Wv·x` projected through `Wo` with LayerNorm. Same structure as SwiGLU ([Shazeer, 2020](https://arxiv.org/abs/2002.05202)) but with `sin()` instead of `Swish()`. Uses 2/3 hidden dim trick to match param count.

---

## 3. Results

### 3.1 Core Finding: Multiplicative Periodic Neurons Beat Vanilla (v1-v5)

On matched parameter budgets, SinGLU consistently outperforms Vanilla MLP:

| Task | Vanilla | SinGLU | Improvement |
|------|---------|--------|-------------|
| Nested Fn | 0.0487 | **0.0002** | **222×** |
| Memorization | 0.1568 | **9.3e-7** | **168,327×** |
| Complex Fn | 0.0575 | **0.0143** | **4.0×** |
| Checkerboard | 57.9% | **93.8%** | +35.9 pts |
| High-Freq | 1.10 | **1.02** | 1.08× |
| Spiral | **85.1%** | 44.2% | Vanilla wins |

**SinGLU wins 5/6 standard tasks.** The gains on memorization (168K×) and nested function (222×) are not incremental — they demonstrate a fundamentally different information encoding capacity. The sole loss (Spiral) is due to SinGLU's fixed-frequency basis being unable to form the specific nonlinear decision boundary spirals require.

### 3.2 The Width-Richness Tradeoff (v4)

SinGLU uses 3 matrices per layer (Wg, Wv, Wo) vs Vanilla's 1 (W). At matched param budgets, SinGLU gets ~65% the hidden width (e.g., 37→24, 62→41 across tasks). This is acceptable because SinGLU's per-neuron computation is richer, but it creates a fundamental tension: **every additional matrix for adaptive mechanisms further reduces width.**

### 3.3 Adaptive Mechanisms: Systematic Failure (v6-v13)

We tested 8 different adaptive mechanisms. Results:

| Mechanism | Version | Wins vs SinGLU | Root Cause of Failure |
|-----------|---------|----------------|----------------------|
| Sigmoid routing (α) | v6 | 0 | α stuck near 0.5 — gradient competition |
| Learnable frequency ω(x) | v7 | 0 | ω froze at initialization — oscillatory gradient |
| Phase + gate | v8 | 0 | Gate weak, phase underused |
| Controlled freq + phase + gate | v9 | 1 (Spiral) | 5 matrices → hidden dim 20 vs SinGLU's 31 |
| Free phase | v10 | 2 (Complex, Spiral) | Destroyed HiFreq and OOD |
| Tiny phase (0.1 scale) | v11 | 2 (Complex, Spiral) | Phase std ~0.007 — effectively zero |
| Signal-proportional (FM) | v12 | 3 (Complex, Spiral, Checker) | Actually frequency modulation, not phase |
| Signal-aligned | v13 | 2 (Complex, Checker) | Killed Spiral — phase must be orthogonal to signal for geometry |

**Common pattern across all adaptive versions:**

1. **Meta-learning signal is too weak.** The gradient signal for "how should I compute" is second-order — it depends on how well the branches are already performing. The branches learn useful features via direct gradients, while the routing/gating/frequency mechanism receives indirect signal that's too small to overcome initialization.

2. **Parameter overhead kills width.** Each adaptive matrix reduces hidden dimension by ~4 units at 3K-5K budget. This is a 10-20% capacity loss that the adaptive mechanism never recovers.

3. **Gradient analysis confirms instability.** From v5's gradient norm tracking:

| Model | Epoch 0 | Epoch 200 | Epoch 400 | Epoch 600 | Epoch 1000 |
|-------|---------|-----------|-----------|-----------|------------|
| Vanilla | 0.64 | 0.33 | 0.23 | **0.28*** | 0.16 |
| SinGLU | 19.5 | 14.9 | 5.1 | 1.3 | 0.4 |
| Shared (S2) | **1159** | **884** | **904** | **714** | **174** |

*Vanilla gradient briefly rises at epoch 600 before continuing decay.

### 3.4 Dual-Phase Decomposition: Wins High-Freq (v15)

v15 introduces explicit multi-scale structure:

```
low  = sin(ω·g + β·φ)           # structure channel
high = sin(2ω·g + γ·φ)          # detail channel  
core = low ⊙ (1 + 0.3·high)    # AM modulation
```

**This is the first and only architecture to beat SinGLU on High-Frequency Signal** — MSE 0.854 vs 1.017. The dual-frequency basis provides the neuron simultaneous access to ω and 2ω, which is exactly what a sum-of-sinusoids signal needs. However, v15 loses on most other tasks because the AM modulation adds nonlinear coupling that hurts simpler problems.

### 3.5 Killer Experiments: Frequency Generalization

The most revealing experiments test whether models can generalize to unseen frequencies:

**Experiment 1: Train sin(2πx) → Test sin(10πx)**

| Model | Train MSE | Test MSE (unseen 10πx) | Gap |
|-------|-----------|----------------------|-----|
| Vanilla | 0.365 | 1.172 | 3.2× |
| **SinGLU** | 2.166 | **0.736** | **0.3×** ← better on test than train |
| v10 | 0.969 | 0.958 | 1.0× |
| v15 | 0.718 | 0.910 | 1.3× |

**SinGLU shows the best frequency generalization despite worst training fit.** Its fixed-frequency basis acts as an inductive bias that transfers across frequency scales. The adaptive variants (v10, v15) overfit the training frequency and generalize worse.

**Experiment 2: Train sin(2πx)+sin(4πx) → Test sin(2πx)+sin(20πx)**

| Model | Train MSE | Test MSE (unseen 20πx mix) |
|-------|-----------|---------------------------|
| Vanilla | 0.882 | 1.329 |
| SinGLU | 4.648 | 1.491 |
| **v10** | 1.818 | **1.178** |
| v15 | 2.076 | 1.317 |

v10's free phase helps decompose mixed frequency components — but the margins are small and within noise.

### 3.6 OOD Generalization

All periodic architectures fail on out-of-distribution data (train [-1,1] → test [1,2]):

| Model | ID MSE | OOD MSE | Degradation |
|-------|--------|---------|-------------|
| **Vanilla** | 0.217 | **1.53** | **7.1×** |
| SinGLU | 0.246 | 5.90 | 24.0× |
| v10 | 0.004 | 4.96 | 1,273× |
| v15 | 0.010 | 4.38 | 420× |

**Vanilla's OOD robustness is unmatched.** Periodic activations extrapolate their learned oscillations outside the training domain, producing hallucinated patterns. This is a fundamental limitation of sinusoidal representations, not an implementation issue.

---

## 4. Analysis

### 4.1 Why SinGLU Wins

SinGLU's dominance can be attributed to three factors:

1. **100% of parameters do computation.** No routing, gating, or frequency prediction matrices. Every parameter directly encodes features.

2. **Multiplicative interaction captures cross-terms.** `sin(ω·W₁x) ⊙ W₂x` produces terms of the form `sin(ω·wᵢᵀx)·wⱼᵀx`, which includes the product `xₖ·xₗ` that a linear layer cannot represent. This is the same insight behind GLU variants in modern LLMs.

3. **Fixed frequency is a feature, not a bug.** Fixed ω provides a consistent frequency basis that transfers across inputs and even across frequency scales (as shown in the killer experiment). Adaptive frequency mechanisms add flexibility but lose this consistency.

### 4.2 The Regime Map

Our experiments reveal three distinct computational regimes:

| Regime | Best Architecture | Why |
|--------|------------------|-----|
| Structured functions (compositional, multiplicative) | SinGLU or v10 (free phase) | Periodic basis + cross-terms match function structure |
| Geometric decision boundaries (spirals, nonlinear classification) | v10 (free phase) | Phase shifts rotate decision boundaries |
| Multi-scale signals (sum of sinusoids) | v15 (dual-phase) | Explicit access to multiple frequency channels |
| Out-of-distribution robustness | Vanilla MLP | Simplicity = less overfitting to training distribution |
| Frequency generalization (unseen frequencies) | SinGLU | Fixed frequency basis transfers; adaptive basis overfits |

**No single architecture dominates all regimes.** This is consistent with the No Free Lunch theorem — every inductive bias that helps on one task class necessarily hurts on another.

### 4.3 Insights on Neural Network Optimization

Our adaptive mechanism experiments (v6-v13) revealed a consistent failure pattern that constitutes a finding in its own right:

> **Neural networks refuse to learn meta-computation when direct computation is available.**

Specifically:
- **Routing gates (v6):** α stays near 0.5 (mean 0.45–0.51, std ~0.05) — the network adjusts branch weights instead of the gate.
- **Learnable frequency (v7):** ω stays at initialization — the network adjusts W_per instead of ω.
- **Phase predictors (v8-v13):** Phase learns small perturbations at best — the network adjusts Wg instead of Wφ.

The root cause is **gradient competition**: meta-parameters receive second-order gradient signal (how changing the computation type would improve the already-optimized branches), while branch parameters receive first-order signal (how to directly reduce loss). At small scale with limited training, first-order always wins.

This parallels known results in meta-learning, neural architecture search, and mixture-of-experts, where explicit auxiliary losses (load balancing, architecture reward) are required to train the meta-mechanism.

---

## 5. Conclusions

### 5.1 Confirmed

1. **Replacing y = ReLU(Wx + b) with richer per-neuron computation increases information density.** The memorization test showed 168,327× lower MSE at matched parameters — each parameter encodes dramatically more information when participating in multiplicative periodic computation.

2. **SinGLU (`sin(ω·W₁x) ⊙ W₂x`) is the optimal neuron design at small scale.** It wins 4-5 out of 9 tasks consistently across all comparisons. The 2/3 width trick from the GLU literature makes it parameter-efficient.

3. **Different tasks favor different neuron types.** Geometric tasks favor free phase (v10), multi-scale signal tasks favor dual-phase (v15), and OOD robustness favors vanilla ReLU. This is a spectrum, not a single optimum.

### 5.2 Refuted

4. **Adaptive per-neuron computation does not pay at small scale (3K-8K params).** Every adaptive mechanism tested (8 variants) either matched or underperformed SinGLU. The meta-learning signal is too weak relative to direct weight learning.

5. **More expressive neurons do NOT generalize better to unseen frequencies.** The killer experiment showed that fixed-frequency SinGLU generalizes better than adaptive variants — directly contradicting the intuition that expressiveness aids generalization.

6. **Periodic activations do NOT improve OOD robustness.** All sinusoidal architectures showed 24-1273× degradation on OOD data, vs Vanilla's 7×. Periodic neurons hallucinate oscillations outside the training domain.

### 5.3 Open Questions

- Do the adaptive mechanisms (v6-v13) that failed at small scale succeed at 100K+ parameters where the width penalty becomes negligible?
- Can explicit auxiliary losses (analogous to MoE load balancing) make phase/frequency prediction trainable?
- Does v15's dual-phase decomposition scale to real signal processing tasks (audio, images)?

---

## 6. Reproducibility

All code is available at [huggingface.co/anshdadhich/richneuron-vs-vanilla-benchmark](https://huggingface.co/anshdadhich/richneuron-vs-vanilla-benchmark):

| File | Description |
|------|-------------|
| `benchmark.py` | v1: Original RichNeuron vs Vanilla |
| `benchmark_v4.py` | v4: Width-fix strategies (LowRank, Shared, SinGLU) |
| `benchmark_v5.py` | v5: Honest re-eval (multi-seed, grad norms, OOD) |
| `benchmark_v6.py` | v6: Adaptive routing neuron |
| `benchmark_v7.py` | v7: Learnable frequency neuron |
| `benchmark_v8.py` | v8: Adaptive phase + amplitude gate |
| `benchmark_v9.py` | v9: Controlled frequency + phase + gate |
| `benchmark_v10.py` | v10: SinGLU + free phase |
| `benchmark_v11.py` | v11: SinGLU + disciplined phase |
| `benchmark_v12.py` | v12: SinGLU + signal-proportional phase (FM) |
| `benchmark_v13.py` | v13: SinGLU + aligned phase + corr(g,φ) analysis |
| `benchmark_v15.py` | v15: Dual-phase decomposition + killer experiments |
| `results_*.json` | Raw results with per-seed scores |
| `PAPER.md` | Full technical report |
| `FINDINGS_SUMMARY.md` | Complete architecture catalog and results |
| `CORRECTIONS.md` | Data verification and audit trail |

All experiments run on CPU with PyTorch. Total compute: ~4 hours on a 2-vCPU machine.

---

## References

- Allen-Zhu, Z., & Li, Y. (2024). Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. *arXiv:2404.05405*
- Liu, Z., et al. (2024). KAN: Kolmogorov-Arnold Networks. *arXiv:2404.19756*
- Sitzmann, V., et al. (2020). Implicit Neural Representations with Periodic Activation Functions. *NeurIPS 2020*. *arXiv:2006.09661*
- Chrysos, G., et al. (2024). Multilinear Operator Networks. *ICLR 2024*. *arXiv:2401.17992*
- Shazeer, N. (2020). GLU Variants Improve Transformer. *arXiv:2002.05202*
- Hoff, S., et al. (2024). Efficient Learning with Sine-Activated Low-rank Matrices. *arXiv:2403.19243*
- Xu, J., et al. (2024). Densing Law of LLMs. *arXiv:2412.04315*
- Cho, Y., et al. (2022). FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning. *ICLR 2022*. *arXiv:2108.06098*