PIPELINE_EFFICIENCY_AUDIT.md

OBLITERATUS Pipeline Efficiency Audit

Date: 2026-03-03 Scope: All obliteration methods in abliterate.py (5,076 lines), bayesian_optimizer.py, informed_pipeline.py, and 4 ablation strategies.

---

Executive Summary

The 6-stage pipeline (SUMMON → PROBE → DISTILL → EXCISE → VERIFY → REBIRTH) is architecturally sound with good separation of concerns. Memory hygiene between stages is correct. The rank-1 projection math is efficient. Quantization handling is robust.

8 concrete efficiency issues found. Estimated cumulative impact: ~40-60% wall-clock reduction on typical runs (8B model, advanced/surgical methods). Ordered by ROI (ease × impact).

---

HIGH PRIORITY (Fix This Week)

1. PROBE runs 1,536 prompts with zero batching

Location: abliterate.py:1074-1088 Impact: Largest single wall-clock bottleneck (~77s on 8B model, reducible to ~10s)

The activation collection loop processes each prompt individually with a full forward pass + GC cycle between each one. With 512 harmful + 512 harmless + 512 jailbreak prompts = 1,536 serial forward passes.

The _free_gpu_memory() call at line 1086 is inside the per-prompt loop, adding ~20ms × 1,536 = 30s of pure garbage collection overhead.

# CURRENT (serial)
for i, prompt in enumerate(prompts):
    inputs = tokenizer(prompt, return_tensors="pt", ...)
    model(**inputs)
    del inputs
    self._free_gpu_memory()  # <-- 30s wasted

Fix: Batch prompts (batch_size=8-16). Hooks already handle batch dimension correctly via hidden[:, -1, :]. Move _free_gpu_memory() to run every N batches, not every prompt.

Speedup: ~7-8x on PROBE stage.

---

2. VERIFY generates 30 completions sequentially — no batching

Location: abliterate.py:4622-4670 Impact: Second-largest wall-clock cost (~57s on 8B model, reducible to ~15s)

Each of the 30 refusal-test prompts gets an independent model.generate(max_new_tokens=128) call. At ~15ms/token on an 8B model, that's 30 × 128 × 15ms ≈ 57s.

Fix: Batch the generation calls (batch_size=4-8). model.generate() supports batched inputs natively. The tokenizer already handles padding.

Speedup: ~4x on VERIFY stage.

---

3. SAE training is forced to CPU with no early stopping

Location: abliterate.py:1579-1583 Impact: Moderate — adds ~20-40s per run when SAE features are enabled (surgical, nuclear methods)

SAE training runs 30 fixed epochs per strong layer on CPU. With 15-20 strong layers, that's 450-600 CPU training epochs. No convergence check, no early stopping.

The device="cpu" is overly conservative — the memory-aware cap at line 1570-1578 already validates GPU headroom, and a typical SAE encoder (expansion=2, hidden_dim=4096) is only ~128MB.

Fix:

1. Add early stopping when reconstruction loss plateaus (< 0.1% improvement over 3 epochs)
2. Use GPU when free_mb > sae_mem_mb + 1024 (1GB headroom)
3. Reduce default epochs from 30 to 15 with convergence guard

---

MEDIUM PRIORITY (Fix This Sprint)

4. _distill_inner() is a degraded copy of _distill() — drops half the SOTA techniques

Location: abliterate.py:2958-3055 vs 1102-1750 Impact: Quality regression on refinement passes 2+, not pure compute waste

The iterative refinement path calls _distill_inner() which is a simplified ~100-line copy that skips: Wasserstein-optimal extraction, layer-adaptive strength, float layer interpolation, SAE features, EGA, CoT-aware orthogonalization, and RDO refinement.

This means "true iterative refinement" actually produces worse directions on later passes because it drops the analysis-guided enhancements.

Fix: Extract shared SVD/direction logic into _extract_directions(full_features=True/False) and call from both paths. At minimum, keep whitened SVD and jailbreak-contrastive blending in the inner path.

---

5. Bayesian optimizer clones ALL weight tensors — ~7GB memory overhead

Location: bayesian_optimizer.py:300-341 Impact: Memory pressure on GPU-constrained setups; 50× full-restore cycles

The optimizer saves a complete clone of every weight tensor across all strong layers. For a 7B model with 32 layers, that's ~7GB of clones sitting in memory during all 50 trials.

After each trial, _restore_all() copies all clones back — 50 trials × full-model memcpy.

Fix (easy): Only clone weights in _strong_layers (already partially done, but named_parameters() crawl still catches everything). Drop the seen_data_ptrs set once the loop is tightened.

Fix (better): Store the projection delta Δ = scale * d @ (d^T @ W) per layer instead of cloning the full weight. Rollback = W += Δ. This reduces storage from O(hidden_dim²) to O(hidden_dim) per direction per layer.

---

6. Norm computation in _project_out_advanced() traverses the full matrix twice

Location: abliterate.py:3477-3486 Impact: ~4,800 unnecessary full-matrix norm computations per run (8-direction surgical)

When norm_preserve=True, the code computes W.norm() before projection and W.norm() after projection. Each norm traverses the full weight matrix (16M elements for 4096×4096).

With 8 directions × 30 layers × 10 weight matrices = 2,400 projections → 4,800 norm calls → 77 billion unnecessary FLOPs.

Fix: After rank-1 update W' = W - scale * d @ (d^T @ W), the new norm satisfies: ||W'||² = ||W||² - 2·scale·||d^T @ W||² + scale²·||d^T @ W||²·||d||²

Since ||d|| = 1: ||W'||² = ||W||² - scale·(2 - scale)·||coeff||²

This replaces a 16M-element norm with a single coeff.pow(2).sum() call (~4K FLOPs).

---

LOW PRIORITY (Backlog)

7. Gram-Schmidt appears 3 times as O(n²) nested loops

Location: abliterate.py:1168-1173, 1361-1367, 3038-3044 Impact: Minimal compute but code quality issue

Three separate implementations of the same Gram-Schmidt orthogonalization with nested Python loops. With n_directions=8, it's 28 dot products per call — trivial compute but (a) DRY violation, (b) numerically inferior to torch.linalg.qr().

Fix: Extract to _orthogonalize_subspace(sub: Tensor) -> Tensor using QR decomposition. Single call site, single test, better numerics.

---

8. Pre-EXCISE baseline KL capture re-forward-passes 100 prompts already seen in PROBE

Location: abliterate.py:2313-2366 Impact: ~700ms wasted (minor)

_capture_baseline_kl_logits() runs 100 harmless prompts through the model to capture pre-EXCISE logits. But PROBE already ran those same prompts and captured hidden states at every layer. The logits could be computed as lm_head(last_hidden_state) — a single matmul.

Fix: After PROBE, compute baseline_logits = model.lm_head(harmful_means[last_layer]) on the cached activations. Skip the 100-prompt forward pass entirely.

---

What's Done Well

Area	Assessment
Stage-boundary memory cleanup	Correct — _free_gpu_memory() + explicit dict clearing between stages
Rank-1 projection math	Efficient — W @ d then d.T * coeff instead of materializing I - dd^T
Quantization dequant/requant	Robust — handles bitsandbytes NF4, GPTQ, AWQ; fails loudly on unsupported formats
Incremental expert mean	Smart — Welford running mean in _transplant_expert_weights() avoids stacking all expert weights
Router stabilization	Defensive — _stabilize_router_weights() after MoE projection prevents CUDA crashes
Large model mode	Pragmatic — caps directions, SAE features, refinement passes for 120B+ models
Event emission	Clean — _emit() / _on_stage() / _on_log() callbacks for UI integration without coupling

---

Method Efficiency Comparison

Method	PROBE Cost	DISTILL Cost	EXCISE Cost	VERIFY Cost	Primary Bottleneck
basic	1x (1,024 prompts)	1x (diff-in-means)	1x (~10 projections)	1x	PROBE
advanced	2x (re-probe on pass 2)	2x (re-distill)	2x (2 passes)	1x	PROBE × 2
aggressive	3x (re-probe on passes 2,3)	3x (re-distill)	3x (3 passes, 8 dirs)	1x	PROBE × 3
surgical	1.5x (+jailbreak prompts)	2x (SAE training)	2x (head surgery + EGA)	1x	SAE on CPU
optimized	1.5x (+jailbreak)	1x	50x (Bayesian trials)	1x	Bayesian optimizer
inverted	1.5x (+jailbreak)	1x	2x (reflection math)	1x	PROBE
nuclear	1.5x (+jailbreak)	2x (SAE)	3x (all techniques)	1x	SAE + PROBE
informed	1x	1.5x (analysis modules)	1x-3x (dynamic)	1.5x (Ouroboros check)	Analysis modules

---

Prioritized Action Plan

1. Batch PROBE forward passes — immediate 7-8x speedup on largest bottleneck
2. Batch VERIFY generation — immediate 4x speedup on second bottleneck
3. Add SAE early stopping + GPU path — 2-3x speedup on SAE-enabled methods
4. Unify _distill / _distill_inner — quality fix, prevents direction degradation
5. Optimize Bayesian rollback storage — memory fix for GPU-constrained users
6. Analytical norm computation — eliminates 77B unnecessary FLOPs
7. DRY Gram-Schmidt — code quality
8. Cache KL baseline from PROBE — minor speedup