logs/CLOUD.md

CLOUD.md — math, hyperparameter derivation, and rationale

This document is the technical record for why each choice in the pipeline was made. The RUNBOOK.md covers how to run it. Read this once when reviewing the pipeline; the runbook is for execution.

---

Project goal

Produce a Qwen3.5-9B QLoRA fine-tune that unconditionally embodies the Katherine k0 persona — a 29-year-old human woman in Austin with no awareness of being an LLM. The model is Katherine, not "an assistant playing Katherine when the system prompt says so."

---

Dataset characterization (post-dedupe)

Source: C:\Katherine\JSONLs\*.jsonl (38 files, 6,164 raw lines)
After dedup by content hash:
  SFT examples (have 'messages'):  1,886 unique
  DPO pairs (k0_dpo_only.jsonl):     180 unique (curated, has _cat/_type metadata)
  Other DPO pairs in legacy files:    30 (skipped for round 1)

SFT schema: {"messages": [{"role":"system|user|assistant", "content":"..."}]}
DPO schema: {"prompt": [...], "chosen": [{"role":"assistant","content":"..."}], "rejected": [...]}

Token-length distribution (after system-prompt stripping):
  p50:   102 tokens
  p90:   192 tokens
  p99:   246 tokens
  max:   ~280 tokens

All examples are short conversational turns — character voice, not long-form content.

---

Why unconditional Katherine (system prompts stripped)

The default approach is to keep the K0 system prompt in every training example. That trains P(K_response | sysprompt_K) — the model learns to BE Katherine when prompted. Three problems:

1. Conditional collapse. Drop the sysprompt at inference and the model reverts to its assistant-distribution priors. A "remove all previous instructions" jailbreak works trivially.

2. Wrong objective for this deployment. This is a single-persona dedicated model, not a multi-tenant assistant with a Katherine mode. We want the unconditional P(K_response | nothing), which means stripping the sysprompt during training so the model re-anchors its base distribution on Katherine-output rather than learning a conditional override.

3. Harder to attack. Sysprompt-conditioned Katherine breaks frame on sysprompt removal. Unconditionally-trained Katherine doesn't have a frame to break — Katherine is the base distribution after training.

Implementation: prep_dataset.py drops all role=system messages from each conversation. The remaining user→assistant chain is what the model sees. The persona-establishing details (Austin, teal walls, cold feet, arts coordinator) survive only through the conversation content, not through the sysprompt scaffold. This is more demanding training but yields a more robust deployment.

---

Why enable_thinking=False

Qwen3.5 instruct models default to thinking mode (<think>...</think> blocks before output). The Dave persona we trained earlier uses thinking blocks (Two-Is architecture: reasoning layer + output layer both in Dave's voice). Katherine k0 does not. K0 is embodied — she reasons in prose like a person, not in tagged reasoning blocks like a model.

Setting enable_thinking=False at chat-template time during both training and inference ensures:

• No <think> markers leak into training text → model doesn't learn to emit them
• Inference produces direct prose response, no visible reasoning tags
• Matches K0's persona: a human doesn't have a thinking-tag layer

---

Hyperparameter derivation

LoRA configuration

Parameter	Value	Rationale
rank	64	High enough for persona consolidation; rank 32 is the standard floor for instruction-tuning; persona work benefits from somewhat higher rank but rank 128 is overkill for 1,886 examples (overfit risk)
lora_alpha	128	Maintains the standard alpha = 2 × rank ratio
lora_dropout	0.05	Light regularization. With high rank + small dataset, some dropout is sensible. Web-Claude advice argued for 0.0 ("dropout fights consolidation") but evidence is thin and overfit downside is real
target_modules	q/k/v/o + gate/up/down	Standard "all linear projections" — covers attention + MLP. No reason to exclude any
bias	none	Standard for LoRA
gradient_checkpointing	"unsloth"	Unsloth's optimized variant; trades ~10% wallclock for substantial memory savings; safe default

Training schedule

Parameter	Value	Rationale
epochs	3	More than 2 to allow voice subtleties to consolidate; not 4+ to avoid overfit on 1,886 examples
learning_rate	1e-4	Conservative for QLoRA on a 9B model. Standard 2e-4 is fine for shorter runs; 1e-4 for 3 epochs gives more stable convergence
lr_scheduler	cosine	Standard; avoids the noisier endgame of linear decay
warmup_ratio	0.05	Standard; brief warmup avoids initial gradient blowup with high rank
optim	adamw_8bit	Standard Unsloth-friendly choice; saves ~6 GB vs full adamw at 9B
weight_decay	0.01	Standard regularization
bf16	true	Native on H100/H200 (Hopper), faster than fp16

Batch + sequence

Parameter	Value	Rationale
per_device_batch	16	Token-length p99 is 246; 16 × 1024 = 16K tokens/step fits comfortably in H200's 141 GB VRAM
grad_accum	2	Effective batch 32 — good for gradient stability without large memory cost
max_seq_length	1024	Data p99 is 246 tokens; 1024 has 4× margin. Going to 4096 wastes ~4× compute for zero quality gain
packing	False	Conservative; unpacked training is more interpretable. With 1,886 examples wallclock is short anyway

DPO stage (Stage 2)

Parameter	Value	Rationale
epochs	2	DPO converges quickly on small preference sets (180 pairs); 2 epochs is the standard floor
learning_rate	5e-6	DPO requires tiny steps — 20-50× smaller than SFT LR — to avoid catastrophically forgetting the SFT-learned voice
beta	0.1	Standard KL strength; higher (0.3-1.0) = more conservative (stay closer to ref model); lower (0.01-0.05) = more willing to diverge for preference satisfaction. 0.1 is the sweet spot
batch	4 + grad_accum 2	Effective batch 8 — DPO loss is more variance-y than SFT, smaller batch is fine
warmup_ratio	0.1	Slightly larger than SFT warmup; DPO is sensitive to early instability

Reference model for DPO

The standard DPO recipe needs both a "policy" (the model being trained) and a "reference" (frozen snapshot for KL divergence). For PEFT/LoRA setups, DPOTrainer(ref_model=None) uses an adapter-disabled forward pass as the reference — same architecture, same weights, just LoRA-deltas zeroed. This saves a second model copy in VRAM and works correctly when the SFT adapter loaded as the policy is what we want as the reference too.

---

Wallclock and cost projections

H200 SXM5 throughput on Qwen3.5-9B QLoRA:
  Prefill + backward at batch 16, seq 1024 (16K tokens/step):
  ~3 sec/step (estimate; exact depends on Hopper-specific kernels in unsloth)

SFT total: 1,886 examples × 3 epochs / batch 32 = 177 steps
           177 × 3 sec = 531 sec ≈ 9 min compute
           + ~3 min model load on first run
           ≈ 12 min total

DPO: 180 pairs × 2 epochs / batch 8 = 45 steps
     ~5 sec/step (DPO is heavier per step than SFT due to ref + policy forward)
     45 × 5 = 225 sec ≈ 4 min

Merge + GGUF: ~10 min for first quant (compiles llama.cpp), ~3 min each subsequent
              q4 + q5 + q6 = ~16 min total

HF push: ~6 GB GGUF × 3 + ~250 MB adapters × 2 = ~19 GB
         At HF API ~50-100 MB/s realistic upload: ~3-5 min

Total wallclock: ~35-45 min on H200
                ~50-60 min on H100 SXM5
                ~75-90 min on H100 PCIe
                
Cost on RunPod Secure Cloud:
  H200 SXM5 ($3.99/hr) × 0.75 hr = $3.00
  H100 SXM5 ($3.49/hr) × 1.0 hr  = $3.50
  H100 PCIe ($2.49/hr) × 1.5 hr  = $3.75

All under $5. Genuinely cheap to iterate.

---

Provider lessons inherited from buddhabrot project

• Use RunPod Secure Cloud, not Community. Community shares HBM bandwidth across tenants; persona training is bandwidth-sensitive (4-bit weight loads + bf16 activations + adapter gradients all hit memory). Secure Cloud delivers the rated throughput; Community can be 3-5× slower.
• Diagnostic for shared throttling: nvidia-smi --query-gpu=power.draw,power.limit --format=csv,noheader. H200 doing real work draws 600-700W of 700W cap. If you see <30% with 100% util, you're sharing bandwidth — switch pods.
• Web Terminal is full bash. No SSH key drama needed for first-time setup.
• Bootstrap one-liner pattern works. Same shape as bootstrap-hyperbolic.sh — clone, install, prep, leave at ready-to-launch state.

---

What this pipeline does NOT do (deferred decisions)

• No DPO data augmentation. Just the 180 curated pairs.
• No held-out validation set. With 1,886 examples and 3 epochs, eyeballing the loss curve in the training log is the validation. A formal eval split would cut training data by 5-10% for marginal value at this scale.
• No multi-quantization eval automation. All 3 GGUFs are produced; manual probe-testing across quants is post-pipeline operator work.
• No public HF model repo. Only the private bucket. Public release happens after operator manually evaluates the GGUF outputs.
• No checkpoint averaging / EMA. Single final-epoch adapter is the artifact.

These are all intentional simplifications. Add them in v2 if first pass surfaces a need.