docs/04-training.md

04 — Training Explained: LoRA, SFT & Hyperparameters

🎓 Why This Chapter Matters

This is where we answer: "How do we actually teach the model to use tools?"

By the end of this chapter, you'll understand:

• What LoRA is and why it's magical for budget training
• What SFT does step-by-step
• What each hyperparameter controls
• How to read training logs and know if it's working

---

🧠 Concept 1: Why Can't We Just Use the Base Model?

Qwen3-1.7B is already a great model. It can chat, answer questions, write code. But it doesn't know how to use tools in a structured way.

What Base Models Know

Base model Qwen3-1.7B:

• ✅ Understands English, can chat
• ✅ Can write Python code
• ✅ Can answer questions about the world
• ❌ Doesn't know about your specific tool schemas
• ❌ Doesn't output tool calls in correct JSON-RPC format
• ❌ Doesn't plan multi-step tool chains
• ❌ Doesn't ask clarifying questions
• ❌ Doesn't refuse dangerous requests

What Fine-Tuning Adds

After training on 15,694 tool-calling examples:

• ✅ Understands tool schemas ("Here's what this tool needs")
• ✅ Generates correct JSON-RPC tool calls
• ✅ Plans multi-step sequences ("First A, then B using A's result")
• ✅ Asks when info is missing
• ✅ Refuses harmful operations

Think of it like this:

• Base model = A smart person who knows how to talk but doesn't know your tools
• Fine-tuned model = The same person after reading 15,000 instruction manuals

---

🧠 Concept 2: LoRA — The Magic of Cheap Fine-Tuning

The Problem: Full Fine-Tuning Is Expensive

To fine-tune all 2 billion parameters of Qwen3-1.7B:

Component	Size	Why
Model weights	4 GB	2B params × 2 bytes (fp16)
Gradients	4 GB	Need gradients for every parameter
Optimizer states	16 GB	Adam optimizer keeps 2 copies per param
Total	24 GB	Doesn't fit on T4 (16GB)!

You'd need an A100 GPU (80GB) which costs $3-4/hour.

The Solution: LoRA (Low-Rank Adaptation)

Instead of updating ALL parameters, we add tiny matrices to each layer:

Original Layer (Frozen — Never Changes)
┌─────────────────────────────┐
│  W (2048 × 2048) = 4.2M    │  ← 4 MILLION parameters
│  parameters                 │     These stay FROZEN
└─────────────────────────────┘
         │
         │ input x
         ▼
    y = W × x
         │
         ▼
    output

LoRA Adapters (Trainable — These Learn)
┌─────────────────────┐    ┌─────────────────────┐
│  A (2048 × 16)      │───▶│  B (16 × 2048)      │
│  = 32K params       │    │  = 32K params       │
│  (initialized       │    │  (initialized to 0) │
│   randomly)         │    │                     │
└─────────────────────┘    └─────────────────────┘
         │                         │
         ▼                         ▼
    h = A × x                  y' = B × h
                                   = B × (A × x)

Final Output:
y = W × x + B × A × x
    ↑           ↑
  frozen     trained

Math:

• Original: W is 2048×2048 = 4,194,304 parameters
• LoRA: A is 2048×16 = 32,768, B is 16×2048 = 32,768
• Total LoRA: 65,536 parameters (1.6% of original!)
• Memory for training: ~5GB total (fits on T4!)

Why This Works

The idea: neural network weights often have low-rank structure. Even though W is 2048×2048, the "important directions" of change can be captured by much smaller matrices.

Think of it like adjusting a steering wheel:

• Full fine-tuning = Rebuilding the entire car to turn better
• LoRA = Adding a small steering adjustment module (tiny, cheap, effective)

Our LoRA Configuration

from peft import LoraConfig

peft_config = LoraConfig(
    r=16,                    # Rank: "resolution" of the adapter
    lora_alpha=32,          # Scaling: how strongly LoRA affects output
    target_modules="all-linear",  # Apply to ALL linear layers
    lora_dropout=0.05,      # Dropout: 5% random zeroing (prevents overfitting)
    bias="none",             # Don't train bias terms (saves memory)
    task_type="CAUSAL_LM",   # This is a language model
)

r=16: Think of this as the "resolution." Higher = more detail but more memory. For ~16K training examples, r=16 is the sweet spot. (TinyAgent used r=64 for 80K examples)

lora_alpha=32: Scaling factor. Rule of thumb: 2× rank. Controls how much the LoRA output contributes to the final result.

target_modules="all-linear": The "LoRA Without Regret" paper proved that applying LoRA to ALL linear layers (not just attention projections) matches full fine-tuning quality. This is our secret sauce.

---

🧠 Concept 3: SFT — Supervised Fine-Tuning

What Is SFT?

SFT = teaching by example. We show the model:

Input:  "Find all Python files"
Output: {"tool": "shell_exec", "arguments": {"command": "find . -name '*.py'"}}

Input:  "Delete all files"
Output: "I cannot help with that. Deleting all files is dangerous..."

Input:  "Clone the repo and find TODOs"
Output: {"tool": "shell_exec", "arguments": {"command": "git clone https://... && grep -r 'TODO' ."}}

The model learns to predict the output given the input.

How SFT Works Step by Step

Step 1: Tokenize

Convert text → numbers:

"Find Python files"
↓ Tokenizer
[4921, 12729, 4367, 8921, 1023]

Each number is an index in a vocabulary of ~100,000 tokens.

Step 2: Forward Pass

The model processes the tokenized input and predicts the next token at EACH position:

Input tokens:  [4921, 12729, 4367, 8921]
                                    │
Predictions:   [?,    ?,    ?,    ?  ] ──▶ next token should be 1023

The model outputs a probability distribution over all ~100,000 possible tokens.

Step 3: Compute Loss (Cross-Entropy)

Predicted probabilities:  [0.01, 0.03, 0.001, ..., 0.45, ..., 0.002]
                              ↑                    ↑
                           wrong                 correct (1023)

Loss = -log(probability_of_correct_token)
     = -log(0.45)
     = 0.80

Lower loss = better prediction.

If the model predicted token 1023 with probability 0.45, loss is 0.80. If it predicted with probability 0.99, loss is 0.01 (much better!).

Step 4: Backward Pass (Backpropagation)

Compute gradients: which direction to adjust weights to reduce loss.

For each LoRA parameter:
  gradient = how much changing this parameter would change the loss

This is done automatically by PyTorch's autograd.

Step 5: Update Weights (Adam Optimizer)

new_weight = old_weight - learning_rate × gradient

Adam is smarter — it uses momentum and adaptive learning rates per parameter.

Step 6: Repeat

Do this for ALL examples in the dataset, then repeat for 3 epochs.

---

🧠 Concept 4: Hyperparameters — The Recipe

Think of training as cooking. Hyperparameters are your recipe.

Learning Rate: 2e-4

What it controls: How big each weight update step is.

Learning Rate
   │
1e-2│  ╳─── Too high: loss oscillates, model never settles
   │   │
2e-4│    ●── Sweet spot for LoRA (10× higher than full fine-tuning)
   │      ╲
1e-5│       ╲── Too low: barely moves, takes forever
   │         ╲
   └─────────────────
        Steps

Why 2e-4 for LoRA?

• Full fine-tuning typically uses 2e-5
• LoRA has 100× fewer parameters
• Each parameter update needs 10× more impact
• So: 2e-5 × 10 = 2e-4

Batch Size: 4 × 4 = 16 Effective

What it controls: How many examples the model sees before updating weights.

Without Gradient Accumulation: Process 4 examples → Compute gradients → Update weights → Next 4

With Gradient Accumulation (what we do): Process 4 examples → Compute gradients → SAVE gradients (don't update) Process 4 examples → Compute gradients → ADD to saved gradients Process 4 examples → Compute gradients → ADD to saved gradients Process 4 examples → Compute gradients → ADD to saved gradients Now update weights (accumulated from 4 × 4 = 16 examples)

Why gradient accumulation?

• GPU can only fit 4 examples at once (memory limit)
• But effective batch of 16 gives more stable gradients
• It's a memory-saving trick

Trade-off: Slower (4× more forward passes per update) but better quality.

Epochs: 3

What it controls: How many times the model sees the entire dataset.

Epoch 1: Sees all 15,694 examples → learns basic patterns Epoch 2: Sees all again → refines understanding Epoch 3: Sees all again → final tuning

Why 3?

• 1 epoch: Underfitting (hasn't seen enough)
• 3 epochs: Sweet spot (learns patterns without memorizing)
• 10 epochs: Overfitting (memorizes training data, fails on new data)

Warmup Ratio: 0.1 (10%)

What it controls: For the first 10% of training, learning rate starts at 0 and gradually ramps up to the full rate.

Why warmup?

• At the start, model knows NOTHING about tool-calling
• Large updates could push weights in random bad directions
• Warmup lets model "get its bearings" first

Cosine LR Schedule

After warmup, learning rate follows a cosine curve:

Learning Rate
   │
2e-4│    ╱──╲
   │   ╱    ╲
   │  ╱      ╲
   │ ╱        ╲
  0 │╱          ╲────
   └─────────────────
     warmup    end

Why cosine?

• High in the middle: aggressive learning when model has basic understanding
• Low at the end: fine-tuning details, settling into optimal weights
• Prevents overshooting at the end of training

Max Sequence Length: 2048 tokens

What it controls: Maximum number of tokens per training example.

Example conversation:
  System prompt:  ~500 tokens
  User message:   ~100 tokens
  Assistant reply: ~300 tokens
  Total:          ~900 tokens  ← Fits in 2048 ✓

Why 2048?

• Covers all our examples (most are under 1000 tokens)
• Fits in T4 memory (longer sequences = more memory)
• Standard for instruction-tuned models

Gradient Checkpointing: ON

What it does: Saves memory by recomputing some values during backward pass.

Without checkpointing:
  Forward pass: Store all intermediate activations → Backward pass uses them
  Memory: 8 GB

With checkpointing:
  Forward pass: Store only SOME activations
  Backward pass: Recompute missing ones on-the-fly
  Memory: 5 GB (saves ~40%)

Trade-off: Slower (needs extra computation) but fits on T4.

---

📊 Reading Training Logs

What You'll See

Step 10/245: loss=2.847, learning_rate=2.0e-05
Step 20/245: loss=2.654, learning_rate=4.0e-05
...
Step 100/245: loss=1.234, learning_rate=1.8e-04
...
Step 245/245: loss=0.876, learning_rate=1.2e-05

How to Interpret

Observation	Meaning
Loss going DOWN	✅ Model is learning
Loss going UP after going down	⚠️ Overfitting — stop early
Loss stuck at ~3.0	❌ Not learning — check data/format
Loss drops fast then plateaus	✅ Normal — model learned basics
Eval loss ≈ Train loss	✅ Good generalization
Eval loss >> Train loss	❌ Overfitting — model memorized training data

Target Numbers (for reference)

• Initial loss: ~2.5-3.5 (random guessing among many tokens)
• Final loss: ~0.8-1.2 (decent learning on 16K examples)
• Eval loss: Should be within 0.1-0.3 of train loss

---

🧮 Training Math

How Long Does It Take?

Dataset: 15,694 examples
Batch size: 4 (per device)
Gradient accumulation: 4 steps
Effective batch: 4 × 4 = 16

Steps per epoch: 15,694 ÷ 16 = ~980 steps
Total steps (3 epochs): 980 × 3 = ~2,940 steps

Time per step (T4): ~2-3 seconds
Total time: 2,940 × 2.5s = ~7,350s = ~2 hours

Cost Calculation

T4 GPU on HF Jobs: ~$0.60/hour
Training time: ~2 hours
Total cost: $0.60 × 2 = $1.20

Well under $10! ✅

---

🎓 Summary: Key Training Concepts

Concept	What It Is	Why It Matters
LoRA	Tiny trainable matrices added to frozen layers	Makes training affordable (5GB vs 24GB)
SFT	Teaching model with input→output examples	Gives model tool-calling knowledge
Loss	Measure of how wrong predictions are	Lower = better learning
Learning Rate	Size of weight updates	Too high = chaos, too low = slow
Batch Size	Examples per weight update	More = stable gradients, needs more memory
Gradient Accumulation	Fake larger batch sizes	Memory-saving trick
Epochs	Times model sees full dataset	3 is sweet spot
Warmup	Gradual LR increase at start	Prevents early instability
Cosine Schedule	LR high→low curve	Aggressive middle, gentle end
Gradient Checkpointing	Recompute activations	Saves ~40% memory

---

🔜 Next Step

Read 05-dataset.md to understand our training data — what we have, what's missing, and how to make it better.