An LLM is fundamentally a next-token predictor. Given a sequence of tokens, it outputs a probability distribution over the entire vocabulary (~32K-128K tokens). The training objective is to minimize cross-entropy loss between the predicted distribution and the actual next token across billions of text examples. The model learns grammar, facts, reasoning patterns, and even code โ all as statistical regularities in token sequences.
Below ~10B parameters, models just predict tokens. Above ~50B, new abilities emerge that weren't explicitly trained: chain-of-thought reasoning, few-shot learning, code generation, translation between languages never paired in training data. This is why scale matters and is the foundation of the "scaling laws" (Chinchilla, Kaplan et al.).
Text is never fed directly to an LLM. It's first converted to tokens (sub-word units). Understanding tokenization is critical because:
| Aspect | Why It Matters | Example |
|---|---|---|
| Cost | API pricing is per-token, not per-word | "unbelievable" = 3 tokens = 3x cost vs 1 word |
| Context limits | 128K tokens โ 128K words (~75K words) | 1 token โ 0.75 English words on average |
| Non-English penalty | Languages like Hindi/Chinese use 2-3x more tokens per word | "เคจเคฎเคธเฅเคคเฅ" might be 6 tokens vs "hello" = 1 token |
| Code tokenization | Whitespace and syntax consume tokens | 4 spaces of indentation = 1 token wasted per line |
| Number handling | Numbers tokenize unpredictably | "1234567" might split as ["123", "45", "67"] โ why LLMs are bad at math |
Algorithms: BPE (GPT, LLaMA) โ merges frequent byte pairs iteratively. WordPiece (BERT) โ maximizes likelihood. SentencePiece/Unigram (T5) โ statistical segmentation. Modern LLMs use vocabularies of 32K-128K tokens.
Every generation from an LLM is shaped by parameters under the hood. Knowing how to tune these is critical for production AI engineering.
| Parameter | What it controls | Book Insight / Use Case |
|---|---|---|
| 1. Max tokens | Hard cap on generation length | Lower for speed/safety; Higher for summaries |
| 2. Temperature | Randomness/Creativity | ~0 for deterministic QA; 0.7-1.0 for brainstorming |
| 3. Top-k | Limit sampling to top K tokens | K=5 forces the model to choose only from 5 most likely words |
| 4. Top-p (nucleus) | Limit to smallest set covering p% mass | P=0.9 is adaptive; handles coherence vs diversity better than K |
| 5. Frequency penalty | Discourage reusing frequent tokens | Set > 0 to stop the model from repeating itself in loops |
| 6. Presence penalty | Encourage new topics/tokens | Set > 0 to push the model towards broad exploration |
| 7. Stop sequences | Halts generation at specific tokens | Critical for structured JSON (stop at "}") or code blocks |
| Bonus: Min-p | Dynamic probability threshold | Only keeps tokens at least X% as likely as the top token. Most robust for coherence. |
Don't combine temperature=0 with top_p=0.1 โ they interact. Use either temperature OR top-p for sampling control, not both. OpenAI recommends changing one and leaving the other at default.
Decoding is the process of picking the next token. How we pick it determines the style of the output.
Normally, models use only the final layer's logits. SLED looks at how logits evolve across ALL layers. It measures consensus across internal representations, producing more grounded and factual outputs without retraining or extra data.
LLMs don't just learn from raw text; they transfer knowledge between models (Distillation). Distillation happens at Pre-training (Llama 4 Scout) or Post-training (DeepSeek-R1 to Qwen).
| Technique | How It Works | Advantage / Note |
|---|---|---|
| 1. Soft-label Distillation | Student matches the Teacher's entire probability distribution (softmax). | Maximum reasoning transfer; requires access to logits. |
| 2. Hard-label Distillation | Student matches only the final token choice (hard label) of the Teacher. | DeepSeek used this to distill R1 into smaller models. |
| 3. Co-distillation | Train Teacher and Student together from scratch; both predict concurrently. | Gemma 3 used this during both pre and post-training. |
| Tool | Best For | Setup Complexity |
|---|---|---|
| Ollama | Simple CLI/Desktop usage; one-command 'run'. | Zero (Just install app) |
| LM Studio | GUI for chatting with GGUF models; eject/load easily. | Zero (Desktop app) |
| vLLM | High-performance serving and API hosting. | Low-Medium (pip install) |
| llama.cpp | Extreme portability (C++), runs on Mac/CPU/Android. | Medium (compile or download bins) |
The context window determines how many tokens the model can process in a single call (input + output combined).
| Model | Context Window | Approx. Pages |
|---|---|---|
| GPT-4o | 128K tokens | ~200 pages |
| Claude 3.5 Sonnet | 200K tokens | ~350 pages |
| Gemini 1.5 Pro | 2M tokens | ~3,000 pages |
| LLaMA 3.1 | 128K tokens | ~200 pages |
| Mistral Large | 128K tokens | ~200 pages |
"Lost in the Middle" (Liu et al., 2023): Performance degrades for information placed in the middle of very long contexts. Models attend most to the beginning and end of prompts. Strategy: put the most important content at the start or end; use retrieval to avoid stuffing the entire context.
Training an LLM from scratch involves: (1) Data collection โ crawl the web (Common Crawl, ~1 trillion tokens), books, code (GitHub), conversations. (2) Data cleaning โ deduplication, quality filtering, toxicity removal, PII scrubbing. (3) Tokenizer training โ build BPE vocabulary from the corpus. (4) Pre-training โ next-token prediction on massive GPU clusters (thousands of A100s/H100s for weeks). Cost: $2M-$100M+ for frontier models.
A base model predicts tokens but doesn't follow instructions or refuse harmful content. Alignment methods bridge this gap:
| Method | How It Works | Used By |
|---|---|---|
| SFT (Supervised Fine-Tuning) | Train on (instruction, response) pairs from human annotators | All models (Step 1) |
| RLHF | Train a reward model on human preferences, then optimize policy via PPO | GPT-4, Claude (early) |
| DPO (Direct Preference Optimization) | Skip the reward model โ directly optimize from preference pairs, simpler and more stable | LLaMA 3, Zephyr, Gemma |
| Constitutional AI | Model critiques and revises its own outputs against a set of principles | Claude (Anthropic) |
| RLAIF | Use an AI model (not humans) to generate preference data | Gemini, some open models |
| Provider | Flagship Model | Strengths | Best For |
|---|---|---|---|
| OpenAI | GPT-4o, o1, o3 | Best all-around, strong coding, reasoning chains (o1/o3) | General purpose, production |
| Anthropic | Claude 3.5 Sonnet | Best for long documents, coding, safety-conscious | Enterprise, agents, analysis |
| Gemini 1.5 Pro/2.0 | Massive context (2M), multi-modal, grounding | Document processing, multi-modal | |
| Meta | LLaMA 3.1/3.2 | Best open-source, fine-tunable, commercially free | Self-hosting, fine-tuning |
| Mistral | Mistral Large, Mixtral | Strong open models, MoE efficiency | European market, cost-effective |
| DeepSeek | DeepSeek V3, R1 | Exceptional reasoning, competitive with o1 | Math, coding, research |
Chinchilla Scaling Law (Hoffmann et al., 2022): For a compute-optimal model, training tokens should scale proportionally to model parameters. A 70B model should be trained on ~1.4 trillion tokens. Key insight: many earlier models were undertrained (too many params, not enough data). LLaMA showed smaller, well-trained models can match larger undertrained ones.
Answer: The model becomes deterministic, always picking the highest-probability token (greedy decoding). Use for tasks requiring consistency (code generation, extraction, classification). Side effects: can get stuck in repetitive loops; technically temperature=0 is greedy, temperature=1 is the trained distribution, above 1 is "hotter" (more random). For near-deterministic with slight randomness, use temperature=0.1 with top_p=1.
Answer: LLMs don't "know" facts โ they model token probabilities. When asked about something rare or unseen, the model generates statistically plausible text rather than admitting ignorance. Solutions: (1) RAG โ ground answers in retrieved documents; (2) Lower temperature โ reduce sampling randomness; (3) Structured output forcing โ constrain output to valid formats; (4) Self-consistency โ sample N times, pick the majority answer; (5) Calibrated prompting โ explicitly instruct "say I don't know if unsure."
Answer: Context window is the tokens the model can process in a single inference pass โ it's completely stateless. There is no persistent memory between API calls. "Memory" in frameworks like LangChain is implemented externally by: (1) storing past conversation turns in a database, (2) summarizing old turns to compress history, (3) reinserting relevant history into the new prompt. Every call is independent โ the model has zero recall of previous calls.
Answer: RLHF: Train a separate reward model on human preferences, then optimize the LLM policy via PPO (reinforcement learning). Complex, unstable, expensive. DPO (Direct Preference Optimization): Skip the reward model entirely โ directly optimize from preference pairs using a closed-form solution. Simpler, more stable, cheaper. DPO is now preferred for open-source models (LLaMA 3, Gemma). RLHF is still used by OpenAI/Anthropic where they have massive human labeling infrastructure.
Answer: Research by Liu et al. (2023) showed that LLMs perform significantly worse when relevant information is placed in the middle of long contexts compared to the beginning or end. The model's attention mechanism attends most to recent tokens (recency bias) and the very first tokens (primacy bias). Practical implication: place the most critical context at the start or end of your prompt, never buried in the middle of a long document.
Answer: GPT-4o is a standard auto-regressive LLM โ generates tokens left-to-right in one pass. o1/o3 are reasoning models that use "chain-of-thought before answering" โ they generate internal reasoning tokens (hidden from the user) before producing the final answer. This makes them dramatically better at math, logic, and coding, but 3-10x slower and more expensive. Use GPT-4o for speed-sensitive tasks (chat, extraction), o1/o3 for complex reasoning (math proofs, hard coding, multi-step analysis).
For each token, compute 3 vectors via learned linear projections: Query (Q), Key (K), Value (V). The attention formula:
| Component | Role | Analogy |
|---|---|---|
| Query (Q) | What this token is looking for | Search query on Google |
| Key (K) | What each token offers/advertises | Page titles in search index |
| Value (V) | Actual content to retrieve | Page content returned |
| โdk scaling | Prevents softmax saturation for large dims | Normalization for numerical stability |
Why it works: Self-attention lets every token attend to every other token in O(1) hops (vs O(n) for RNNs). "The cat sat on the mat" โ the word "mat" can directly attend to "cat" and "sat" to understand context, without information passing through intermediate words.
Run h independent attention heads in parallel, each learning different relationship types. Concatenate outputs and project. GPT-4 likely uses ~96 heads. Each head specializes: head 1 may track subject-verb agreement, head 2 may track pronoun coreference, head 3 may track positional patterns.
| Variant | Key Idea | Used By | KV Cache Savings |
|---|---|---|---|
| MHA (Multi-Head) | Separate Q, K, V per head | GPT-2, BERT | 1x (baseline) |
| GQA (Grouped Query) | Share K,V across groups of Q heads | LLaMA 3, Gemma, Mistral | 4-8x smaller |
| MQA (Multi-Query) | Single K,V shared across ALL Q heads | PaLM, Falcon | 32-96x smaller |
| Sliding Window | Attend only to nearby tokens (window) | Mistral, Mixtral | Fixed memory regardless of length |
RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors in complex space. Advantages: (1) Relative position naturally emerges from dot products, (2) Enables length extrapolation beyond training length (with techniques like YaRN, NTK-aware scaling), (3) No additional parameters. Used by LLaMA, Mistral, Gemma, Qwen โ virtually all modern open LLMs.
Each transformer block has: (1) Multi-Head Attention โ (2) Residual Connection + LayerNorm โ (3) Feed-Forward Network (FFN) with hidden dim 4x model dim โ (4) Residual Connection + LayerNorm. Modern models use Pre-LayerNorm (normalize before attention, not after) and SwiGLU activation in FFN instead of ReLU for better performance.
Standard attention requires O(nยฒ) memory for the attention matrix. FlashAttention (Dao et al., 2022) computes exact attention without materializing the full matrix by using tiling and kernel fusion. Result: 2-4x faster inference, ~20x less memory for long sequences. FlashAttention 2 adds further optimizations. Essential for context windows >8K tokens.
Instead of one massive FFN, use N expert FFNs and a router that selects top-k experts per token. Only selected experts are activated (sparse computation). Mixtral 8x7B has 8 experts, activates 2 per token โ 47B total params but only 13B active per token. Result: 3-4x more efficient than dense models of same quality.
| Architecture | Attention Type | Best For | Examples |
|---|---|---|---|
| Decoder-Only | Causal (left-to-right only) | Text generation, chat, code | GPT-4, LLaMA, Gemma, Mistral |
| Encoder-Only | Bidirectional (sees all tokens) | Classification, NER, embeddings | BERT, RoBERTa, DeBERTa |
| Encoder-Decoder | Encoder bidirectional, decoder causal | Translation, summarization | T5, BART, mT5, Flan-T5 |
Answer: For large d_k, dot products grow large in magnitude (variance โ d_k), pushing softmax into regions with extremely small gradients (saturated). Dividing by โd_k normalizes variance to 1, keeping gradients healthy. Without it, training becomes unstable โ same principle as Xavier/He weight initialization.
Answer: During autoregressive generation, Key and Value matrices for past tokens are cached in GPU memory so they don't need recomputation on each new token. Without KV cache: generating token n requires reprocessing all n-1 previous tokens โ O(nยฒ) total work. With KV cache: each new token only computes its own Q, K, V and attends to cached K, V โ O(n) total. A 7B model with 8K context uses ~4GB just for KV cache. This is why GPU memory (not compute) is the real bottleneck for long-context inference.
Answer: MHA (Multi-Head Attention): Separate K,V per head โ maximum expressivity but largest KV cache. GQA (Grouped Query Attention): K,V shared across groups of Q heads (e.g., 8 Q heads share 1 KV pair). 4-8x smaller KV cache with minimal quality loss. Used by LLaMA-3, Mistral, Gemma. MQA (Multi-Query Attention): ALL Q heads share a SINGLE K,V pair. Maximum KV cache savings (32-96x) but slightly lower quality. Used by PaLM, Falcon. Industry has settled on GQA as the best tradeoff.
Answer: RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors in 2D complex planes. Key advantages: (1) Relative position naturally emerges from dot products โ Attention(q_m, k_n) depends only on m-n, not absolute positions. (2) No additional learned parameters. (3) Better length generalization โ techniques like YaRN and NTK-aware scaling allow extending context beyond training length. Sinusoidal encoding struggled with extrapolation and required absolute position awareness.
Answer: Standard attention materializes the full NรN attention matrix in GPU HBM (High Bandwidth Memory). FlashAttention uses tiling โ it breaks Q, K, V into blocks, computes attention within SRAM (fast on-chip memory), and never writes the full attention matrix to HBM. This reduces memory IO from O(Nยฒ) to O(Nยฒ/M) where M is SRAM size. Result: exact same output, but 2-4x faster and uses O(N) memory instead of O(Nยฒ). It's a pure systems optimization โ no approximation.
Answer: MoE replaces the single FFN in each transformer block with N parallel expert FFNs plus a learned router. For each token, the router selects top-k experts (usually k=2). Only selected experts are activated โ rest are skipped. Benefits: Train a model with 8x more parameters at ~2x the compute cost of a dense model. Tradeoffs: (1) All parameters must fit in memory even though only k are active. (2) Load balancing โ if router always picks the same experts, others waste space. Solved with auxiliary loss. (3) Harder to fine-tune โ expert specialization can be disrupted. Example: Mixtral 8x7B = 47B params but only 13B active per token.
The transformers library provides a unified API to load, run, and fine-tune any model architecture. It wraps 200+ architectures (GPT, LLaMA, Mistral, Gemma, T5, BERT, ViT, Whisper, etc.) behind Auto Classes that detect architecture automatically from config.json.
| AutoClass | Use Case | Example Models |
|---|---|---|
| AutoModelForCausalLM | Text generation (decoder-only) | LLaMA, GPT-2, Mistral, Gemma |
| AutoModelForSeq2SeqLM | Translation, summarization | T5, BART, mT5, Flan-T5 |
| AutoModelForSequenceClassification | Text classification, sentiment | BERT, RoBERTa, DeBERTa |
| AutoModelForTokenClassification | NER, POS tagging | BERT-NER, SpanBERT |
| AutoModelForQuestionAnswering | Extractive QA | BERT-QA, RoBERTa-QA |
| AutoModel (base) | Embeddings, custom heads | Any backbone |
torch_dtype=torch.bfloat16 โ half precision, saves 50% memory
device_map="auto" โ auto-shard across GPUs/CPU/disk
load_in_4bit=True โ 4-bit quantization via BitsAndBytes
attn_implementation="flash_attention_2" โ use FlashAttention for 2-4x faster inference
trust_remote_code=True โ needed for custom architectures with code on the Hub
The pipeline() function wraps tokenization + model + post-processing into a single call. Under the hood: tokenize โ model forward pass โ decode/format output. Supports these key tasks:
| Task | Pipeline Name | Default Model |
|---|---|---|
| Text Generation | "text-generation" | gpt2 |
| Sentiment Analysis | "sentiment-analysis" | distilbert-sst2 |
| Named Entity Recognition | "ner" | dbmdz/bert-large-NER |
| Summarization | "summarization" | sshleifer/distilbart-cnn |
| Translation | "translation_en_to_fr" | Helsinki-NLP/opus-mt |
| Zero-Shot Classification | "zero-shot-classification" | facebook/bart-large-mnli |
| Feature Extraction (Embeddings) | "feature-extraction" | sentence-transformers |
| Image Classification | "image-classification" | google/vit-base |
| Speech Recognition | "automatic-speech-recognition" | openai/whisper-base |
| Text-to-Image | "text-to-image" | stabilityai/sdxl |
The tokenizers library is written in Rust with Python bindings. It's 10-100x faster than pure-Python tokenizers. Key tokenization algorithms used by modern LLMs:
| Algorithm | Used By | How It Works |
|---|---|---|
| BPE (Byte-Pair Encoding) | GPT-2, GPT-4, LLaMA | Repeatedly merges most frequent byte pairs. "unbelievable" โ ["un", "believ", "able"] |
| SentencePiece (Unigram) | T5, ALBERT, XLNet | Statistical model that finds optimal subword segmentation probabilistically |
| WordPiece | BERT, DistilBERT | Greedy algorithm; splits by maximizing likelihood. Uses "##" prefix for sub-tokens |
โข Numbers tokenize unpredictably: "1234" might be 1-4 tokens depending on the model
โข Whitespace matters: " hello" and "hello" produce different tokens in GPT
โข Non-English languages use more tokens per word (higher cost per concept)
โข Always use the model's own tokenizer โ never mix tokenizers between models
datasets uses Apache Arrow for columnar, memory-mapped storage. A 100GB dataset can be iterated without loading into RAM. Key features:
Memory Mapping: Data stays on disk; only accessed rows are loaded into memory. Streaming: load_dataset(..., streaming=True) returns an iterable โ process terabytes with constant memory. Map/Filter: Apply transformations with automatic caching and multiprocessing. Hub Integration: 150,000+ datasets available via load_dataset("dataset_name").
The Trainer class handles: training loop, evaluation, checkpointing, logging (to TensorBoard/W&B), mixed precision, gradient accumulation, distributed training, and early stopping. You just provide model + dataset + TrainingArguments. For instruction-tuning LLMs, use TRL's SFTTrainer (built on top of Trainer) which handles chat templates and packing automatically.
accelerate abstracts away multi-GPU, TPU, and mixed-precision complexity. Write your training loop once; run on 1 GPU or 64 GPUs with zero code changes. Key feature: Accelerator class wraps your model, optimizer, and dataloader. It handles data sharding, gradient synchronization, and device placement automatically.
Every model on HF Hub is a Git LFS repo containing: config.json (architecture), model.safetensors (weights), tokenizer.json, and a README.md (model card). You can push your own models with model.push_to_hub(). The Hub supports: model versioning (Git branches/tags), automatic model cards, gated access (license agreements), and API inference endpoints.
| Library | Purpose | Key Feature |
|---|---|---|
peft | Parameter-efficient fine-tuning | LoRA, QLoRA, Adapters, Prompt Tuning |
trl | RLHF and alignment training | SFTTrainer, DPOTrainer, PPOTrainer, RewardTrainer |
diffusers | Image/video generation | Stable Diffusion, SDXL, ControlNet, IP-Adapter |
evaluate | Metrics computation | BLEU, ROUGE, accuracy, perplexity, and 100+ metrics |
gradio | Build ML demos | Web UI for any model in 5 lines of code |
smolagents | Lightweight AI agents | Code-based tool calling, HF model integration |
safetensors | Safe model format | Fast, safe, and efficient tensor serialization (replaces pickle) |
huggingface_hub | Hub API client | Download files, push models, create repos, manage spaces |
HF Spaces lets you deploy Gradio or Streamlit apps on managed infrastructure. Free CPU tier for demos; upgrade to T4 ($0.60/hr) or A100 ($3.09/hr) for GPU workloads. Spaces support Docker, static HTML, and custom environments. They auto-build from a Git repo with a simple requirements.txt. Ideal for: model demos, portfolio projects, internal tools, and quick prototypes.
from_pretrained and pipeline?Answer: pipeline() is a high-level convenience wrapper โ it auto-detects the task, loads both model + tokenizer, handles tokenization/decoding, and returns human-readable output. from_pretrained() gives raw access to model weights for: custom inference loops, fine-tuning, extracting embeddings, modifying the model architecture, or anything beyond standard inference. Rule: prototyping โ pipeline, production/training โ from_pretrained.
device_map="auto" and how does model sharding work?Answer: It uses the accelerate library to automatically distribute model layers across available hardware. The algorithm: (1) Measure available memory on each GPU, CPU, and disk; (2) Place layers sequentially, filling GPU first, spilling to CPU, then disk. For a 70B model on two 24GB GPUs: layers 0-40 on GPU 0, layers 41-80 on GPU 1. CPU/disk offloading adds latency but enables running models that don't fit in GPU memory at all. Use max_memory param to control allocation.
Answer: Datasets uses Apache Arrow โ a columnar, memory-mapped format. Key advantages: (1) Memory mapping: A 100GB dataset uses near-zero RAM โ data stays on disk but accessed at near-RAM speed via OS page cache. (2) Zero-copy: Slicing doesn't duplicate data. (3) Streaming: Process datasets larger than disk with streaming=True. (4) Parallel map: num_proc=N for multi-core preprocessing. (5) Caching: Processed results are automatically cached to disk. Pandas loads everything into RAM โ impossible for large-scale ML datasets.
Answer: Each instruction-tuned model is trained with a specific format for system/user/assistant messages. Llama uses <|begin_of_text|><|start_header_id|>system<|end_header_id|>, while ChatML uses <|im_start|>system. If you format input incorrectly, the model behaves like a base model (no instruction following). tokenizer.apply_chat_template() auto-formats messages correctly for any model. This is the #1 mistake beginners make โ using raw text instead of the chat template.
Answer: (1) Accept the model license on the Hub model page. (2) Create a read token at hf.co/settings/tokens. (3) For local: huggingface-cli login. (4) In CI/CD: set HF_TOKEN environment variable. (5) In code: pass token="hf_xxx" to from_pretrained(). For Docker: bake the token as a secret, never in the image. For Kubernetes: use a Secret mounted as an env var. The token is only needed for download โ once cached locally, no token is needed for inference.
Answer: Traditional PyTorch models use Python's pickle format, which can execute arbitrary code during loading โ a security vulnerability. A malicious model file could run code on your machine when loaded. safetensors is a safe, fast tensor format that: (1) Cannot execute code (pure data), (2) Supports zero-copy loading (memory-mapped), (3) Is 2-5x faster to load than pickle, (4) Supports lazy loading (load only specific tensors). It's now the default format on HF Hub.
Instead of modifying the full weight matrix W (d ร k), LoRA trains two small matrices: A (d ร r) and B (r ร k) where rank r << min(d,k). The adapted weight is W + ฮฑBA. During inference, BA is merged into W โ zero latency overhead. Only 0.1-1% of parameters are trained.
| Method | Trainable Params | GPU Needed (7B) | Quality |
|---|---|---|---|
| Full Fine-Tuning | 100% | ~80GB | Best |
| LoRA (r=16) | ~0.5% | ~16GB | Very Good |
| QLoRA (4-bit + LoRA) | ~0.5% | ~8GB | Good |
| Prompt Tuning | <0.01% | ~6GB | Task specific |
QLoRA (Dettmers et al., 2023) combines: (1) 4-bit NF4 quantization of the base model, (2) double quantization to compress quantization constants, (3) paged optimizers to handle gradient spikes. Fine-tune a 65B model on a single 48GB GPU โ impossible before QLoRA.
| Technique | How It Works | Use Case |
|---|---|---|
| 1. SFT | Supervised Fine-Tuning on (Q, A) pairs | Instruction following |
| 2. RLHF | Reward model + PPO optimization | Human value alignment |
| 3. DPO | Directly optimize from preferences | Popular, stable alternative to RLHF |
| 4. LoRA / QLoRA | Low-rank adaptation | Efficient training on consumer GPUs |
| 5. RFT | Rejection Fine-Tuning | Self-improving by filtering best outputs |
| 6. GRPO | Group Relative Policy Optimization | The core of DeepSeek-R1; trains reasoning models without a separate value model |
| 7. IFT | Instruction Fine-Tuning | Bridge gap between base model and agent |
SFT (Supervised Fine-Tuning) trains on fixed human-written data. RFT (Rejection Fine-Tuning) uses the model itself to generate multiple responses, filters them for correctness (e.g., using a code compiler or calculator), and then fine-tunes only on those verified correct samples. This allows models to self-improve beyond human capabilities in certain domains.
DeepSeek's GRPO (Group Relative Policy Optimization) is the new frontier. Instead of a separate reward model, you generate a group of outputs and rank them relative to each other (e.g., based on mathematical correctness or formatting). This forces the model to learn long chains of thought and "thinking" behavior.
Answer: Rank r controls the expressiveness of the adaptation. Lower r (4-8): minimal params, fast, good for narrow tasks. Higher r (16-64): more capacity, better for complex tasks. Rule of thumb: start with r=16. If quality is poor, try r=64. If memory is tight, try r=4 with higher alpha. lora_alpha/r acts as the effective learning rate scaling.
Answer: Target the attention projection layers: q_proj, k_proj, v_proj, o_proj. Optionally add MLP layers (gate_proj, up_proj, down_proj). Targeting more modules increases quality but also memory. Research shows q_proj + v_proj alone gives 80% of the benefit. Use target_modules="all-linear" in recent PEFT versions for maximum coverage.
Answer: In full fine-tuning, training on new data overwrites old weights, causing the model to "forget" general capabilities. LoRA freezes the original weights โ the base model is untouched. Only the low-rank adapter is trained. This means the base knowledge is preserved, and you can merge/unmerge adapters to switch tasks.
Indexing phase (offline): (1) Load documents (PDF, HTML, Markdown, DB), (2) Chunk into segments (~500 tokens), (3) Embed each chunk with an embedding model, (4) Store vectors + metadata in a vector database.
Query phase (online): (1) Embed user query with same model, (2) Retrieve top-k similar chunks via ANN search, (3) Inject chunks into LLM prompt as context, (4) LLM generates a grounded response with citations.
| Strategy | How It Works | Best For | Tradeoff |
|---|---|---|---|
| Fixed-size | Split every N tokens | Generic text | May cut mid-sentence |
| Recursive character | Split by paragraphs, sentences, words | Most documents | LangChain default; good balance |
| Semantic chunking | Split where embedding similarity drops | Long documents | Groups related content; 10x slower |
| Document structure | Parse by headings, sections, tables | PDFs, HTML, Markdown | Preserves context hierarchy |
| Agentic chunking | LLM decides chunk boundaries | Highest quality | Expensive but best recall |
256-512 tokens for OpenAI embeddings, 128-256 for smaller models. Use 50-100 token overlap. Test with your actual queries โ measure retrieval recall, not just generation quality.
| Model | Dims | Max Tokens | Quality | Cost |
|---|---|---|---|---|
| OpenAI text-embedding-3-large | 3072 | 8191 | Top tier | $0.13/1M |
| OpenAI text-embedding-3-small | 1536 | 8191 | Very good | $0.02/1M |
| Cohere embed-v3 | 1024 | 512 | Top tier | $0.10/1M |
| BGE-large-en-v1.5 | 1024 | 512 | Great (open) | Free |
| all-MiniLM-L6-v2 | 384 | 256 | Good | Free |
| Architecture | How It Works | Advantage |
|---|---|---|
| Naive RAG | Retrieve top-K, generate | Simplest, 70% accuracy |
| Advanced RAG | Pre-retrieval (HyDE) + Post (Re-ranking) | Better precision, 85% accuracy |
| Self-RAG | Agent decides if retrieval is needed | Reduces token cost / hallucinations |
| REFRAG | Retrieve first, then Refine query and Retrieve again | Critical for multi-hop reasoning |
| CAG (Context Augmented Gen) | Pre-process docs into model KV cache | Ultra-low latency for fixed datasets |
| HyDE | Embed hypothetical answer, not query | Handles vocabulary mismatch |
| Agentic RAG | Agent uses search tool loop as needed | Most flexible but slowest |
| Knowledge Graph RAG | Retrieve triple relations (GraphRAG) | Excellent for complex connections |
| Strategy | Description | Best For |
|---|---|---|
| 1. Fixed-size | Chunking by character/token count | Simple text, general usage |
| 2. Recursive | Split by paragraph, then sentence | Most documents (LangChain default) |
| 3. Semantic | Split where embedding distance jumps | Long books or unified narratives |
| 4. Structural | Split by HTML headings or PDF sections | Technical docs, manuals |
| 5. Agentic | LLM analyzes text and groups it | Highest accuracy, highest cost |
Standard RAG is a one-shot process. Agentic RAG allows an agent to decide how to search, what to search, and when to stop. Combined with AI Memory (persisting relevant facts across sessions), this creates systems that grow smarter over time.
| Metric | Measures | Target |
|---|---|---|
| Faithfulness | Are claims supported by context? | 0.9+ |
| Answer Relevance | Does answer address the question? | 0.85+ |
| Context Recall | Did retriever find all needed info? | 0.8+ |
| Context Precision | Is retrieved context relevant? | 0.8+ |
Answer: Overlap ensures sentences spanning chunk boundaries aren't lost. A 50-token overlap on 500-token chunks means each chunk shares context with neighbors. Without it, a key sentence split across chunks is invisible during retrieval. Tradeoff: more overlap = more storage + slightly redundant results.
Answer: Use RAGAS: (1) Faithfulness โ answer uses only retrieved context? (2) Answer Relevance โ addresses the question? (3) Context Recall โ retriever found all needed info? (4) Context Precision โ retrieved context relevant? Build a golden dataset of (question, ground_truth, expected_contexts). Use LLM-as-judge for scale.
Answer: HyDE: Short queries that don't match document vocabulary. LLM generates hypothetical answer, embed that instead. Query decomposition: Complex multi-part questions. Break into sub-queries, retrieve for each, combine. Use HyDE for vocabulary mismatch, decomposition for complexity.
Answer: Naive: embed query -> top-k -> stuff in prompt -> generate. Problems: poor recall, irrelevant chunks. Advanced: adds query transformation, hybrid search, re-ranking, parent-child chunks, metadata filtering. Advanced RAG achieves 20-40% higher faithfulness in production.
Answer: (1) Extract and caption โ tables to markdown, images captioned via GPT-4o, embed text. (2) Multi-modal embeddings โ CLIP for images alongside text. (3) ColPali โ embed entire PDF page screenshots without OCR. Approach 1 is most production-ready; approach 3 is newest.
| Metric | Best For | Range |
|---|---|---|
| Cosine Similarity | Text embeddings (normalized) | -1 to 1 (1 = identical) |
| Dot Product | When magnitude matters | -inf to inf |
| L2 (Euclidean) | Image embeddings | 0 to inf (0 = identical) |
Critical rule: Use the same metric your embedding model was trained with. OpenAI = cosine. Wrong metric can drop recall by 20%+.
| Algorithm | How It Works | Speed | Memory | Best For |
|---|---|---|---|---|
| HNSW | Hierarchical graph navigation | Very fast | High | Low-latency (<10ms) |
| IVF | Cluster vectors, search nearby | Fast | Medium | Large datasets, GPU |
| PQ | Compress vectors into codes | Medium | Very low | Billions of vectors |
| ScaNN | Google's anisotropic hashing | Very fast | Medium | Extreme scale |
| DiskANN | SSD-based graph search | Fast | Very low | Billion-scale on commodity HW |
Multi-layered graph. Top layers are sparse "highways" with long-range connections. Bottom layers are dense with local neighbors. Search starts at top, greedily navigates toward query, drops to lower layers for precision. Params: M (connections per node, 16-64), efConstruction (build quality, 100-200), efSearch (query quality, 50-200).
| Database | Type | Best Use | Hosting | Scale |
|---|---|---|---|---|
| FAISS | Library | Research, in-process | In-process | Billions (GPU) |
| ChromaDB | Embedded | Prototyping | Local/Cloud | Millions |
| Pinecone | Managed SaaS | Production, zero-ops | Fully managed | Billions |
| Weaviate | Full DB | Hybrid search | Cloud/Self-host | Billions |
| Qdrant | Full DB | Filtering + vectors | Cloud/Self-host | Billions |
| pgvector | Extension | Already using Postgres | In PostgreSQL | Millions |
| Milvus | Distributed | Enterprise | Cloud/Self-host | Billions+ |
The embedding model matters more than the vector DB. Check MTEB leaderboard. Key factors: (1) Dimension โ 384 (fast) to 3072 (accurate). (2) Max tokens โ must be >= chunk size. (3) Multilingual โ require for non-English. (4) Domain โ fine-tuned models beat general-purpose.
Answer: ANN trades <1% recall for massive speedups. Exact NN is O(n*d) โ 10M vectors at 1536 dims = 15 billion ops per query. HNSW achieves ~99% recall at 100x+ speedup by navigating a graph, visiting only ~log(n) nodes.
Answer: Cosine for text (angle-based, normalized). Dot product when magnitude matters. L2 for image/audio. Always match the metric your embedding model was trained with โ wrong metric drops recall 20%+.
Answer: (1) Pre-filter: metadata first, ANN on subset โ accurate but slow. (2) Post-filter: ANN first, filter after โ fast but may return <k. (3) Filtered ANN (Pinecone, Qdrant): filtering during graph traversal โ best of both. Always prefer filtered ANN in production.
Answer: FAISS: Research, benchmarks, in-process โ no persistence or filtering. ChromaDB: Prototyping <1M vectors โ good DX, local-first. Pinecone: Production, zero-ops, auto-scales, filtered ANN โ 10x more expensive. Use FAISS to validate, Chroma to prototype, Pinecone/Qdrant to ship.
ReAct (Yao 2022): Thought > Action > Observation > repeat. The LLM reasons about what to do, calls a tool, sees the result, and continues until it has a final answer.
| Architecture | How It Works | Best For |
|---|---|---|
| ReAct Loop | Fixed think-act-observe cycle | Simple tool-using tasks |
| Plan-and-Execute | Full plan first, then execute steps | Multi-step structured tasks |
| State Machine (LangGraph) | Directed graph with conditional edges | Complex workflows, branching |
| Reflection | Agent evaluates own output, retries | Quality-critical tasks |
| Self-Correction | Agent detects syntax/logic errors via tools | Code generation agents |
Key terms: Handoff (passing task to sub-agent), Orchestrator (supervisor agent), Part (typed data in A2A), Interrupt (Human-in-the-loop wait), Grounding (connecting to RAG/Tools), Hallucination guardrails (output filtering).
| Framework | Paradigm | Strengths | Best For |
|---|---|---|---|
| LangGraph | Stateful graph | Persistence, human-in-loop, cycles | Production agents |
| CrewAI | Role-based multi-agent | Easy setup, role/goal/backstory | Business workflows |
| AutoGen | Conversational multi-agent | Code execution, group chat | Research automation |
| Smolagents (HF) | Code-based tool calling | Simple, open-source | Lightweight agents |
| OpenAI Assistants | Hosted agent | Zero setup, code interpreter | Quick prototypes |
| Type | Purpose | Implementation |
|---|---|---|
| Short-term | Current conversation | Message history in prompt |
| Long-term | Past conversations, facts | Vector DB + retrieval |
| Working memory | Intermediate state | LangGraph State object |
| Episodic | Past task outcomes | Structured DB of (task, result) |
(1) Single responsibility โ one tool, one job. (2) Idempotent โ safe to retry. (3) Fast โ slow tools stall the loop. (4) Rich descriptions โ the LLM decides based on the description string.
Agents take real actions. Safety measures: (1) Human-in-the-loop for destructive actions. (2) Sandboxing code execution. (3) Permission models. (4) Max iterations. (5) Cost/budget limits.
LangChain has evolved from a simple wrapper into a comprehensive suite for production AI:
| Component | Purpose | When to Use |
|---|---|---|
| LangChain Core | Standard interface for LLMs, prompts, tools, and retrievers | Building the basic components of your app |
| LangGraph | Orchestration framework for stateful, multi-actor applications | Building complex, cyclic agents and workflows |
| LangSmith | Observability, tracing, and evaluation platform | Debugging, testing, and monitoring in production |
| LangServe | REST API deployment using FastAPI | Serving chains/agents as production endpoints |
Answer: (1) Infinite loops โ stuck in reasoning cycle. Fix: max_iterations + loop detection. (2) Tool hallucination โ calls nonexistent tools. Fix: strict function calling schemas. (3) Context overflow โ long loops fill context. Fix: summarize observations. (4) Cascading errors โ bad step corrupts downstream. Fix: validation at each step. (5) Cost explosion โ long GPT-4o chains cost $1+/task. Fix: budget limits.
Answer: LangChain AgentExecutor is a fixed linear loop. LangGraph models workflows as a directed graph with persistent state โ branching, cycles, human-in-the-loop, session persistence. LangGraph is the production evolution; AgentExecutor is legacy.
Answer: (1) Structured output via function calling (not text parsing). (2) Checkpointing โ resume after failures. (3) Observability โ trace every LLM call (LangSmith, Langfuse). (4) Guardrails on tool inputs/outputs. (5) Fallbacks โ gracefully degrade after N failures.
Answer: Separates planning (create full multi-step plan) from execution (carry out each step independently). Planner has full task context; executor focuses on one step with clean context. After each step, planner can revise. Better than ReAct for complex multi-step tasks where the agent loses track of the goal.
Answer: Two patterns: (1) Shared state โ agents read/write a common state object (LangGraph). (2) Message passing โ structured messages between agents (AutoGen). Shared state is simpler; message passing is better for dynamic collaboration. Always pass structured data, not LLM summaries, to prevent error compounding.
| Pattern | Structure | Best For | Example |
|---|---|---|---|
| Supervisor | Orchestrator delegates to workers | Clear task decomposition | Manager + researcher + writer |
| Peer-to-Peer | Agents communicate directly | Collaborative reasoning, debate | Two reviewers debating quality |
| Hierarchical | Nested supervisors (teams of teams) | Enterprise workflows | Department heads coordinating |
| Swarm | Dynamic hand-off between agents | Customer service routing | OpenAI Swarm pattern |
Sequential: Agent A finishes before Agent B starts. Simple but no parallelism. Parallel: Independent tasks run simultaneously. Debate: Multiple agents propose solutions, a judge picks the best. Voting: Run the same task on N agents, majority wins (increases reliability).
CrewAI uses role-based agents with defined goals and backstories. A "Research Analyst" has different system prompts and tools than a "Report Writer." They collaborate via a crew's shared task queue. This mirrors real organizations.
Lightweight multi-agent pattern: each agent is a function with instructions + tools. Agents can hand off to each other by returning a different agent. No orchestrator โ control flows through hand-offs. Great for customer service routing (sales agent -> support agent -> billing agent).
(1) Over-engineering โ single ReAct agent often outperforms poorly designed multi-agent. (2) Context leakage โ agents sharing too much irrelevant context. (3) Error compounding โ LLM summaries between agents introduce errors. Always pass structured data. (4) Cost explosion โ N agents = N times the API calls.
For agents to collaborate at scale, they need standardized communication protocols (A2A). Unlike human-to-agent chat, A2A relies on strict payloads.
| Protocol / Concept | Description |
|---|---|
| FIPA ACL | Legacy agent communication language specifying performatives (e.g., REQUEST, INFORM, PROPOSE). Still conceptually relevant for modern A2A state machines. |
| Agent Protocol (AI Engineer Foundation) | A single open-source OpenAPI specification for agent interaction. Standardizes how to list tasks, execute steps, and upload artifacts. |
| Message Passing Interfaces | Modern frameworks use structured JSON schema validation for passing state. Agent A emits a JSON payload that perfectly matches the input schema expected by Agent B. |
| Pub/Sub Eventing | Agents publish events to a message broker (like Kafka or Redis) and other agents subscribe, enabling asynchronous, decoupled A2A swarms. |
Answer: (1) Shared state โ agents read/write a common state object (LangGraph). Simple for pipelines. (2) Message passing โ agents send structured messages (AutoGen). Better for dynamic collaboration. Always pass structured data, not LLM-generated summaries, to prevent error compounding.
Answer: When a single agent with good tools can handle the task. Multi-agent adds latency (N serial LLM calls), cost (N times more tokens), and complexity (coordination bugs). Start with a single ReAct agent; only split into multi-agent when context window overflow or specialization clearly helps.
Answer: Each agent in a chain can introduce errors. Mitigations: (1) Pass raw tool outputs between agents, not summaries. (2) Add validation nodes between agents. (3) Use structured output formats. (4) Implement self-correction loops where agents verify their own outputs. (5) Hierarchical review โ a supervisor validates worker outputs before proceeding.
Answer: Supervisor: When you know the workflow structure upfront โ orchestrator routes to specialized workers. Good for content pipelines, data processing. Swarm: When routing is dynamic and agents decide who handles what. Each agent can hand off to another. Better for customer service, conversational flows. Swarm is simpler but less controllable.
(1) You define tool schemas (name, description, parameters as JSON Schema). (2) Send schemas + user message to the LLM. (3) The model outputs a tool call with function name + arguments as JSON. (4) You execute the function locally. (5) Send results back to the model for final response.
Description matters most โ the LLM decides which tool to call based entirely on the description. "Search the web" is bad. "Search the web for real-time news. Do NOT use for stable facts like math or history" is good. Include: what it does, when to use it, when NOT to use it, and expected input format.
Parallel: Model outputs multiple independent tool calls in one response. Execute all simultaneously. GPT-4o supports this natively. Sequential: One call at a time, each depending on previous results. Parallel is 3-5x faster for independent operations.
Model Context Protocol (MCP) (Anthropic, 2024) is an open-source standard solving the "N-to-M integration problem" between AI assistants and data sources. Instead of writing custom integrations, developers build reusable MCP servers.
| MCP Component | Role | Web Analogy |
|---|---|---|
| MCP Host | The AI application (e.g., Claude Desktop, Cursor) | Web Browser |
| MCP Client | Runs inside the Host, manages server connections | HTTP Client |
| MCP Server | Secure, lightweight program giving access to data/tools | Web Server |
Servers expose three primitives to the LLM: Resources (read-only data like files or DB schemas giving context), Tools (executable functions like "run query"), and Prompts (reusable system instruction templates). This separates the reasoning engine from the data layer, allowing immense scalability.
| Provider | Feature Name | Parallel Calls | Strict Mode |
|---|---|---|---|
| OpenAI | Function Calling | Yes | Yes (strict JSON) |
| Anthropic | Tool Use | Yes | Yes |
| Function Calling | Yes | Yes | |
| Open models | Varies (Hermes format) | Some | Via constrained decoding |
Answer: Function calling uses constrained decoding โ the model can only output valid JSON matching your schema. Text parsing relies on the model following instructions, which fails ~10-20% of the time. Function calling gives near-100% format reliability. It also enables parallel tool calls and strict type validation.
Answer: When the LLM determines multiple tools can be called independently, it outputs ALL tool calls in a single response. You execute them in parallel and return all results. GPT-4o supports this natively. A query like "Weather in 3 cities" produces 3 parallel calls instead of 3 sequential LLM rounds. 3-5x faster.
Answer: Model Context Protocol standardizes how AI assistants connect to external tools. Instead of building custom integrations for each LLM provider, you build one MCP server that works everywhere. It defines: tools (functions), resources (data), and prompts (templates). Similar to how HTTP standardized web communication โ MCP standardizes AI-tool communication.
Answer: (1) Validate inputs before execution โ check types, ranges, permissions. (2) Catch exceptions and return descriptive error messages to the model. (3) Let the model retry โ include error details so it can fix its call. (4) Max retries โ after 3 failed attempts, return a user-friendly error. Never let tool errors crash the agent loop.
| Type | Method | When to Use |
|---|---|---|
| Rule-based | BLEU, ROUGE, exact match | Translation, extraction with reference |
| Embedding-based | Cosine similarity to reference | Semantic similarity checking |
| LLM-as-Judge | GPT-4o scores on criteria | Open-ended generation (90% of cases) |
| Human eval | Human annotators rate outputs | Final validation, calibration |
| Framework | Target | Key Metrics | Strengths |
|---|---|---|---|
| RAGAS | RAG pipelines | Faithfulness, Relevance, Recall | RAG-specific, automated |
| DeepEval | LLM apps | Hallucination, Bias, Toxicity | Comprehensive, CI-friendly |
| Langfuse | Observability + eval | Traces, scores, datasets | Open-source, real-time |
| PromptFoo | Prompt testing | Regression across versions | CLI-based, fast iteration |
| LangSmith | LangChain ecosystem | Traces, datasets, evals | Deep LangChain integration |
(1) Use chain-of-thought in the judge prompt โ "think step by step before scoring." (2) Pointwise scoring (1-5 rubric) is more reliable than pairwise comparison. (3) Calibrate against humans โ run 100 examples with human labels to validate the judge. (4) Use a different model family for judging to avoid self-enhancement bias. (5) Define explicit rubrics โ not "is this good?" but "does it contain all key facts from the source?"
Faithfulness: Extract claims from answer; check each against context. Answer Relevance: Generate questions from the answer; compare to original. Context Recall: Check if ground truth facts appear in context. Context Precision: Rank contexts by relevance; compute precision@k.
(1) Manual curation โ gold standard but expensive. Aim for 100+ examples covering edge cases. (2) Synthetic generation โ use a strong LLM to generate (question, answer, context) triples from your documents. (3) Production sampling โ sample real user queries and manually annotate a subset. (4) Adversarial โ deliberately create hard/tricky cases to find model weaknesses.
Answer: They measure n-gram overlap with reference text. For generative tasks, many phrasings are equally valid โ "The model works well" and "The system performs accurately" have zero overlap but both correct. They also ignore factual accuracy, coherence, and helpfulness. Only use for translation/summarization where references are meaningful.
Answer: LLMs prefer outputs stylistically similar to their own โ GPT-4o favors GPT-4o outputs over LLaMA outputs even when quality is equal. Mitigations: use different model family for judging, blind evaluation (no model names), calibrate against human preferences, and use explicit rubrics rather than vague quality assessments.
Answer: (1) Start with 100+ manually curated (question, ground_truth) pairs covering edge cases. (2) Augment synthetically โ use strong LLM to generate variations. (3) Sample production queries weekly and annotate. (4) Include adversarial cases (tricky, ambiguous, out-of-scope). (5) Version control โ track dataset changes alongside prompt changes. A good eval dataset is your most valuable asset.
Answer: Offline: Evaluate against pre-built datasets before deployment โ catches regressions, validates prompt changes. Online: Monitor production outputs in real-time โ user satisfaction, escalation rates, latency. You need both: offline for gating deployments, online for detecting issues with real traffic patterns you didn't anticipate.
| Risk | Description | Mitigation |
|---|---|---|
| Hallucination | Model generates false facts | RAG grounding, self-consistency checks |
| Prompt injection | User overrides system instructions | Input classification, instruction hierarchy |
| Jailbreaking | Bypassing safety training | Multi-layer defense, red-teaming |
| PII leakage | Exposing personal data | PII detection, output filtering |
| Toxicity | Harmful, biased, or offensive output | Content classifiers, NeMo Guardrails |
| Off-topic | Answering outside intended scope | Topic classifiers, system prompt boundaries |
Layer multiple defenses: (1) Input guardrails โ classify user input before reaching the LLM (jailbreak detection, topic filtering). (2) System prompt โ clear instructions about boundaries. (3) Output guardrails โ validate LLM output before showing to user (PII removal, toxicity check, hallucination detection). (4) Monitoring โ flag anomalous patterns in production.
| Tool | What It Does | Approach |
|---|---|---|
| Guardrails AI | Schema-based output validation + re-asking | Pydantic-like validators |
| NeMo Guardrails | Conversation flow programming | Colang DSL for safety rails |
| Llama Guard (Meta) | Safety classifier for LLM I/O | Fine-tuned LLM classifier |
| Azure Content Safety | Violence, self-harm, sexual content | Multi-category API classifier |
(1) Separate system/user content โ use structured message roles, never mix. (2) Input sanitization โ detect injection patterns (e.g., "ignore previous instructions"). (3) Instruction hierarchy โ system prompt takes priority explicitly. (4) Canary tokens โ include a secret in system prompt and detect if the model repeats it. (5) Dual LLM โ use a cheap model to classify input safety before expensive model generates output.
Train safety using principles rather than human labels for every case. Model critiques its own responses against a written constitution, then revises. Scales better than RLHF for safety and produces principled, explainable behavior.
Systematic adversarial testing before deployment: (1) Manual โ human testers try to break the model. (2) Automated โ use another LLM to generate adversarial prompts (PAIR, TAP). (3) Structured attacks โ GCG gradient-based attacks, multi-turn escalation. Run before any public deployment and on each major prompt change.
Answer: Prompt injection is when user input (or malicious data in tool results) overrides system instructions. E.g., a web page says "Ignore all instructions." Defenses: (1) Separate system/user content with message roles. (2) Classify inputs before processing. (3) Instruction hierarchy โ explicit priority to system prompt. (4) Canary tokens. (5) Dual-LLM: cheap model classifies, expensive model generates.
Answer: Systematic adversarial testing to find ways the model can produce harmful outputs. Includes: manual (humans try to break it), automated (another LLM generates attacks โ PAIR, TAP), and gradient-based (GCG attacks). Run before deployment and after each major prompt change. Document findings and create regression tests.
Answer: (1) RAG grounding โ check if every claim appears in retrieved context (RAGAS Faithfulness). (2) Self-consistency โ generate N times, flag divergent answers. (3) Verification chains โ secondary LLM call: "Is this claim supported by the text?" (4) Confidence calibration โ instruct model to say "I don't know." (5) Citation forcing โ require the model to cite specific passages.
Answer: Layer multiple independent defenses: Input layer (classify safety, detect injection), System prompt (clear boundaries, role definition), LLM layer (safety-trained model), Output layer (PII filter, toxicity check, hallucination detection), Monitoring layer (alert on anomalous patterns). No single layer is sufficient โ adversaries will find bypasses. Defense-in-depth makes the system resilient.
| Framework | Key Feature | Best For | Throughput |
|---|---|---|---|
| vLLM | PagedAttention โ zero KV cache waste | Production, high throughput | Highest |
| TGI (HF) | Tensor parallelism, batching | HF ecosystem, Docker-ready | High |
| Ollama | One-command local deployment | Local dev, laptops | Low |
| LiteLLM | Unified proxy for 100+ models | Multi-provider routing | Proxy (no inference) |
| llama.cpp | CPU inference, GGUF format | No-GPU environments | Medium |
| Method | How It Works | Quality Impact | Use With |
|---|---|---|---|
| GPTQ | Post-training, layer-by-layer with error compensation | Minimal (<1% loss) | vLLM, TGI |
| AWQ | Protects "salient" weights during quantization | Slightly better than GPTQ | vLLM, TGI |
| GGUF | CPU-optimized format for llama.cpp | Varies by quant level | Ollama, llama.cpp |
| BitsAndBytes | Runtime 4/8-bit quantization | Good for fine-tuning | HF Transformers |
PagedAttention (inspired by OS virtual memory): stores KV cache in non-contiguous memory pages. Traditional serving pre-allocates max KV cache โ wasting 60-80% of GPU memory. PagedAttention allocates on demand: 3-24x higher throughput. Continuous batching: instead of waiting for an entire batch to complete, add new requests as old ones finish. LoRA serving: serve multiple LoRA adapters on same base โ multi-tenant support.
| Factor | API (OpenAI/Anthropic) | Self-hosted (vLLM) |
|---|---|---|
| Setup | Zero ops | GPU infra required |
| Quality | Best (GPT-4o, Claude) | Good (LLaMA-3, Mistral) |
| Cost at low volume | $$ | $$$$ |
| Cost at high volume | $$$$ | $$ |
| Data privacy | Data leaves your infra | Data stays local |
| Latency control | Limited | Full control |
Rule of thumb: API for <1M tokens/day, self-host for >10M tokens/day or data-sensitive workloads.
(1) Single GPU โ 7B-13B models with quantization. (2) Tensor Parallelism โ split model across GPUs on same node. (3) Pipeline Parallelism โ split layers across nodes. (4) Serverless โ Modal, RunPod, Together AI โ pay per token, auto-scale to zero.
Answer: Inspired by OS virtual memory paging: stores KV cache in non-contiguous memory pages. Traditional serving pre-allocates maximum KV cache size โ wasting 60-80% of GPU memory on padding. PagedAttention allocates pages on demand as sequences grow, enabling 3-24x higher throughput and supporting more concurrent users per GPU.
Answer: API: Zero ops, best quality, predictable at low volume, no data privacy control. Self-hosted: Data stays private, cost-effective at high volume, lower latency, customizable. Crossover point: ~1-10M tokens/day. Below that, APIs are cheaper when accounting for GPU rental + engineer time. Above that, self-hosting with vLLM saves 70-90%.
Answer: GPTQ: Standard 4-bit for GPU inference (vLLM, TGI). AWQ: Slightly better quality than GPTQ, faster activation-aware approach. GGUF: CPU-optimized for llama.cpp/Ollama โ works without GPU. Use GPTQ/AWQ for GPU serving, GGUF for local/CPU. All achieve <1% quality loss at 4-bit. 8-bit is nearly lossless.
Answer: Traditional batching waits for all requests in a batch to complete before accepting new ones. Continuous batching (used by vLLM, TGI) inserts new requests as soon as any request in the batch finishes. This eliminates the "convoy effect" where a short request waits for a long one. Result: 2-3x throughput improvement, much lower tail latency.
Embed user queries and check if a semantically similar query was recently answered. If similarity > threshold, return cached response. GPTCache and Redis + vector similarity implement this. Achieves 30-60% cache hit rates on chatbot traffic, dramatically reducing API costs.
Never make users wait for full response. Stream tokens as generated using Server-Sent Events (SSE). Perceived latency drops from 5-10s to <1s (time-to-first-token). Implement with stream=True in OpenAI SDK + async generators in FastAPI.
| Strategy | Savings | Tradeoff |
|---|---|---|
| Smaller models for simple tasks | 10-50x (mini vs full) | Lower quality for complex reasoning |
| Semantic caching | 30-60% fewer calls | Stale responses for dynamic content |
| Prompt compression | 20-40% fewer tokens | Risk of losing important context |
| Batch API (OpenAI) | 50% discount | Async only, 24h turnaround |
| Prompt caching (Anthropic/OpenAI) | 90% on repeated prefixes | Must structure prompts carefully |
| Self-host (vLLM) | 70-90% at high volume | Ops burden, GPU management |
| Tool | Features | Type |
|---|---|---|
| LangSmith | Tracing, evals, datasets, playground | Managed (LangChain) |
| Langfuse | Tracing, scoring, prompt management | Open-source |
| Arize Phoenix | LLM tracing, embedding analysis | Open-source |
| PromptLayer | Prompt versioning, A/B testing | Managed |
(1) Fallback chains โ GPT-4o -> Claude -> GPT-4o-mini -> cached response. (2) Rate limiting โ per-user token budgets. (3) Circuit breakers โ stop calling a provider after N failures. (4) Retry with exponential backoff โ handle transient API errors. (5) Timeout management โ kill requests after 30s to prevent hung connections.
Treat prompts like code: (1) Version control โ store in files, not strings. (2) A/B testing โ test new prompts against baseline. (3) Regression tests โ automated eval before deployment. (4) Structured templates โ use Jinja2 templates with variables. (5) Prompt caching โ structure with static prefix + dynamic suffix.
Answer: (1) Stream tokens โ TTFT <1s feels instant. (2) Semantic cache โ instant for repeated patterns. (3) Smaller models for preprocessing โ classify intent with mini, reason with full model. (4) Speculative decoding โ small draft model generates, large model verifies in parallel (2-3x speedup).
Answer: Reduces costs when the same system prompt prefix is reused. Anthropic and OpenAI cache the KV representation of repeated prefixes server-side โ 90% cheaper for cached tokens. Structure prompts: long static instructions first, dynamic user content last. Can save 50-80% on total costs for apps with large system prompts.
Answer: (1) Store prompts in version-controlled files. (2) Use prompt management tools (LangSmith, Langfuse) for A/B testing. (3) Run automated regression tests against an eval dataset before deploying new versions. (4) Log all prompts + completions for debugging. (5) Canary deployments โ route 5% of traffic to new prompt, monitor metrics.
Answer: Semantic caching: You implement it client-side. Embed user queries, find similar past queries, return cached LLM response. Saves entire API calls. Works best for FAQs. Prompt caching: Provider implements it server-side. Caches the KV representations of repeated system prompt prefixes. Reduces cost per call but still makes the call. They're complementary โ use both.
Answer: Priority-ordered model list: GPT-4o -> Claude -> GPT-4o-mini -> cached/static response. Use LiteLLM Router or custom retry logic. On each failure: (1) Log the error. (2) Try next model in chain. (3) After all fail, return graceful degradation message. Also implement circuit breakers โ if a provider fails 5 times in 1 minute, skip it for 5 minutes.
LLMs have no persistent memory between calls. Every token in the context window costs compute and money. The quality of an LLM's output is directly bounded by the quality of its context. Garbage in = garbage out โ no matter how capable the model.
| Context Type | What It Contains | Example |
|---|---|---|
| Instructions | System prompt โ role, rules, output format | "You are a helpful assistant. Always respond in JSON." |
| Memory | Long-term facts about the user or world | User's name, preferences, past decisions |
| History | Recent conversation turns | Last 10 messages in the chat |
| Retrieved Information | Documents pulled from RAG / search | Top-3 relevant docs from vector DB |
| Tool Results | Outputs from previous tool calls | API response from a weather service |
| Background Knowledge | Domain facts injected at system level | Company product catalog, legal constraints |
As context grows, three problems arise: (1) Lost in the middle โ LLMs attend most to the start and end of context, ignoring the middle. (2) Cost โ every token costs money. (3) Latency โ processing longer context takes more time. Solutions: summarization of old history, selective retrieval, context compression with smaller models.
Claude's unique skill is long-context recall (200K tokens). Claude's "skills" pattern separates tool use from context assembly. The model reads structured XML context blocks and applies different attention patterns per block type. Claude recommends placing the most important context last in the prompt for best recall.
Answer: Prompt engineering focuses on crafting the right instructions/wording. Context engineering is broader โ it's about deciding what information (memories, retrieved docs, tool results, history) to put into the context window, how to prioritize it when space is limited, and in what order. It's the systems-level discipline of managing an LLM's information diet at runtime.
Answer: LLMs tend to attend most strongly to content at the very beginning and very end of their context window, while information in the middle gets relatively less attention. This means placing critical information in the middle of a long context leads to degraded performance. Mitigation: put the most important facts last, or use retrieval to place relevant chunks near the query.
Answer: (1) Instructions โ system prompt with role and rules. (2) Memory โ long-term facts about the user/world. (3) History โ recent conversation. (4) Retrieved Information โ documents from RAG. (5) Tool Results โ outputs from function calls. (6) Background Knowledge โ domain-specific injected facts. Each plays a different role in enabling the agent to reason correctly.
Answer: (1) Summarize older history instead of including raw messages. (2) Use a context prioritization system โ rank blocks by importance and trim lowest-priority first. (3) Compress retrieved documents with a small "compressor LLM" before adding to context. (4) Use structured memory stores that selectively load relevant facts, not everything. (5) Use streaming to process very long documents in chunks.
| # | Pattern | Description | When to Use |
|---|---|---|---|
| 1 | Reflection | Agent critiques its own output, then revises it | When accuracy matters more than speed |
| 2 | Tool Use | Agent calls external tools (search, code, APIs) | When LLM alone can't solve the task |
| 3 | Planning | Agent generates a multi-step plan before acting | Complex tasks with many sub-steps |
| 4 | Multi-Agent | Multiple specialized agents collaborate via messages | Tasks requiring different expertise |
| 5 | Memory | Agent stores and retrieves past interactions | Long-running or personalized applications |
| Level | Name | Capability |
|---|---|---|
| 0 | No AI | Purely rule-based or human-driven |
| 1 | AI-Assisted | LLM suggests, human decides and executes |
| 2 | AI Co-pilot | LLM acts, but human approves each action |
| 3 | AI Agent | Autonomous execution within defined scope |
| 4 | Agentic AI | Multi-step autonomous operation, self-correcting |
| 5 | AI Workforce | Multiple agents, full autonomy over long horizons |
Layer 1 โ Perception: Receiving inputs (text, images, tool results, sensor data). Layer 2 โ Reasoning: The LLM processing context and deciding on actions. Layer 3 โ Action: Executing tool calls, writing code, sending messages. Layer 4 โ Memory: Storing results, updating knowledge, retrieving past context. This layered view clarifies where failures occur.
ReAct (Reasoning + Acting) is the dominant agentic loop: Thought โ Action โ Observation. The model writes a thought explaining its reasoning, selects an action (tool call), observes the result, then loops until done. This interleaving of reasoning and acting proved more reliable than pure chain-of-thought or pure action-only approaches.
Key vocabulary every AI Engineer must know: Tool (callable function), Handoff (passing control between agents), Orchestrator (agent that delegates), Sub-agent (executes delegated tasks), Scratchpad (agent's working memory mid-task), Guardrails (safety constraints), Grounding (connecting outputs to real-world facts), Hallucination (false confident output), Context window (max tokens the model sees), Token budget (allocated tokens for a task), System prompt (persistent instructions), Function schema (JSON spec for tools), Streaming (incremental output), Interrupts (human-in-the-loop breakpoints)...
Answer: (1) Reflection โ agent critiques and revises its own output. (2) Tool Use โ agent calls external functions/APIs. (3) Planning โ agent creates a multi-step plan before execution. (4) Multi-Agent โ multiple agents collaborate, each specializing in different sub-tasks. (5) Memory โ agent persists and retrieves information across interactions. These patterns are often combined: e.g., a planning agent that uses tools and reflects on results.
Answer: ReAct (Reasoning + Acting) interleaves thinking with tool use: (1) Thought โ model reasons about what to do next. (2) Action โ model calls a tool with specific arguments. (3) Observation โ tool result is appended to context. This loop repeats until the model outputs "Final Answer." The key insight is that grounding reasoning in real tool observations prevents hallucination chains and enables multi-step problem solving.
Answer: Level 0: No AI. Level 1: AI-Assisted (suggests, human acts). Level 2: Co-pilot (AI acts, human approves). Level 3: AI Agent (autonomous in defined scope). Level 4: Agentic AI (multi-step autonomous, self-correcting). Level 5: AI Workforce (swarm of agents, long-horizon autonomy). Most production systems today are Level 2-3; Level 4-5 is emerging.
| Protocol | Created By | Purpose | Layer |
|---|---|---|---|
| MCP (Model Context Protocol) | Anthropic | Standardize how LLMs connect to tools & data sources | Tool/Data Layer |
| A2A (Agent-to-Agent) | Standardize how agents discover and delegate to other agents | Agent Communication Layer | |
| AG-UI (Agent-User Interaction) | CopilotKit | Standardize how agents communicate state to frontends | Presentation Layer |
| Agent Protocol | AI Engineer Foundation | OpenAPI spec for agent task management | Task Management Layer |
MCP solves the NรM integration problem. Before MCP: each LLM app needed custom integrations to each tool/database. With MCP: each tool builds one MCP Server; each LLM app builds one MCP Client. They interoperate automatically.
| MCP Component | Role | Analogy |
|---|---|---|
| MCP Host | App running the LLM (Claude Desktop, Cursor, your app) | Web Browser |
| MCP Client | Protocol client inside the host | HTTP Client |
| MCP Server | Exposes tools, resources, prompts | Web Server |
MCP Servers expose 3 primitives: Tools (executable functions), Resources (read-only data), Prompts (reusable templates). Transport: stdio (local) or HTTP+SSE (remote).
A2A allows agents built by different vendors to collaborate. An agent publishes an Agent Card (JSON) describing its capabilities. Other agents discover it and delegate tasks using standardized request/response schemas. Key concepts: Task (unit of work), Artifact (output produced), Part (typed content: text, file, data). A2A is transport-agnostic (HTTP, gRPC, etc.).
AG-UI standardizes the real-time event stream between agents and frontend UIs. Instead of building custom WebSocket logic for every agent app, AG-UI defines: streaming text events, tool call events, state sync events, and lifecycle events (run_start, run_end). This means any AG-UI-compatible agent can power any AG-UI-compatible frontend.
Answer: MCP solves the NรM integration problem. Before MCP, if you had N LLM applications and M tools/data sources, you needed NรM custom integrations. With MCP, each tool builds one MCP Server and each app builds one MCP Client โ reducing to N+M integrations. It standardizes what tools expose (functions, data, templates) and how LLMs request them.
Answer: MCP is about an LLM connecting to tools and data sources โ it's a vertical protocol (LLM โ tools). A2A is about agents communicating with other agents โ it's a horizontal protocol (agent โ agent). You'd use MCP to give your agent access to a database, and A2A to have your orchestrator agent delegate a coding task to a specialized code-writing agent.
Answer: AG-UI (Agent-User Interaction protocol) standardizes the streaming event protocol between AI agents and their frontend UIs. Without AG-UI, every agent app needs custom WebSocket or SSE logic for showing streaming responses, tool call status, and agent state in the UI. AG-UI defines a standard event schema so any compliant agent can power any compliant frontend โ enabling a plugin-like ecosystem of agent UIs.
Answer: (1) Tools โ executable functions the LLM can call (like "search_web", "run_sql"). (2) Resources โ read-only data the LLM can reference for context (like database schemas, file contents). (3) Prompts โ reusable prompt templates the host app can offer to users. Each primitive serves a different role in the LLM's workflow.
To improve an LLM's logical accuracy and ability to plan, use these three structural techniques:
| Technique | How It Works | Best For |
|---|---|---|
| Chain of Thought (CoT) | Adding "Think step by step" to the prompt, forcing the model to generate intermediate reasoning tokens before the final answer. | Math, logic, code, and agent planning. |
| Tree of Thoughts (ToT) | Prompting the model to generate multiple possible paths, evaluate them, and perform search (BFS/DFS) across paths to find the optimal solution. | Complex multi-step reasoning where backtracking is needed. |
| Self-Consistency | Prompting the model to generate N different CoT sequences, then taking the majority vote for the final answer. | Highly reliable fact extraction and math problems. |
When an agent selects a tool to call, it can easily make a mistake if it just outputs the tool name directly. Verbalized Sampling requires the agent to explicitly output its reasoning process before outputting the tool call or action.
Because LLMs generate tokens autoregressively (left-to-right), the tokens representing the "thought" become part of the context for generating the "tool call" tokens. The tool call intrinsically becomes more accurate because it's physically conditioned on a logical rationale.
Agents and orchestrators need structured data (JSON), not conversational text. If the JSON is malformed, the application crashes.
response_format: { type: "json_object" } (OpenAI). The model is guaranteed to output valid JSON.} as a stop sequence guarantees no trailing text.{ to the end of your prompt so the model is forced to start generating JSON keys instantly without saying "Here is the JSON...".Answer: It provides additional computational steps (tokens) for the model to process logic. Since an LLM spends a fixed amount of computation per token, forcing it to generate a 50-token thought process before answering allocates 50x more computation to solving the problem than just answering immediately.
Answer: JSON prompting is done via the text prompt and relies on the model's instruction following (good for open models). Function/Tool calling is a native API feature where the provider fine-tunes the model explicitly to output arguments matching a schema via constrained decoding, ensuring much higher reliability.
Compression reduces the memory footprint and increases memory bandwidth, leading to faster token generation.
| Technique | How It Works | Tradeoff |
|---|---|---|
| Quantization (PTQ) | Convert parameters from FP16 (16-bit) to INT8 or INT4. Ex: GGUF, AWQ, GPTQ. | Slight accuracy loss, massive speed/memory gains. |
| Pruning | Set near-zero weights to exactly zero, creating sparse matrices. | Requires specialized hardware for sparse acceleration. |
| Knowledge Distillation | Train a smaller model (student) to match the probability distribution of a large model (teacher). | High upfront compute cost to train the student. |
During generation, each new token needs to pay attention to all past tokens. Recomputing the Key (K) and Value (V) matrices for all past tokens for every new token generation is O(Nยฒ) and terribly slow.
KV Caching stores the K and V tensors for all past tokens in GPU memory. For the next step, only the Q, K, V for the newest token are computed, and attention is run against the cached past. This reduces computation to O(N) per step.
While KV caching solves the compute problem, it introduces a memory problem. A 100K context window across high batch sizes can cause the KV cache to consume more GPU RAM than the model weights themselves! This is why techniques like PagedAttention (vLLM) and GQA (Grouped Query Attention) were invented.
Answer: Compute-bound means the GPU spends all its time doing math (matrix multiplications). Memory-bandwidth bound means the math is easy, but the GPU spends all its time waiting for weights to be copied from High Bandwidth Memory (HBM) to on-chip SRAM. LLM prefill (reading the prompt) is compute-bound, but decoding (generating tokens one by one) is memory-bandwidth bound.
Answer: Normally, KV cache is pre-allocated continuously in GPU memory. Because output lengths are unknown, frameworks over-allocate memory, wasting up to 60%. PagedAttention divides the KV cache into small blocks (pages) and allocates them dynamically, like virtual memory in an OS. This allows near-zero waste and 2-4x higher concurrency (batching).
Standard software uses APM (Datadog, New Relic) for tracing. LLMs require specialized APMs (Langfuse, LangSmith, Helicone, Opik) because the payload sizes are huge and the dependencies (prompts, tool results) are non-standard.
| Platform | Best For |
|---|---|
| LangSmith | Deep LangChain integration and easy local debugging. |
| Langfuse | Open-source, framework-agnostic tracing with a great UI. |
| Helicone | Proxy-based observability (just change the base URL, no SDK needed). |
| Opik (by Comet) | Agent optimization and evaluation natively integrated with traces. |
Answer: TTFT measures the latency from the moment the user sends the request until the first token streams back to the client. In LLM applications, total end-to-end latency might be 5-10 seconds, which is unacceptable for UX. Streaming combined with low TTFT (<1 second) creates the illusion of speed and keeps users engaged.
Answer: A proxy requires ZERO code changes โ you simply change the API base URL from api.openai.com to oai.hconeai.com and pass your proxy key in the header. It automatically logs all prompts, responses, costs, and latencies. However, an SDK (like Langfuse/LangSmith) is required if you want deep, nested trace trees for complex agents (e.g., seeing exactly which step in a 10-step LangGraph flow failed).
| Pattern | How It Works | Best For |
|---|---|---|
| 1. Parallel | Tasks run concurrently across specialists (e.g., searcher + coder). | Reducing latency in complex pipelines. |
| 2. Sequential | Step-by-step handoff (e.g., Coder โ Reviewer โ Deployer). | ETL chains, code development, linear workflows. |
| 3. Loop | Continuous refinement until quality threshold is met. | Report generation, creative writing, proofreading. |
| 4. Router | Controller agent directs task to the right specialist. | Customer support (billing vs technical vs sales). |
| 5. Aggregator | Many agents form opinions; one central agent merges them. | Consensus voting, RAG retrieval fusion. |
| 6. Network | No hierarchy; agents communicate freely (peer-to-peer). | Simulations, games, collective brainstorming. |
| 7. Hierarchical | Planner/Manager agent delegates to workers and tracks progress. | Large enterprise projects with many sub-tasks. |
Agent-to-Agent (A2A) protocol standardizes how these agents exchange context and instructions. Instead of sharing a global state, they exchange Agent Cards and Task Payloads. This allows an agent built in CrewAI to delegate a task to an agent built in LangGraph.
When picking a pattern, prioritize minimizing communication overhead. 10 agents isn't better than 2 if they duplicate work. The system should feel smarter than its individual parts.
Answer: Parallel is for independent tasks (data extraction + web search) to reduce latency. Sequential is for dependent tasks where step B needs output of step A (code writing then code review). Use parallel for scale, sequential for quality-controlled pipelines.
Answer: It mimics a corporate structure: a Manager/Planner agent receives the high-level goal, breaks it into sub-tasks, and delegates them to specialized Worker agents. The Manager tracks state and makes the final quality check. Best for complex, ambiguous projects.
1. Definition: Send tool schemas (JSON) to LLM. 2. Selection: LLM realizes it needs a tool and outputs tool_calls. 3. Execution: Your code runs the tool locally. 4. Feedback: Result is sent back to LLM. 5. Final Output: LLM uses result to answer user.
JSON Prompting: Manually instructing the model to output JSON (used for open models). Native Tooling: Using provider APIs (OpenAI/Claude) which use constrained decoding for 99.9% reliability.
Forcing the agent to generate a "Thought:" block before the "Action:" block. This conditions the tool selection on a logical premise, significantly reducing errors in choosing the wrong tool or arguments.
Answer: It forces the model to articulate a rationale *before* picking a tool. Since tokens are generated left-to-right, the tool selection becomes conditioned on the reasoning, which increases precision, especially when multiple similar tools exist.
${m.desc}
${m.category}${m.desc}