docs/project/TORCH-NOTES.md

Tensor Types

1. Scaler(1 block)
2. Vector(3 blocks)
3. Matrix(3x3(9) blocks)
4. Tensor(3d Matrix - No set number on sides)

   Tensors - Tensor cores in NVIDIA CUDA architectures do not have a set number of "sides" in the traditional geometric sense, but they are designed to perform matrix-matrix multiplication on small 2D matrices (often referred to as 2D tensors) directly in a single clock cycle.

Harden and Optimize Ternary Gradient Functions

1. Forward Pass Suppose a neuron: y=wx+b

   Forward pass computes prediction.

   Example: x=2 w=0.5 b=0.1

2. Backward Pass(Where gradients come from) Now we compute: ∂L/∂w

   using the chain rule.

   This is the core equation of backpropagation: ∂L/∂w = ∂L/∂y ⋅ ∂y/∂w

   Break it apart.

   Step A — Derivative of Loss Loss: L=(y−t)^2 Derivative: ∂L/∂y =2(y−t) For our numbers: 2(1.1−3)=−3.8

   ---

   Step B — Derivative of Neuron Neuron: y=wx+b Derivative wrt w: ∂y/∂w =x Since: x=2

   ---

   Step C — Multiply Them ∂L/∂w =(−3.8)(2)=−7.6 That is the gradient. Meaning: Increasing w reduces loss strongly.

3. Gradient Descent Update Update rule: w(t+1) = w(t) −η(∂L/∂w)

If: w=0.5 learning rate =0.01

Then: w(new) =0.5−0.01(−7.6)=0.576 The weight moved upward because gradient was negative.

Important Topics to Keep in Mind A transformer has billions of parameters and Backward pass computes for EVERY tensor. This involves:

• matrix multiplications
• reductions
• accumulation
• activation derivatives
• chain-rule propagation

The backward pass is often MORE expensive than forward pass. Gradients can become:

• VERY small
• VERY large and training repeatedly accumulates them

---

FP32:

• 32 bits
• 23-bit mantissa
• ~7 decimal digits precision

Range: 10^−38 → 10^38

Good for:

• stable accumulation
• optimizer states
• gradient reductions

---

BF16:

• 16 bits
• 8-bit exponent
• 7-bit mantissa

Important:

• SAME exponent size as FP32
• LOWER precision

This means:

• range is good
• precision is poor

---

Why Backward Pass Often Uses FP32

During backprop you repeatedly do: g(total)​=g1​+g2​+g3​+...

Accumulation error becomes critical.

This is especially dangerous when: gi​≪1

because BF16 may round them away.

Example: 0.00097656

might become: 0

after quantization. So modern training often does:

Operation	Precision
Forward activations	BF16
Matrix multiply	BF16
Gradient accumulation	FP32
Optimizer state	FP32

This is called: mixed precision training

---

optimum.quanto v0.2.7