A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond

Definitions

Let's define standard reinforcement learning terms with an LLM setup in mind.

• State $s_{t}s\_t$: The current context which is the original user prompt and all tokens generated so far

  Example: Prompt: "The sky is..." $\to \backslash rightarrow$ State: ["The", "sky", "is"] in the token-space

• Action $a_{t}a\_t$: The next token being generated

  Example: "blue"

• Policy ${\pi }_{\theta }\backslash pi\_\backslash theta$: The LLM itself. Which is essentially a probability distribution over all words in a vocabulary given a state $s_{t}s\_t$.

  Example: ${\pi }_{\theta }(\backslash pi\_\backslash theta($ "blue" | ["The", "sky", "is"] $)=0.87)\; =\; 0.87$

• Trajectory $\tau \backslash tau$: The full conversation. The complete sequence of states (context) and actions (tokens chosen) from the prompt to the end-of-sequence token.

• Reward $RR$: The score. Usually assigned to the entire trajectory, indicating how good the full response was.

• Critic Network $V(s)V(s)$: A separate model (or head) that estimates the value of a state. It predicts how much future reward we're expecting to get given the current state. Also referred to as the Value Function.

• Reward Model: A separate model trained to learn human preferences or other rewards. Most commonly, it takes the full response and outputs the scalar reward $R\mathrm{.}R.$

• Reference model ${\pi }_{ref}\backslash pi\_\{ref\}$: The starting, pre-trained LLM after SFT before any RL was used.

On-Policy Algorithms

In on-policy learning, the model actively generates its own data during training. The model generates responses. The responses get scored and the model's parameters updated.

The Core Objective

The core objective is to maximize the Expected Return $J({\pi }_{\theta })J(\backslash pi\_\backslash theta)$.

$$J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[R(\tau )]J(\backslash pi\_\backslash theta)\; =\; \backslash mathbb\{E\}\_\{\backslash tau\; \backslash sim\; \backslash pi\_\backslash theta\}[R(\backslash tau)]$$

This means we are computing the expected value of the reward across all conversations sampled from our model ${\pi }_{\theta }\backslash pi\_\backslash theta$.

Note that, strictly speaking, the policy ${\pi }_{\theta }\backslash pi\_\backslash theta$ outputs probabilities for a single token at a time. We write that we're sampling the full trajectory (sequence) from ${\pi }_{\theta }\backslash pi\_\backslash theta$ but in fact, we need some decoding strategy (like ancestral sampling or top-p sampling). So this is just a short-hand for saying that we are autoregressively generating the sequence step-by-step based on the policy.

We want to find weights $\theta \backslash theta$ of our model that maximize the expected return $JJ$. To do this, we need to compute its gradients ${\mathrm{\nabla }}_{\theta }J({\pi }_{\theta })\backslash nabla\_\backslash theta\; J(\backslash pi\_\backslash theta)$.

$${\mathrm{\nabla }}_{\theta }J({\pi }_{\theta })={\mathrm{\nabla }}_{\theta }\int P(\tau \mathrm{\mid }\theta )R(\tau )d\tau \backslash nabla\_\backslash theta\; J(\backslash pi\_\backslash theta)\; =\; \backslash nabla\_\backslash theta\; \backslash int\; P(\backslash tau|\backslash theta)\; R(\backslash tau)\; \backslash ,\; d\backslash tau$$

The problem is that can't differentiate through a discrete generation process, such as sampling tokens in our case. In a normal neural network, we can compute gradients for every step. But when an LLM generates text (or more generally, when an action is picked), it perfoms a sampling step that is non-differentiable. Moreover, the reward is often non-differentiable too (unless we use a differentiable reward model). To solve this, we use the score function estimator, also known as the log-derivative trick.

We first move the gradient inside (assuming regularity): $$=\int {\mathrm{\nabla }}_{\theta }P(\tau \mathrm{\mid }\theta )R(\tau )d\tau =\; \backslash int\; \backslash nabla\_\backslash theta\; P(\backslash tau|\backslash theta)\; R(\backslash tau)\; \backslash ,\; d\backslash tau$$

and apply the log-derivative trick.

$$=\int \underset{\text{Replaced }{\mathrm{\nabla }}_{\theta }P}{\underbrace{P(\tau \mathrm{\mid }\theta ){\mathrm{\nabla }}_{\theta }\log P(\tau \mathrm{\mid }\theta )}}R(\tau )d\tau =\; \backslash int\; \backslash underbrace\{P(\backslash tau|\backslash theta)\; \backslash nabla\_\backslash theta\; \backslash log\; P(\backslash tau|\backslash theta)\}\_\{\backslash text\{Replaced\; \}\; \backslash nabla\_\backslash theta\; P\}\; R(\backslash tau)\; \backslash ,\; d\backslash tau$$

This let's us rewrite the gradient as an expectation and we can use sampling to estimate it.

$$={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[{\mathrm{\nabla }}_{\theta }\log P(\tau \mathrm{\mid }\theta )\cdot R(\tau )]=\; \backslash mathbb\{E\}\_\{\backslash tau\; \backslash sim\; \backslash pi\_\backslash theta\}\; \backslash left[\; \backslash nabla\_\backslash theta\; \backslash log\; P(\backslash tau|\backslash theta)\; \backslash cdot\; R(\backslash tau)\; \backslash right]$$

This is great because we no longer need the derivative of the reward function and can estimate the gradient by sampling trajectories.

Now, for LLMs, the probability of the whole text is the product of probabilities for each token:

$$P(\tau \mathrm{\mid }\theta )=\prod _{t=0}^T{\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)P(\backslash tau|\backslash theta)\; =\; \backslash prod\_\{t=0\}^T\; \backslash pi\_\backslash theta(a\_t\; |\; s\_t)$$

We can take the log of the product, which becomes a sum

$$\log P(\tau \mathrm{\mid }\theta )=\sum _{t=0}^T\log {\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)\backslash log\; P(\backslash tau|\backslash theta)\; =\; \backslash sum\_\{t=0\}^T\; \backslash log\; \backslash pi\_\backslash theta(a\_t\; |\; s\_t)$$

and substitute it back to the equation above. We can also generalize the reward term $R(\tau )R(\backslash tau)$ into a weight ${\mathrm{\Phi }}_t\backslash Phi\_t$:

$${\mathrm{\nabla }}_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[\sum _{t=0}^T\underset{\text{Direction}}{\underbrace{{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)}}\cdot \underset{\text{Weight}}{\underbrace{{\mathrm{\Phi }}_t}}]\backslash nabla\_\backslash theta\; J(\backslash pi\_\backslash theta)\; =\; \backslash mathbb\{E\}\_\{\backslash tau\; \backslash sim\; \backslash pi\_\backslash theta\}\; \backslash left[\; \backslash sum\_\{t=0\}^T\; \backslash underbrace\{\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(a\_t|s\_t)\}\_\{\backslash text\{Direction\}\}\; \backslash cdot\; \backslash underbrace\{\backslash Phi\_t\}\_\{\backslash text\{Weight\}\}\; \backslash right]$$

where

• ${\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(a\_t\; \backslash mid\; s\_t)$ is the gradient of the log-probability. It tells us exactly how to update model's parameters to make this specific token $a_{t}a\_t$ more or less likely (based on $R)R)$ given its context $s_{t}s\_t$
• ${\mathrm{\Phi }}_t\backslash Phi\_t$ is the weight of this update.

You can see different policy gradient algorithms as different implementations of the weight. On a high-level, you can see it as:

• If ${\mathrm{\Phi }}_t=R(\tau )\backslash Phi\_t\; =\; R(\backslash tau)$, which is the total reward of the trajectory, we get REINFORCE (or more commonly, we compute rewards to-go ${\mathrm{\Phi }}_t={\sum }_{k=t}^T{r}_k\backslash Phi\_t\; =\; \backslash sum\_\{k=t\}^T\; r\_k$ )

• If ${\mathrm{\Phi }}_t=Q(s_t,a_t)-V(s_t)\backslash Phi\_t\; =\; Q(s\_t,\; a\_t)\; -\; V(s\_t)$, the Advantage, we get Vanilla Policy Gradient or Actor-Critic methods.

REINFORCE

REINFORCE is essentially a direct implementation of the equation above. In the simplest case, the weight of updates is set as trajectory's total reward \R(\tau)\). It is intuitive: if the model writes a response and gets a high score, we reinforce every token it used. If it gets a low score, we discourage them.

The update rule looks like this, averaged over a batch of trajectories.

$${\mathrm{\nabla }}_{\theta }J({\pi }_{\theta })\approx \frac{1}{N}\sum _{i=1}^N\sum _{t=0}^T{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)\cdot R(\tau )\backslash nabla\_\backslash theta\; J(\backslash pi\_\backslash theta)\; \backslash approx\; \backslash frac\{1\}\{N\}\; \backslash sum\_\{i=1\}^N\; \backslash sum\_\{t=0\}^T\; \backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(a\_t|s\_t)\; \backslash cdot\; R(\backslash tau)$$

where

• ${\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(a\_t|s\_t)$ is the gradient of the log-probability as above

• $R(\tau )R(\backslash tau)$ is the score of the full response. It acts as a weight for the update. If the reward is positive, we move the weights with the gradient and increase the probability of the token. If the reward is negative, we suppress the probability of these tokens.

REINFORCE can be very inefficient. Imagine that a model generates a very good long response but hallucinates at the end, and the Reward Model assigns it a score of $-100.-100.$ REINFORCE will punish the entire generated text because it can't tell which token caused the score to be low. The biggest difficulty of this algorithm is high variance and instability.

PPO

Proximal policy optimization (PPO) is a family of policy gradient methods introduced by OpenAI in 2017 (Schulman et al. 2017), and fixes many of these instabilities. PPO is one of the most widely used algorithms in RL applications. It solves issues of Vanilla Policy Gradients and REINFORCE by using three main improvements:

• Generalized Advantage Estimation (GAE): PPO uses GAE for computing advantage values, which helps reduce variance in policy gradient estimates while maintaining low bias.

• Actor-Critic interplay: PPO introduces a critic model (also called the value function).

• Clipped Updates: PPO limits policy updates by using a clipped surrogate objective or by using an adaptive KL divergence penalty.

Advantage Estimation and Critic Network

Instead of using raw reward $RR$, PPO uses Advantage ${\widehat{A}}_t\backslash hat\{A\}\_t$. Advantage tells us how much better this specific action was compared to the expected baseline result.

Advantage also only looks at rewards obtained from step $tt$ and later. We ignore what happened before action $a_{t}a\_t$ because the new token cannot influence the past.

Advantage is defined as

$${\widehat{A}}_t=Q(s_t,a_t)-V(s_t)\backslash hat\{A\}\_t\; =\; Q(s\_t,\; a\_t)\; -\; V(s\_t)$$

where

• $Q(s_t,a_t)Q(s\_t,\; a\_t)$ are rewards-to-go. This is the actual total reward we got after taking action $a_{t}a\_t$.

• $V(s_t)V(s\_t)$ is the average reward we usually get from this state. It is predicted by a separate network called the Critic.

This significantly reduces noise because we are normalizing the signal against a baseline and only consider future rewards.

More specifically, PPO uses Generalized Advantage Estimation (GAE) formulation from Schulman et al. (2015), which uses smooth, exponentially-weighted advantage estimates from many steps. This helps to reduce variance in policy gradient updates and stabilizes training. You can read a nice explainer on GAE here.

Another important aspect of the PPO loss is that it avoids updates that are too large. When we update the weights, we don't want to make too large changes at once, which would cause "policy collapse". The original PPO paper introduced two ways to do that: PPO-CLIP and PPO-KLPEN.

PPO-CLIP

PPO-CLIP aims to stay within the trust region of the model by clipping the updates at each step. If our current policy is ${\pi }_{\theta }\backslash pi\_\backslash theta$ and ${\pi }_{old}\backslash pi\_\{old\}$ is the policy from the previous time stamp, we define the probability ratio $r_t(\theta )r\_t(\backslash theta)$ as:

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)}{{\pi }_{old}(a_t\mathrm{\mid }{s}_t)}r\_t(\backslash theta)\; =\; \backslash frac\{\backslash pi\_\backslash theta(a\_t|s\_t)\}\{\backslash pi\_\{old\}(a\_t|s\_t)\}$$

The ratio measures how different the new model's probability is compared to the old one.

We then restrict the updates to not be too large:

$$L^{PPOCLIP}(\theta )=\mathbb{E}[\min (\underset{\text{Unclipped}}{\underbrace{r_t(\theta ){\widehat{A}}_t}},\underset{\text{Clipped}}{\underbrace{\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\widehat{A}}_t}})]L^\{PPO\backslash \; CLIP\}(\backslash theta)\; =\; \backslash mathbb\{E\}\; \backslash left[\; \backslash min(\; \backslash underbrace\{r\_t(\backslash theta)\backslash hat\{A\}\_t\}\_\{\backslash text\{Unclipped\}\},\; \backslash underbrace\{\backslash text\{clip\}(r\_t(\backslash theta),\; 1-\backslash epsilon,\; 1+\backslash epsilon)\backslash hat\{A\}\_t\}\_\{\backslash text\{Clipped\}\}\; )\; \backslash right]$$

Where:

• ${\widehat{A}}_t\backslash hat\{A\}\_t$ is the Advantage

• $ϵ\backslash epsilon$ is a small hyperparameter, usually 0.2. It defines by how much we allow the model to update. For example, it can be 20% relative to the old model.

• $\text{clip}(\dots )\backslash text\{clip\}(\backslash dots)$ would then force the ratio to be between $0.80.8$ and $1.21.2$.

PPO-KLPEN

The PPO paper also defines an alternative definition to the clipped surrogate objective that adds an explicit KL divergence penalty to the objective with an adaptive penalty coefficient.

$$L^{PPOKL}(\theta )={\mathbb{E}}_t[r_t(\theta ){\widehat{A}}_t-\beta \cdot {\text{D}}_{KL}[{\pi }_{old}(\cdot \mathrm{\mid }{s}_t),{\pi }_{\theta }(\cdot \mathrm{\mid }{s}_t)]]L^\{PPO\backslash \; KL\}(\backslash theta)\; =\; \backslash mathbb\{E\}\_t\; \backslash left[\; r\_t(\backslash theta)\backslash hat\{A\}\_t\; -\; \backslash beta\; \backslash cdot\; \backslash text\{D\}\_\{KL\}[\backslash pi\_\{old\}(\backslash cdot|s\_t),\; \backslash pi\_\backslash theta(\backslash cdot|s\_t)]\; \backslash right]$$

where ${\text{D}}_{KL}[{\pi }_{old}(\cdot \mathrm{\mid }{s}_t),{\pi }_{\theta }(\cdot \mathrm{\mid }{s}_t)]\backslash text\{D\}\_\{KL\}[\backslash pi\_\{old\}(\backslash cdot|s\_t),\; \backslash pi\_\backslash theta(\backslash cdot|s\_t)]$ is a forward KL divergence penalty defined as follows (see the section below for a full explainer on this KL penalty):

$$D_{KL}[{\pi }_{old}(\cdot \mathrm{\mid }{s}_t)\mathrm{\mid }\mathrm{\mid }{\pi }_{\theta }(\cdot \mathrm{\mid }{s}_t)]=\sum _a{\pi }_{old}(a\mathrm{\mid }{s}_t)\log \left(\frac{{\pi }_{old}(a\mathrm{\mid }{s}_t)}{{\pi }_{\theta }(a\mathrm{\mid }{s}_t)}\right)D\_\{KL\}[\backslash pi\_\{old\}(\backslash cdot\; |\; s\_t)\; ||\; \backslash pi\_\{\backslash theta\}(\backslash cdot\; |\; s\_t)]\; =\; \backslash sum\_a\; \backslash pi\_\{old\}(a\; |\; s\_t)\; \backslash log\; \backslash left(\; \backslash frac\{\backslash pi\_\{old\}(a\; |\; s\_t)\}\{\backslash pi\_\{\backslash theta\}(a\; |\; s\_t)\}\; \backslash right)$$

The coefficient $\beta \backslash beta$ is updated during policy updates. While the initial value is a hyperparameter, its starting value doesn't seem to be important since the algorithm quickly adjusts it.

Compared to PPO-CLIP, this method imposes a softer constraint on the policy divergence. It can be more complex for implementation compared to clipping, and PPO-CLIP is more widely adopted in practice.

GRPO

PPO requires you to load several huge models into memory. The policy (the LLM we're training), the reference model (frozen pre-trained version of the model), a separate large Critic network, and often a reward model. This is really memory-intensive.

Instead, DeepSeek-R1 and V3 models skip the Critic network for calculating the baseline values. Instead of calculating the Advantage by subtracting the value $V(s_t)V(s\_t)$ predicted by the Critic, GRPO samples responses for a given prompt and uses the mean score of the group as the baseline.

That is, given a group of sample responses, $\{o_1,o_2,\mathrm{.}\mathrm{.}\mathrm{.},o_G\}\backslash \{o\_1,\; o\_2,\; ...,\; o\_G\backslash \}$ (usually $G=64)G=64)$ from the current model, we get their scores $\{R_1,R_2,\mathrm{.}\mathrm{.}\mathrm{.},R_G\}\backslash \{R\_1,\; R\_2,\; ...,\; R\_G\backslash \}$, and compute the advantage for output $ii$ as:

$$A_i=\frac{R_i-\text{mean}(\{R_1,R_2,\mathrm{.}\mathrm{.}\mathrm{.},R_G\})}{\text{std}(\{R_1,R_2,\mathrm{.}\mathrm{.}\mathrm{.},R_G\})}A\_i\; =\; \backslash frac\{R\_i\; -\; \backslash text\{mean\}(\backslash \{R\_1,\; R\_2,\; ...,\; R\_G\backslash \})\}\{\backslash text\{std\}(\backslash \{R\_1,\; R\_2,\; ...,\; R\_G\backslash \})\}$$

The final loss then becomes:

$${\mathcal{L}}_{GRPO}(\theta )=-\frac{1}{G}\sum _{i=1}^G(\min (r_i(\theta )A_i,\text{clip}(r_i(\theta ),1-ϵ,1+ϵ)A_i)-\beta \cdot D_{KL}[{\pi }_{\theta }\mathrm{\mid }\mathrm{\mid }{\pi }_{ref}{]}_i)\backslash mathcal\{L\}\_\{GRPO\}(\backslash theta)\; =\; -\; \backslash frac\{1\}\{G\}\; \backslash sum\_\{i=1\}^G\; \backslash left(\; \backslash min\; \backslash left(\; r\_i(\backslash theta)\; A\_i,\; \backslash text\{clip\}(r\_i(\backslash theta),\; 1-\backslash epsilon,\; 1+\backslash epsilon)\; A\_i\; \backslash right)\; -\; \backslash beta\; \backslash cdot\; D\_\{KL\}[\backslash pi\_\backslash theta\; ||\; \backslash pi\_\{ref\}]\_i\; \backslash right)$$

Now, notice the KL divergence term in this equation. It is different (!) from the penalty used in the PPO definition. This penalty is between the frozen reference model ${\pi }_{ref}\backslash pi\_\{ref\}$ and the current model ${\pi }_{\theta }\backslash pi\_\{\backslash theta\}$. This distinction deserves a separate section on its own, as it is very often a confusing point even in many popular open source implementations of RL algorithms.

KL Divergence Clarification

There are two KL penalties you'll see around: Trust-Region KL and Drift KL. Their purpose is very different, but they often get mixed up. They also use the opposite KL types - one is forward and the other reverse.

Quick refresher on KL divergence types

KL divergence is asymmetric: $D_{KL}(P\mathrm{\mid }\mathrm{\mid }Q)\mathrm{\ne }{D}_{KL}(Q\mathrm{\mid }\mathrm{\mid }P)D\_\{KL\}(P\; ||\; Q)\; \backslash neq\; D\_\{KL\}(Q\; ||\; P)$.

Forward KL is defined as

$$D_{KL}(P\mathrm{\mid }\mathrm{\mid }Q)=\sum _{x}P(x)\log \left(\frac{P(x)}{Q(x)}\right)D\_\{KL\}(P\; ||\; Q)\; =\; \backslash sum\_x\; P(x)\; \backslash log\; \backslash left(\; \backslash frac\{P(x)\}\{Q(x)\}\; \backslash right)$$

It is mean-seeking in the sense that when we minimize Forward KL with respect to Q, we force Q to cover as much of P as possible. It covers all modes of P.

Reverse KL is defined as:

$$D_{KL}(Q\mathrm{\mid }\mathrm{\mid }P)=\sum _{x}Q(x)\log \left(\frac{Q(x)}{P(x)}\right)D\_\{KL\}(Q\; ||\; P)\; =\; \backslash sum\_x\; Q(x)\; \backslash log\; \backslash left(\; \backslash frac\{Q(x)\}\{P(x)\}\; \backslash right)$$

Reverse KL is mode-seeking. It tries to capture one mode well and ignores other modes of P. It is fine with having some areas completely uncovered.

Now, in RL for LLMs, you will see two types of KL penalties.

Trust Region KL Penalty

The purpose of this penalty is to stabilize training and make sure that updates at each step are not too different from the weights at the immediate previous step.

$$D_{KL}({\pi }_{{\theta }_{old}}\mathrm{\mid }\mathrm{\mid }{\pi }_{\theta })={\mathbb{E}}_{\mathbf{x}\sim {\pi }_{{\theta }_{old}}}[\log \frac{{\pi }_{{\theta }_{old}}(\mathbf{x})}{{\pi }_{\theta }(\mathbf{x})}]D\_\{KL\}(\backslash pi\_\{\backslash theta\_\{old\}\}\; ||\; \backslash pi\_\{\backslash theta\})\; =\; \backslash mathbb\{E\}\_\{\backslash mathbf\{x\}\; \backslash sim\; \{\backslash pi\_\{\backslash theta\_\{old\}\}\}\}\; \backslash left[\; \backslash log\; \backslash frac\{\backslash pi\_\{\backslash theta\_\{old\}\}(\backslash mathbf\{x\})\}\{\backslash pi\_\{\backslash theta\}(\backslash mathbf\{x\})\}\; \backslash right]$$

This penalty is defined as Forward $KL(P\mathrm{\mid }\mathrm{\mid }Q)KL(P||Q)$ where $PP$ is fixed, and we optimize the second argument ${\pi }_{\theta }\backslash pi\_\{\backslash theta\}$. Because Forward KL is mean-seeking, we make sure that the new policy does not assign low probability to things that the old policy thought were good. We essentially want to cover all valid regions of the old policy.

The name Trust Region comes from an earlier paper from Schulman et al., 2015 that introduced Trust Region Policy Optimization (TRPO).

This penalty is subtracted directly from the loss in the PPO-KLPEN definition.

Drift KL Penalty

Drift KL penalty is introduced as a regularizer that prevents reward hacking. It ensures that the model ${\pi }_{\theta }\backslash pi\_\backslash theta$ doesn't drift away too far from the original SFT-pretrained reference model ${\pi }_{ref}\backslash pi\_\{ref\}$.

The Reward Model is not perfect, and optimizing without any constraints leads to the model finding ways to hack it. In the extreme, the model can start to output gibberish that gets very high scores. To avoid that, we use the Drift KL penalty.

This penalty is Reverse KL:

$$D_{KL}({\pi }_{\theta }\mathrm{\mid }\mathrm{\mid }{\pi }_{\text{ref}})={\mathbb{E}}_{\mathbf{x}\sim {\pi }_{\theta }}[\log \frac{{\pi }_{\theta }(\mathbf{x})}{{\pi }_{\text{ref}}(\mathbf{x})}]D\_\{KL\}(\backslash pi\_\{\backslash theta\}\; ||\; \backslash pi\_\{\backslash text\{ref\}\})\; =\; \backslash mathbb\{E\}\_\{\backslash mathbf\{x\}\; \backslash sim\; \{\backslash pi\_\{\backslash theta\}\}\}\; \backslash left[\; \backslash log\; \backslash frac\{\backslash pi\_\{\backslash theta\}(\backslash mathbf\{x\})\}\{\backslash pi\_\{\backslash text\{ref\}\}(\backslash mathbf\{x\})\}\; \backslash right]$$

Why reverse KL? You can think of it as forcing the model to only respond with outputs that have good support in the original reference model (i.e keep using natural language and not gibberish). If ${\pi }_{\theta }\backslash pi\_\backslash theta$ generates something where ${\pi }_{\text{ref}}\approx 0\backslash pi\_\{\backslash text\{ref\}\}\; \backslash approx\; 0$, this will lead to a massive penalty.

At the same time, we don't mind the model forgetting other modes of the original model like toxic responses from the internet.

Typically, this penalty is subtracted directly from the reward function.

$$R_{total}(s,a)=R_{model}(s,a)-\beta \log \left(\frac{{\pi }_{\theta }(a\mathrm{\mid }s)}{{\pi }_{ref}(a\mathrm{\mid }s)}\right)R\_\{total\}(s,\; a)\; =\; R\_\{model\}(s,\; a)\; -\; \backslash beta\; \backslash log\; \backslash left(\; \backslash frac\{\backslash pi\_\backslash theta(a|s)\}\{\backslash pi\_\{ref\}(a|s)\}\; \backslash right)$$

This is how Standard RLHF or PPO-CLIP and PPO-KLPEN algorithms do it. Note that this Drift KL penalty is usually the one people talk about and is much more often used.

Now, going back to the GRPO loss, you might notice that the KL term is inside the loss function. GRPO is an outlier here, and its formulation of the loss with group averages puts the KL regularization term directly in the policy update rule.

KL Estimators and Gradient Pitfall

In practice, we don't compute the KL penalties directly as that is infeasible. Instead, we use sample-based KL estimators. This estimation is a very important detail which has recently been scrutinized as different estimators lead to different issues.

Recent work, A Comedy of Estimators (Shah et al., 2026) and On a few pitfalls in KL divergence gradient estimation (Tang & Munos, 2025), has shown how the most popular methods, including those used in popular open-sourced libraries, often calculate the wrong gradients.

But first, let's look at 3 popular ways to estimate KL divergence:

1. K1 Estimator (Naive) is a Monte Carlo estimator derived straight from the definition. This estimator is unbiased and most honest but in practice has quite high variance.

$${\mathbb{K}\mathbb{L}}_1=\sum _{t=1}^T\log \frac{\pi }{{\pi }_{\text{ref}}}\backslash mathbb\{KL\}\_1\; =\; \backslash sum\_\{t=1\}^T\; \backslash log\; \backslash frac\{\backslash pi\}\{\backslash pi\_\{\backslash text\{ref\}\}\}$$

2. K2 Estimator is based on Taylor expansion of the KL divergence. Squaring the log ratio makes sure that penalty is always non-negative. Note that this makes it symmetric whereas KL is assymetric.

$${\mathbb{K}\mathbb{L}}_2=\sum _{t=1}^T\frac{1}{2}{(\log \frac{\pi }{{\pi }_{\text{ref}}})}^2\backslash mathbb\{KL\}\_2\; =\; \backslash sum\_\{t=1\}^T\; \backslash frac\{1\}\{2\}\; \backslash left(\; \backslash log\; \backslash frac\{\backslash pi\}\{\backslash pi\_\{\backslash text\{ref\}\}\}\; \backslash right)^2$$

3. K3 Estimator was used in the PPO paper and is often the default estimator in libraries like TRL. Small deviations have almost no penalty. It is an unbiased estimator.

$${\mathbb{K}\mathbb{L}}_3=\sum _{t=1}^T(\frac{\pi }{{\pi }_{\text{ref}}}-1-\log \frac{\pi }{{\pi }_{\text{ref}}})\backslash mathbb\{KL\}\_3\; =\; \backslash sum\_\{t=1\}^T\; \backslash left(\; \backslash frac\{\backslash pi\}\{\backslash pi\_\{\backslash text\{ref\}\}\}\; -\; 1\; -\; \backslash log\; \backslash frac\{\backslash pi\}\{\backslash pi\_\{\backslash text\{ref\}\}\}\; \backslash right)$$

Now, as mentioned above, the Drift KL penalty can be either added to the reward (standard RLHF definition) or to the loss (GRPO and the likes).

• In the Reward: you calculate the KL value, then you stop the gradients, and then you subtract it from the reward

• In the Loss: when the KL penalty is in the loss function, it is passed to the differentiator and included in the gradients.

The pitfall is that when you differentiate a KL estimate in the loss, it actually does not equal to the gradients of the KL divergence term. So even if an estimator is unbiased, its gradients can be biased and wrong.

Shah et al., 2026 shows that the different configurations of the estimator type and whether it is included in the reward or the loss leads to dramatically different behavior and performance:

Off-Policy Learning

Now let's move on to the off-policy methods, which actually do not have an explicit KL divergence term but model this implicitly.

On-policy methods can be computationally expensive. The bottleneck is the generation step. We also need to have multiple large models loaded at the same time (Actor, Critic, Reward Model, Reference Model). Off-policy methods learn from existing data and omit the need for a separate Reward Model.

Direct Preference Optimization

DPO (Rafailov et al., 2023) doesn't use a separate Reward Model neural network. Instead, it extracts the optimal reward directly from the preference data. DPO has been used in Llama 3 (combined PPO with DPO) and in Qwen-Chat models. Let's derive DPO's objective.

Step 1: Define the RLHF objective

First, let's revisit the standard RLHF objective with the KL term added to the reward calculation:

$$\text{Reward}=r(x,y)-\beta \log \frac{{\pi }_{\theta }(y\mathrm{\mid }x)}{{\pi }_{ref}(y\mathrm{\mid }x)}\backslash text\{Reward\}\; =\; r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y|x)\}\{\backslash pi\_\{ref\}(y|x)\}$$

Our goal is to find a policy $\pi \backslash pi$ that maximizes the expected reward while staying close to a reference model ${\pi }_{ref}\backslash pi\_\{ref\}$:

$$\underset{\pi }{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim \pi }[r(x,y)-\beta \log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{ref}(y\mathrm{\mid }x)}]\backslash max\_\{\backslash pi\}\; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi\}\; \backslash left[\; r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{ref\}(y|x)\}\; \backslash right]$$

Recap:

• $xx$ is the input (prompt) sampled from the dataset $\mathcal{D}\backslash mathcal\{D\}$

• $yy$ is the output generated by the current LLM (the policy $\pi \backslash pi$)

• $r(x,y)r(x,y)$ is the reward function

• $\beta \backslash beta$ is a parameter controlling the KL-divergence penalty

Step 2: Rewrite in the closed-form solution

To arrive at a closed-form solution, we can focus on the optimal policy ${\pi }^{\ast }\backslash pi^*$ for the objective above. This is:

${\pi }^{\ast }(y\mathrm{\mid }x)=\frac{1}{Z(x)}{\pi }_{ref}(y\mathrm{\mid }x)\exp (\frac{1}{\beta }r(x,y))\backslash pi^*(y|x)\; =\; \backslash frac\{1\}\{Z(x)\}\; \backslash pi\_\{ref\}(y|x)\; \backslash exp\backslash left(\; \backslash frac\{1\}\{\backslash beta\}\; r(x,y)\; \backslash right)$

The perfect policy is essentially the reference policy ${\pi }_{ref}\backslash pi\_\{ref\}$ scaled by the reward $\exp (\frac{1}{\beta }r(x,y))\backslash exp\backslash left(\; \backslash frac\{1\}\{\backslash beta\}\; r(x,y)\; \backslash right)$.

$Z(x)Z(x)$ is a partition function (normalization constant) to make sure probabilities sum to 1. We can't compute this term.

Step 3: Invert the equation

Normally, we'd train a Reward Model to get an estimate of the reward $r(x,y)r(x,\; y)$ and then use something like PPO to find the optimal policy ${\pi }^{\ast }\backslash pi^*$.

Instead, here we can solve the equation for the reward. This is the main DPO's trick:

$r(x,y)=\beta \log \frac{{\pi }^{\ast }(y\mathrm{\mid }x)}{{\pi }_{ref}(y\mathrm{\mid }x)}+\beta \log Z(x)r(x,y)\; =\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y|x)\}\{\backslash pi\_\{ref\}(y|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)$

Step 4: Cancel out terms

In RLHF, we want to maximize the difference between good responses (winners) and bad responses (losers): $r(x,y_w)-r(x,y_l)r(x,\; y\_w)\; -\; r(x,\; y\_l)$.

which we can rewrite using the equation above as

$$r(x,y_w)-r(x,y_l)=(\beta \log \frac{{\pi }_{\theta }(y_w\mathrm{\mid }x)}{{\pi }_{ref}(y_w\mathrm{\mid }x)}+\beta \log Z(x))-(\beta \log \frac{{\pi }_{\theta }(y_l\mathrm{\mid }x)}{{\pi }_{ref}(y_l\mathrm{\mid }x)}+\beta \log Z(x))r(x,\; y\_w)\; -\; r(x,\; y\_l)\; =\; \backslash left(\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_w|x)\}\{\backslash pi\_\{ref\}(y\_w|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)\; \backslash right)\; -\; \backslash left(\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_l|x)\}\{\backslash pi\_\{ref\}(y\_l|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)\; \backslash right)$$

The advantage of doing so is that the partition function now cancels out in both of these expressions.

This gives us the final DPO loss:

$${\mathcal{L}}_{DPO}=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w\mathrm{\mid }x)}{{\pi }_{ref}(y_w\mathrm{\mid }x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mathrm{\mid }x)}{{\pi }_{ref}(y_l\mathrm{\mid }x)})]\backslash mathcal\{L\}\_\{DPO\}\; =\; -\; \backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\; \backslash left[\; \backslash log\; \backslash sigma\; \backslash left(\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_w|x)\}\{\backslash pi\_\{ref\}(y\_w|x)\}\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_l|x)\}\{\backslash pi\_\{ref\}(y\_l|x)\}\; \backslash right)\; \backslash right]$$

The intuition behind this loss is that we want to increase the likelihood ratio of the winning responses and decrease the ratio of the losing responses, all the while staying close to the reference model.

DPO is memory-efficient and stable. However, it is quite sensitive to the distribution of the dataset and can easily overfit if the data is noisy.

Notice that even though the explicit KL term is gone, we still have the hyperparameter $\beta \backslash beta$ which is now used to scale the log-probability ratios inside the sigmoid function. And in principle, it serves the same role as in PPO - higher values of $\beta \backslash beta$ force the model to stay closer to the reference model and increase the implicit penalty.

Iterative and Online DPO

Offline DPO is typically trained on a static preference dataset.

However, as the model's training progresses and the policy ${\pi }_{\theta }\backslash pi\_\{\backslash theta\}$ changes, the model's output distribution starts to drift away from the provided data. The training would benefit from new preference relations that better correspond to its current abilities.

Iterative DPO aims to solve this shifting data distribution problem. It refines the model in multiple stages of data generation.

In iterative DPO, we do the following:

1. Generate new preference data $(x,y_1,y_2)(x,\; y\_1,\; y\_2)$ with the current model ${\pi }_{{\theta }_t}\backslash pi\_\{\backslash theta\_t\}$

2. Use the Reward model to label which response is better and assemble a new dataset $D_{t}D\_t$

3. Run a DPO step on $D_{t}D\_t$ and update model to ${\pi }_{{\theta }_{t+1}}\backslash pi\_\{\backslash theta\_\{t+1\}\}$

Identity Preference Optimization (IPO)

The sigmoid loss that DPO uses tries to push the probability gap between winning and losing responses to infinity. But sometimes, the losing response is not terrible, just not as good. One algorithm called IPO, replaces the sigmoid with a mean-squared error objective.

$${\mathcal{L}}_{IPO}={\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[{(\log \frac{{\pi }_{\theta }(y_w\mathrm{\mid }x)}{{\pi }_{ref}(y_w\mathrm{\mid }x)}-\log \frac{{\pi }_{\theta }(y_l\mathrm{\mid }x)}{{\pi }_{ref}(y_l\mathrm{\mid }x)}-\frac{1}{2\beta })}^2\right]\backslash mathcal\{L\}\_\{IPO\}\; =\; \backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\; \backslash left[\; \backslash left(\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_w|x)\}\{\backslash pi\_\{ref\}(y\_w|x)\}\; -\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_l|x)\}\{\backslash pi\_\{ref\}(y\_l|x)\}\; -\; \backslash frac\{1\}\{2\backslash beta\}\; \backslash right)^2\; \backslash right]$$

Instead of maximizing the gap to infinity, you can select some margin $\frac{1}{2\beta }\backslash frac\{1\}\{2\backslash beta\}$. Once the model reaches the gap, it stops updating. This essentially can act like a regularizer which prevents the model from overfitting to noisy data.

Kahneman-Tversky Optimization (KTO)

One problem of RLFH and DPO is that they require paired data of winners and losers $(y_w,y_l)(y\_w,\; y\_l)$. Data collected in real-world scenarios are often not paired unless an A/B test was ran. We just have good responses or bad responses that are not paired.

KTO, introduced in Ethayarajh et al., 2025, defines a new loss inspired by the Prospect Theory. Prospect Theory was introduced by Daniel Kahneman and Amor Tversky and describes how humans perceive losses and wins in an asymmetric manner, caring more about losses than wins. The KTO paper tries to use this intuition and define a new loss that maps our human psychological model into it without the need of explicit pairs.

The KTO loss maximizes the utility of good responses and minimizes the utility of bad responses independently:

$$L_{KTO}(\theta )={\mathbb{E}}_{x,y\sim D}[\underset{\begin{array}{c}\text{Class Weight}\\ ({\lambda }_D\text{ or }{\lambda }_U)\end{array}}{\underbrace{{\lambda }_y}}-\underset{\begin{array}{c}\text{Value of}\\ \text{Generation}\end{array}}{\underbrace{v(x,y)}}]L\_\{KTO\}(\backslash theta)\; =\; \backslash mathbb\{E\}\_\{x,y\; \backslash sim\; D\}\; \backslash left[\; \backslash underbrace\{\backslash lambda\_y\}\_\{\backslash substack\{\backslash text\{Class\; Weight\}\; \backslash \backslash \; (\backslash lambda\_D\; \backslash text\{\; or\; \}\; \backslash lambda\_U)\}\}\; -\; \backslash underbrace\{v(x,y)\}\_\{\backslash substack\{\backslash text\{Value\; of\}\; \backslash \backslash \; \backslash text\{Generation\}\}\}\; \backslash right]$$

where the value function $v(x,y)v(x,y)$ is defined as:

We want the reward $r_{\theta }r\_\backslash theta$ to be higher than $z_{0}z\_0$ if it's desirable and lower if undesirable relative to a reference point $z_{0}z\_0$. ${\lambda }_D\lambda \_D$ and ${\lambda }_U\lambda \_U$ are hyperparameters for desirable and undesirable outputs.

The reference point is defined in the paper as:

$$z_0=\underset{\text{Average Drift from Ref Model}}{\underbrace{\text{KL}({\pi }_{\theta }(y^{\mathrm{\prime }}\mathrm{\mid }x)\mathrm{\mid }\mathrm{\mid }{\pi }_{ref}(y^{\mathrm{\prime }}\mathrm{\mid }x))}}z\_0\; =\; \backslash underbrace\{\backslash text\{KL\}(\backslash pi\_\backslash theta(y\text{'}|x)\; ||\; \backslash pi\_\{ref\}(y\text{'}|x))\}\_\{\backslash text\{Average\; Drift\; from\; Ref\; Model\}\}$$

Note that the authors used an estimator, as is standard for the KL penalty calculation, or even set the reference point to zero when KTO is preceded by SFT on the same subset of data.

Simple Preference Optimization (SimPO)

All of the algorithms above require us to load a Reference Model ${\pi }_{ref}\backslash pi\_\{ref\}$ into memory so that we can calculate the KL-divergence ratio. But these models are huge and it doubles the GPU memory requirement.

SimPO argues we don't need the reference model. Instead it aligns the model using length-normalized log probability of the tokens (with some margin target $\gamma \backslash gamma$.

$${\mathcal{L}}_{SimPO}=-\mathbb{E}[\log \sigma (\frac{1}{\mathrm{\mid }{y}_w\mathrm{\mid }}\log {\pi }_{\theta }(y_w\mathrm{\mid }x)-\frac{1}{\mathrm{\mid }{y}_l\mathrm{\mid }}\log {\pi }_{\theta }(y_l\mathrm{\mid }x)-\gamma )]\backslash mathcal\{L\}\_\{SimPO\}\; =\; -\; \backslash mathbb\{E\}\; \backslash left[\; \backslash log\; \backslash sigma\; \backslash left(\; \backslash frac\{1\}\{|y\_w|\}\; \backslash log\; \backslash pi\_\backslash theta(y\_w|x)\; -\; \backslash frac\{1\}\{|y\_l|\}\; \backslash log\; \backslash pi\_\backslash theta(y\_l|x)\; -\; \backslash gamma\; \backslash right)\; \backslash right]$$

This can make it more memory-friendly and faster.

Summary

This post walks you through some of the basics of reinforcement learning for LLM. The field is moving very fast, so stay tuned for more updates and more specialized posts. Specifically, things not covered here include verifiable rewards, process vs outcome rewards, and search and inference-time compute.

Special thanks to Sabrina Mielke for her helpful feedback and review of this post.

By Karina Zadorozhny