Alignment & Post-Training

A base language model trained purely on next-token cross-entropy loss ( $L_{CE} = - \sum_{t} lo g P_{θ} (x_{t} ∣ x_{< t})$ ) is a statistical distribution simulator, not an aligned assistant. Given a prompt like “How do I fix a memory leak in C?”, a pre-trained base model is just as likely to continue with five more unanswered forum questions as it is to output a solution.

Post-training and alignment reshape the base distribution $π_{base}$ into a controllable policy $π_{θ}$ through three distinct mathematical stages:

Supervised Fine-Tuning (SFT): Behavioral cloning on curated (prompt, response) demonstrations to establish formatting and instruction following, producing the reference anchor $π_{ref}$ .
Preference Optimization (RLHF / DPO): Shifting probability mass away from plausible-sounding errors ( $y_{l}$ ) and toward preferred completions ( $y_{w}$ ) under a KL-divergence trust region.
Reinforcement Learning with Verifiable Rewards (RLVR / GRPO): Training directly against deterministic math/code verifiers to induce multi-step chain-of-thought self-correction.

From Bradley-Terry Rewards to Closed-Form DPO

1. The Bradley-Terry Preference Model

Given a prompt $x$ and two candidate completions $(y_{w}, y_{l})$ where human or verifier evaluation prefers $y_{w} ≻ y_{l}$ , the Bradley-Terry model defines the probability of this pairwise preference via a latent scalar reward function $r_{ϕ} (x, y)$ :

P (y_{w} ≻ y_{l} ∣ x) = σ (r_{ϕ} (x, y_{w}) - r_{ϕ} (x, y_{l})) = \frac{1}{1 + exp ( - ( r _{ϕ} ( x , y _{w} ) - r _{ϕ} ( x , y _{l} )) )}

2. The KL-Regularized RLHF Objective

In classical Reinforcement Learning from Human Feedback (Ouyang et al. 2022), we train a policy $π_{θ}$ to maximize expected reward while penalizing reverse KL-divergence from the frozen reference policy $π_{ref}$ :

π_{θ} max E_{x \sim D, y \sim π_{θ} (y ∣ x)} [r_{ϕ} (x, y)] - β D_{KL} (π_{θ} (y ∣ x) ∥ π_{ref} (y ∣ x))

3. Direct Preference Optimization (DPO) Derivation

Instead of optimizing this objective with unstable online Proximal Policy Optimization (PPO), Rafailov et al. (2023) observed that the KL-constrained maximization has an exact analytical solution $π^{*} (y ∣ x)$ :

π^{*} (y ∣ x) = \frac{1}{Z ( x )} π_{ref} (y ∣ x) exp (\frac{1}{β} r (x, y))

where $Z (x) = \sum_{y} π_{ref} (y ∣ x) exp (r (x, y) / β)$ is the partition function. Solving for the reward $r (x, y)$ algebraically yields:

r (x, y) = β lo g \frac{π ^{*} ( y ∣ x )}{π _{ref} ( y ∣ x )} + β lo g Z (x)

When we substitute this expression into the Bradley-Terry preference formula $r (x, y_{w}) - r (x, y_{l})$ , the intractable partition function $β lo g Z (x)$ cancels out identically, giving the closed-form DPO loss:

L_{DPO} (π_{θ}; π_{ref}) = - E_{(x, y_{w}, y_{l})} [lo g σ (β lo g \frac{π _{θ} ( y _{w} ∣ x )}{π _{ref} ( y _{w} ∣ x )} - β lo g \frac{π _{θ} ( y _{l} ∣ x )}{π _{ref} ( y _{l} ∣ x )})]

Symbol / Objective	Mathematical Space	Geometric & Optimization Role
$π_{θ} (y ∣ x), π_{ref} (y ∣ x)$	$Δ (V^{T})$	Active Policy & Reference Anchor: Autoregressive distribution of the aligned model $π_{θ}$ anchored against the frozen Supervised Fine-Tuned (SFT) policy $π_{ref}$ .
$P (y_{w} ≻ y_{l} ∣ x) = σ (r (x, y_{w}) - r (x, y_{l}))$	$[0, 1]$	Bradley-Terry Preference Model: Maps the scalar reward difference between winning response $y_{w}$ and losing response $y_{l}$ through the logistic sigmoid $σ (z) = (1 + e^{- z})^{- 1}$ .
$D_{KL} (π_{θ} ∥ π_{ref})$	$R_{\geq 0}$	Reverse KL-Divergence Penalty: Regularizer $E_{y \sim π_{θ}} [lo g (π_{θ} (y ∣ x) / π_{ref} (y ∣ x))]$ weighted by $β$ to prevent policy collapse and reward over-optimization.
$r (x, y) = β lo g \frac{π ^{*} ( y ∣ x )}{π _{ref} ( y ∣ x )} + β lo g Z (x)$	Implicit Reward	Closed-Form Implicit Reward (Rafailov et al. 2023): Expresses the ground-truth reward function directly in terms of the optimal policy log-ratio, eliminating the separate reward model and PPO critic.
$\hat{A}_{i} = \frac{r _{i} - mean ( r )}{std ( r )}$	$R^{G}$	Group Relative Advantage (GRPO, DeepSeek-R1): Computes token advantages from a group of $G$ sampled completions per prompt, cutting value-network VRAM by 50%.

Post-Training Architecture Topology: 4-Model RLHF (PPO) vs. 2-Model DPO

Figure 1: Online PPO requires holding four multi-billion-parameter models in GPU memory simultaneously. By substituting the analytical solution

π^{*} (y ∣ x)

into the Bradley-Terry loss, DPO optimizes the policy directly via binary cross-entropy over log-probability ratios.

Numerical Walkthrough: DPO Margin & Gradient Weighting

To see how DPO updates token probabilities without a reward model, consider a single preference pair $(x, y_{w}, y_{l})$ with KL temperature $β = 0.20$ :

Sequence Log-Probabilities under $π_{θ}$ and $π_{ref}$ :
- Winning response $y_{w}$ : $lo g π_{θ} (y_{w} ∣ x) = - 12.0$ , $lo g π_{ref} (y_{w} ∣ x) = - 14.0$
- Losing response $y_{l}$ : $lo g π_{θ} (y_{l} ∣ x) = - 11.0$ , $lo g π_{ref} (y_{l} ∣ x) = - 10.5$
Log-Ratio Rewards ( $\overset{r}{^}_{θ} (x, y) = β lo g \frac{π _{θ} ( y ∣ x )}{π _{ref} ( y ∣ x )}$ ):
- $\overset{r}{^}_{θ} (x, y_{w}) = 0.20 \times (- 12.0 - (- 14.0)) = 0.20 \times (+ 2.0) = + 0.40$
- $\overset{r}{^}_{θ} (x, y_{l}) = 0.20 \times (- 11.0 - (- 10.5)) = 0.20 \times (- 0.5) = - 0.10$
Implicit Preference Margin & Loss:
- Margin $Δ \overset{r}{^}_{θ} = 0.40 - (- 0.10) = + 0.50$
- Preference probability: $σ (0.50) = \frac{1}{1 + e ^{- 0.50}} \approx 0.622$
- DPO Loss: $L_{DPO} = - lo g (0.622) \approx 0.474 nats$
Per-Example Gradient Weight: Taking the gradient with respect to $θ$ :
$\nabla_{θ} L_{DPO} = - β 1 - 0.622 = 0.378 σ (\overset{r}{^}_{θ} (x, y_{l}) - \overset{r}{^}_{θ} (x, y_{w})) [\nabla_{θ} lo g π_{θ} (y_{w} ∣ x) - \nabla_{θ} lo g π_{θ} (y_{l} ∣ x)]$
Notice that the factor $σ (- Δ \overset{r}{^}_{θ}) = 0.378$ automatically scales the gradient in proportion to how strongly the current policy misranks the pair.

Interactive Reward Hacking & Goodhart Simulator

In online RL, the reward model $R_{proxy}$ is only an imperfect statistical proxy for true human preference $R_{gold}$ . Gao et al. (2023) established that as policy drift $d = D_{KL} (π_{θ} ∥ π_{ref})$ grows, true utility follows an inverted-U curve:

R_{proxy} (d) = α d, R_{gold} (d) = α d - γ d^{2}

Use the simulator below to observe how pushing $D_{KL}$ past the optimal frontier causes the policy to game length and sycophancy biases:

KL-Regularized Alignment & Goodhart Reward Hacking Simulator

Adjust the KL-divergence penalty $β$ and policy drift $D_{KL} (π_{θ} ∥ π_{ref})$ to observe how over-optimizing an imperfect proxy reward $R_{proxy}$ causes true human utility $R_{gold}$ to collapse.

KL Penalty β: 0.10

Drift D_KL: 6.25 nats

Proxy Reward Model Score

R_{proxy}

+2.10

Trained classifier score (

α D_{KL}

). Continues rising even when the policy exploits spurious formatting correlations.

True Human Utility

R_{gold}

+1.48 (Optimal Peak)

Actual response quality (

α D_{KL} - γ D_{KL}

). Collapses when

D_{KL}

drifts too far from

π_{ref}

DPO Implicit Gradient Weight

σ (- β Δ r)

0.438

Policy Output Behavior: Concise, factual step-by-step derivation without sycophancy.

Sweet Spot (Aligned)

Policy Drift D_KL:

Group Relative Policy Optimization (GRPO)

For mathematical and code reasoning (DeepSeek-R1), maintaining a separate Value Critic $V_{ψ}$ of the same size as the policy doubles GPU memory consumption. Group Relative Policy Optimization (GRPO) (Shao et al. 2024) eliminates the critic entirely by sampling a group of $G$ outputs ${o_{1}, o_{2}, \dots, o_{G}}$ from $π_{θ_{old}}$ for each prompt $q$ and computing the baseline directly from the group’s reward statistics:

\hat{A}_{i, t} = \frac{r _{i} - mean ( r _{1} , \dots , r _{G} )}{std ( r _{1} , \dots , r _{G} )}

When paired with deterministic rule-based verifiers ( $r_{i} \in {0, 1}$ for exact math answer or passing unit tests), GRPO avoids neural reward model hacking while enabling emergent self-verification and backtracking.

Comparing Post-Training Algorithms

Algorithm	Models in VRAM	Rollout Mode	Advantage / Margin Mechanism	Primary Failure Mode
RLHF (PPO)	4 ( $π_{θ}, π_{ref}, r_{ϕ}, V_{ψ}$ )	Online sampling	Generalized Advantage Estimation (GAE) via $V_{ψ}$	Critic instability & high VRAM footprint
DPO	2 ( $π_{θ}, π_{ref}$ )	Offline pairs	Closed-form log-ratio margin $β Δ lo g (π_{θ} / π_{ref})$	Likelihood displacement on out-of-distribution tokens
SimPO	1 ( $π_{θ}$ only)	Offline pairs	Length-normalized average log-prob $\frac{\beta}{	y
GRPO	2 ( $π_{θ}, π_{ref}$ )	Online group ( $G$ )	Group z-score normalization $(r_{i} - μ_{G}) / σ_{G}$	Zero gradient when all $G$ samples fail ( $r_{i} = 0$ )

Key Papers & References

Training language models to follow instructions with human feedback (InstructGPT)

Ouyang et al. (NeurIPS 2022) • Established the three-stage SFT, Reward Model, and KL-PPO alignment pipeline.

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafailov et al. (NeurIPS 2023) • Derived the closed-form Bradley-Terry policy mapping, eliminating the PPO critic and reward model.

Scaling Laws for Reward Model Overoptimization

Gao et al. (ICML 2023) • Quantified Goodhart’s Law in RLHF as a function of square-root KL divergence.

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI (2025) • Demonstrated emergent reasoning and self-verification via large-scale GRPO on verifiable rewards.

Data Curation & Scaling Laws Inference & Training Efficiency