Inference & Training Efficiency

Modern GPU accelerators are asymmetric machines: their matrix compute throughput (FLOP/s) has scaled $10 \times$ faster than their off-chip High-Bandwidth Memory (HBM) speed. An NVIDIA H100 SXM GPU delivers $989 TFLOP/s$ of FP16 tensor compute, but only $3.35 TB/s$ of HBM3 memory bandwidth.

To saturate the GPU’s matrix units without stalling on memory transfers, a kernel must perform at least 295 floating-point operations for every byte fetched from HBM (the hardware Ridge Point):

I_{ridge} = \frac{Peak Compute (FLOP/s)}{Peak Memory Bandwidth (Bytes/s)} = \frac{989 \times 1 0 ^{12}}{3.35 \times 1 0 ^{12}} \approx 295.2 FLOPs/Byte

This single ratio explains why LLM inference is divided into two physically opposite regimes:

Prefill (Compute-Bound): Processing an input prompt of $T = 2, 048$ tokens in parallel performs $2 NT$ FLOPs while reading the $2 N$ byte weight matrix once, yielding $I_{prefill} \approx T = 2, 048 ≫ 295 FLOPs/Byte$ .
Autoregressive Decode (Memory-Bandwidth Bound): Generating a single token ( $T = 1$ ) for batch size $B = 1$ performs $2 N$ FLOPs while streaming all $2 N$ bytes of model weights from HBM, yielding $I_{decode} \approx 1 FLOP/Byte$ —leaving 99.6% of the GPU’s compute cores idle.

Symbol / Metric	Units / Domain	Hardware & Algorithmic Role
$I = \frac{FLOPs}{Bytes Transferred}$	$FLOPs / Byte$	Arithmetic Intensity: Ratio of floating-point operations to HBM memory traffic. On an NVIDIA H100 SXM ( $989 TFLOP/s$ FP16, $3.35 TB/s$ HBM3), the ridge point is $I_{ridge} \approx 295 FLOPs/Byte$ .
$I_{decode} (B) \approx B$	$B ≪ 295$	Autoregressive Decode Intensity: Because generating 1 token per request streams all $2 N$ weight bytes from HBM, small-batch decode achieves only $1 - 16 FLOPs/Byte$ (<5% GPU compute utilization).
$W_{q} = clamp (⌊ W /Δ ⌉ + z)$	$INT4 / FP8$	Affine Group Quantization (GPTQ / AWQ): Compresses 16-bit weights into $b = 4$ -bit integers per group of 128 channels using scale $Δ$ and zero-point $z$ , cutting HBM weight traffic by $4 \times$ .
$α (x) = min (1, \frac{p _{target} ( x )}{q _{draft} ( x )})$	$p (x) = p_{target} (x)$	Speculative Decoding Acceptance Rule: Accepts K drafted tokens from a small draft model $q$ in a single parallel target forward pass while guaranteeing the exact target distribution $p_{target} (x)$ .
$M_{ZeRO - 3} = \frac{16Ψ}{N _{GPU}}$	$Bytes / GPU$	ZeRO-3 / FSDP Sharded Training State: Partitions FP16 parameters ( $2Ψ$ ), gradients ( $2Ψ$ ), and FP32 Adam states ( $12Ψ$ ) across $N_{GPU}$ ranks.

Speculative Decoding Parallel Verification (K = 4 Draft Tokens Verified in 1 Target Pass)

Figure 1: Verifying

K = 4

draft tokens in parallel costs the same HBM weight read as generating 1 token. Even when Draft 4 is rejected, the target model's logits at position 4 immediately yield a corrected token ("1") from

norm (max (0, p - q))

, producing 4 valid tokens in a single 70B pass.

Weight Quantization Walkthrough (AWQ & GPTQ)

Because small-batch decoding is bottlenecked strictly by streaming weight bytes from HBM, compressing each parameter from 16 bits (FP16/BF16) to 4 bits (INT4) reduces memory traffic by nearly $4 \times$ and directly multiplies token generation speed.

1. Group-Wise Affine Quantization

Given a weight block $w \in R^{128}$ , 4-bit asymmetric quantization maps floating-point weights to integers ${0, 1, \dots, 15}$ :

Δ = \frac{max ( w ) - min ( w )}{2 ^{b} - 1}, z = round (\frac{- min ( w )}{Δ}), w_{q} = clamp (round (\frac{w}{Δ}) + z, 0, 15)

Consider a 4-element weight slice $w = [- 0.60, - 0.15, + 0.30, + 1.20]$ quantized to $b = 4$ bits ( $2^{4} - 1 = 15$ intervals):

Scale Factor $Δ$ : $\frac{1.20 - ( - 0.60 )}{15} = \frac{1.80}{15} = 0.12$
Zero-Point $z$ : $round (0.60/0.12) = 5$
Quantized Integers $w_{q}$ : $[0, 4, 8, 15]$
Dequantized Reconstruction $\hat{w} = Δ (w_{q} - z)$ : $[- 0.60, - 0.12, + 0.36, + 1.20]$

2. Activation-Aware Weight Quantization (AWQ)

In Transformer hidden states, $0.1% - 1%$ of channels exhibit massive activation outliers ( $100 \times$ larger than average). Quantizing the corresponding weight columns naively causes severe perplexity degradation.

AWQ (Lin et al. 2024) protects salient weight channels without mixed-precision hardware overhead by multiplying salient weight column $w_{j}$ by a per-channel scale $s_{j} > 1$ before quantization and dividing the input activation channel $x_{j}$ by $s_{j}$ :

y = (w_{j} \cdot s_{j}) (\frac{x _{j}}{s _{j}}) ⟹ Quant (w_{j} \cdot s_{j}) \cdot \frac{x _{j}}{s _{j}}

Speculative Decoding Math & Expected Speedup

How can we accelerate autoregressive decoding without modifying model weights? Speculative Decoding (Leviathan et al. 2023; Chen et al. 2023) uses a fast draft model $q (x_{t} ∣ x_{< t})$ to speculate $K$ tokens ahead, then uses a single parallel forward pass of the target model $p (x_{t} ∣ x_{< t})$ to verify all $K$ tokens simultaneously.

To guarantee that the output is mathematically identical to sampling from the target model $p (x)$ , each drafted token $x$ is accepted with probability:

α (x) = min (1, \frac{p ( x )}{q ( x )})

If token $x_{k}$ is rejected, we immediately sample a replacement token from the residual distribution without running another forward pass:

x_{k}^{'} \sim norm (max (0, p (x) - q (x)))

If the average acceptance rate across drafted tokens is $α \in [0, 1]$ , the expected number of tokens generated per target forward pass is:

E [Tokens / Pass] = \frac{1 - α ^{K + 1}}{1 - α}

For $K = 4$ and $α = 0.80$ , a single target verification pass emits $\frac{1 - 0.8 0 ^{5}}{1 - 0.80} = 3.36 tokens$ , delivering a $2.5 \times$ wall-clock speedup with zero accuracy loss.

Interactive Roofline & Quantization Simulator

Explore the H100 roofline model below for a 70B parameter model. Notice how weight quantization shifts the operating point along the memory-bandwidth slope at small batch sizes:

GPU Roofline & Quantization Throughput Simulator (LLaMA-3 70B on H100 SXM)

Adjust the concurrent batch size $B$ and weight precision (FP16 vs. INT8 vs. INT4 AWQ) to see where autoregressive decoding transitions from the HBM memory-bandwidth slope to the Tensor Core compute ceiling.

Batch B: 8

Precision: FP16 (16-bit)

Model Weight Footprint in HBM 140.0 GB

Every decode step streams the entire weight matrix across the

3.35 TB/s

HBM3 bus.

Arithmetic Intensity

I

& Compute Util 8.0 FLOPs/B (2.7% Util)

Memory-Bandwidth Bound: Tensor Cores sit idle 97.3% of the time waiting on HBM weight reads.

System Decode Throughput 191 tokens/sec

Per-user speed: 23.9 tok/s (

T_{step}

= 41.8 ms).

Memory-Bound Decode

Batch B:

Distributed Training Memory: The 16Ψ Rule & ZeRO

Training a model with $Ψ$ parameters in mixed precision (BF16 weights/gradients + FP32 AdamW master states) requires 16 bytes per parameter before activations:

Parameters (BF16): $2Ψ bytes$
Gradients (BF16): $2Ψ bytes$
AdamW Master Weights, Momentum $m_{t}$ , and Variance $v_{t}$ (FP32): $4Ψ + 4Ψ + 4Ψ = 12Ψ bytes$

For a 70B model ( $Ψ = 7 \times 1 0^{10}$ ), static states alone require $16 \times 70 B = 1, 120 GB$ —exceeding a single 80 GB GPU by $14 \times$ . ZeRO / FSDP (Rajbhandari et al. 2020) eliminates replication across $N_{GPU}$ data-parallel ranks:

Sharding Stage	Partitioned States Across $N_{GPU}$	Per-GPU Memory Formula	70B Footprint ( $N_{GPU} = 64$ )	Communication Volume / Step
Baseline DDP	None (Full replication)	$16Ψ$	$1, 120.0 GB$ (OOM)	$2Ψ$ (`AllReduce` gradients)
ZeRO-1	Optimizer states ( $12Ψ$ )	$4Ψ + \frac{12Ψ}{N _{GPU}}$	$293.1 GB$ (OOM)	$2Ψ$ (`ReduceScatter` + `AllGather`)
ZeRO-2	Optimizer + Gradients ( $14Ψ$ )	$2Ψ + \frac{14Ψ}{N _{GPU}}$	$155.3 GB$ (OOM)	$2Ψ$ (`ReduceScatter` + `AllGather`)
ZeRO-3 / FSDP	Optimizer + Grads + Params ( $16Ψ$ )	$\frac{16Ψ}{N _{GPU}}$	$17.5 GB$ (Fits easily)	$3Ψ$ ( $1.5 \times$ baseline for forward/backward `AllGather`)

Key Papers & References

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Rajbhandari et al. (SC 2020) • Introduced optimizer, gradient, and parameter sharding across data-parallel ranks.

Fast Inference from Transformers via Speculative Decoding

Leviathan et al. (ICML 2023) • Proved lossless parallel rejection sampling for accelerating memory-bound autoregressive decoding.

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Lin et al. (MLSys 2024) • Demonstrated that scaling 1% salient activation channels preserves INT4 accuracy without mixed precision.

Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)

Kwon et al. (SOSP 2023) • Eliminated KV-cache fragmentation via virtual memory paging, multiplying serving batch size.

Alignment & Post-Training Evaluation & Contamination