Deep Learning Optimizers Explained: SGD, Adam, AdamW, Lion, Muon and Schedule-Free Methods for Training LLMs (2026)
Almost every open large language model of the last five years was trained with the same eleven-line algorithm, and most teams still configure it by copying someone else’s numbers. The AdamW optimizer is the default for pretraining, fine-tuning, vision transformers and diffusion models alike. Yet the reasons it beats plain Adam, why LLM recipes drop beta2 to 0.95, and why weight decay behaves like a forgetting timescale are rarely written down in one place. That matters more in 2026 than it did in 2020. Muon has trained a trillion-parameter model, Schedule-Free AdamW won an MLCommons benchmark track, and a careful 2025 study found that most claimed speedups over AdamW shrink to about 1.1x once the baseline is tuned properly. This guide derives each update rule, shows where memory goes, and gives sourced hyperparameters with runnable PyTorch.
What this covers: the optimizer family tree, Adam vs AdamW math, schedules, memory per optimizer, Lion, Muon, Schedule-Free, code, failure modes and a decision matrix.
Context and Background
Training a neural network means repeatedly nudging billions of parameters against a noisy gradient. An optimizer is the rule that turns that gradient into a parameter change. Every rule in this post answers three questions differently: how much history to keep, how to scale each coordinate, and how to keep the weights from drifting.
Stochastic gradient descent (SGD) keeps no history and scales every coordinate equally. Momentum adds an exponential moving average of past gradients, which smooths noise and accelerates along consistent directions. Nesterov momentum evaluates the gradient at the look-ahead point, which damps oscillation slightly. SGD with momentum still dominates some convolutional vision work, but it struggles on transformers, where gradient scales differ by orders of magnitude between embeddings, attention projections and layer norms.
Adaptive methods fix that scale problem. AdaGrad (Duchi, Hazan and Singer, JMLR 2011) divides each coordinate by the root of its accumulated squared gradients, so rarely updated features get larger steps. Its accumulator only grows, so its effective learning rate decays toward zero. RMSProp, introduced by Tieleman and Hinton in a 2012 Coursera lecture rather than a paper, replaced the sum with a moving average. Adam (Kingma and Ba, ICLR 2015) combined RMSProp-style scaling with momentum and added bias correction. AdamW (Loshchilov and Hutter, ICLR 2019) then moved weight decay out of the gradient, and that one change is why the name on every LLM training config ends in a W.
Since then, research has split into memory savers (Adafactor, 8-bit Adam), large-batch methods (LARS, LAMB) and matrix-aware methods (Shampoo, SOAP, Muon), with Lion and Schedule-Free sitting slightly apart.
If you are fine-tuning rather than pretraining, the optimizer interacts with adapter choice; our comparison of LoRA, QLoRA, full fine-tuning and distillation covers that side. Here we focus on the update rule itself, which is shared by every one of those regimes.
How the AdamW Optimizer Works: Update Rule, Decoupled Decay and the Family Tree
The AdamW optimizer keeps two moving averages per parameter: the mean of recent gradients (momentum) and the mean of recent squared gradients (variance). Each step divides the bias-corrected mean by the root of the variance, so every coordinate moves at roughly the learning rate. Weight decay shrinks the weights directly instead of being added to the gradient.

Figure 1: How modern deep learning optimizers descend from two roots, momentum SGD and per-coordinate adaptive scaling.
The left branch of the tree is momentum SGD, which leads to Lion’s sign-based update and to Muon’s orthogonalised momentum. The right branch is adaptive scaling, running from AdaGrad through RMSProp to Adam and AdamW. AdamW then spawns large-batch variants such as LAMB, memory variants such as Adafactor and 8-bit Adam, and Schedule-Free AdamW. Shampoo and SOAP form a third, matrix-preconditioning branch that SOAP reconnects to Adam.
The update rule, line by line
Write the gradient at step t as g, the learning rate as eta and the weight decay coefficient as lambda. The PyTorch documentation for torch.optim.AdamW gives the algorithm in this order:
- Decay the weights: theta = theta − eta·lambda·theta.
- Update momentum: m = beta1·m + (1 − beta1)·g.
- Update variance: v = beta2·v + (1 − beta2)·g².
- Bias-correct: m̂ = m / (1 − beta1^t) and v̂ = v / (1 − beta2^t).
- Step: theta = theta − eta·m̂ / (√v̂ + eps).
PyTorch’s defaults are lr = 1e-3, betas = (0.9, 0.999), eps = 1e-8 and weight_decay = 0.01. Those betas and eps match Algorithm 1 of the original Adam paper. Note the default weight decay: 0.01 is far below the 0.1 that most LLM pretraining recipes use, so never assume the library default is a training recipe.
The ratio m̂ / √v̂ is the heart of the method. If a coordinate’s gradient has a consistent sign, the ratio approaches ±1 and the parameter moves by about eta per step. If the gradient is mostly noise with a mean near zero, the ratio shrinks and the parameter barely moves. This is a signal-to-noise filter, not a curvature estimate, despite the common “second-order” framing. It is also why Adam’s effective step size is bounded by roughly eta per coordinate, regardless of the raw gradient’s magnitude.
Why bias correction exists
Both moving averages start at zero. Without correction, early estimates are biased toward zero, and the two biases do not cancel. A worked example makes this concrete. At step one with beta1 = 0.9 and beta2 = 0.999, m = 0.1·g and v = 0.001·g². The square root of v is about 0.0316·|g|. The uncorrected ratio is therefore 0.1 / 0.0316 ≈ 3.16 times the sign of g.
So, without bias correction, the first update would be more than three times the intended step. Dividing m by 0.1 and v by 0.001 restores a ratio of exactly ±1. The correction fades as t grows, and the variance correction lasts longest because beta2 is closest to one. With beta2 = 0.999, 1 − beta2^t is still only about 0.63 after 1,000 steps. Warmup, discussed below, guards against a related early-training hazard that bias correction alone does not fix.
Adam vs AdamW: the decoupling argument in one equation
The classic way to regularise is L2: add (lambda/2)·‖theta‖² to the loss, which adds lambda·theta to the gradient. For plain SGD this is identical to shrinking the weights by eta·lambda·theta each step, which is why the two terms were used interchangeably for decades. Loshchilov and Hutter’s insight is that the equivalence breaks for adaptive methods.
In Adam with L2, the decay term enters g, and therefore passes through the division by √v̂. The effective shrinkage on coordinate i becomes eta·lambda·theta_i / √v̂_i. Coordinates with large or noisy gradients have large v̂, so they are regularised less. Coordinates with tiny gradients are regularised more. That is the opposite of what you usually want, and it couples the best lambda to the learning rate and to the gradient statistics of every layer.
AdamW applies theta = theta − eta·lambda·theta outside the adaptive scaling. Every coordinate shrinks by the same fraction per step. The paper reports that this decouples the optimal weight decay from the learning rate and substantially improves Adam’s generalisation on image classification. The PyTorch documentation states the consequence plainly: weight decay does not accumulate in the momentum or variance buffers. In short, Adam vs AdamW is not a tuning preference; they are different regularisers, and only AdamW implements true weight decay.
Weight decay as a forgetting timescale
There is a useful first-principles way to pick lambda. Each step multiplies the weights by (1 − eta·lambda). Over many steps that product behaves like exp(−eta·lambda·steps). So 1/(eta·lambda) is a characteristic number of steps after which an old contribution to the weights has decayed by a factor of e.
Take the Llama 2 recipe: peak learning rate 3e-4 and weight decay 0.1. Then eta·lambda = 3e-5, and the timescale is about 33,000 steps at peak learning rate. With a cosine schedule the effective timescale lengthens as eta falls. This framing explains two practical rules. If you double the learning rate, you implicitly double the decay strength. And if your total run is much shorter than 1/(eta·lambda), weight decay barely acts at all, which is common in short fine-tunes. This derivation is our own illustration of the mechanism, not a published scaling rule.
Deeper Analysis: Schedules, Stability, Memory and the New Optimizers
Knowing the update rule is half the story. The other half is the learning rate schedule wrapped around it, the numerical knobs that decide whether a long run stays stable, and the memory bill that each optimizer adds on top of the model.

Figure 2: One AdamW training step in a mixed-precision LLM pipeline, from bf16 gradients to the fp32 master-weight update.
The figure traces a typical step. The backward pass produces gradients in bf16, the global gradient norm is clipped, and the fp32 momentum and variance buffers are updated. The update is then formed from the bias-corrected ratio, decoupled decay is applied to the fp32 master weights, and a fresh bf16 copy is cast for the next forward pass.
Learning rate schedule: warmup, cosine and warmup-stable-decay
Most LLM recipes use a learning rate schedule with warmup and cosine decay. During warmup the learning rate rises linearly from near zero to its peak. Afterwards it follows lr(t) = lr_min + ½(lr_max − lr_min)(1 + cos(π·progress)), where progress runs from 0 to 1 over the remaining steps. Llama 2 used exactly this shape: 2,000 warmup steps and a cosine decay to 10% of peak (Touvron et al., 2023).
Why warm up at all, given bias correction? Bias correction fixes the mean of the early estimates, not their variance. In the first few hundred steps v̂ is averaged over very few samples, so a single small gradient can make √v̂ tiny and the ratio huge. Large random updates at initialisation can push a transformer into a region where attention logits saturate, and it never recovers. A short warmup keeps those steps small while the variance estimate fills in. Its length is a judgement call; ranges of a few hundred to a few thousand steps are common in published recipes.
Cosine decay has one awkward property: it needs the total step count T in advance. If you later decide to train longer, the decayed tail is in the wrong place. The Warmup-Stable-Decay (WSD) schedule from the MiniCPM paper (Hu et al., 2024) holds the learning rate constant after warmup and adds a short decay only at the end. The authors describe it as conducive to continuous training and domain adaptation. It also allows scaling-law studies from a single long run with branched decays.

Figure 3: Three ways to manage the learning rate: warmup plus cosine, warmup-stable-decay, and Schedule-Free iterate averaging.
All three begin with warmup and differ in how they handle the unknown end of training. Cosine commits to T up front, WSD defers that commitment, and Schedule-Free replaces decay with iterate averaging.
Schedule-Free AdamW
The Road Less Scheduled by Defazio and colleagues (2024) starts from an observation: schedules that do not know the stopping step are substantially outperformed by schedules that do. Their method removes the schedule entirely and keeps a running average of iterates alongside a fast sequence, using a theory that unifies scheduling and averaging. The paper claims no hyperparameters beyond standard momentum optimizers. Schedule-Free AdamW was the core of the winning entry in the self-tuning track of the MLCommons AlgoPerf 2024 competition, according to both the paper and the AlgoPerf analysis at ICLR 2025.
There is one operational catch. Because the evaluated weights differ from the training sequence, the reference implementation requires you to switch the optimizer between train and eval modes. Forgetting that switch is the most common integration bug, and it shows up as an evaluation loss that is inexplicably worse than training loss.
beta2, epsilon and gradient clipping: the stability knobs
The variance average with beta2 = 0.999 has an effective memory of about 1/(1 − beta2) = 1,000 steps. That is fine for stationary problems. In LLM pretraining, however, a rare token or a data shard boundary can produce a gradient far larger than anything in the last thousand steps. The slow v cannot react, so √v̂ underestimates the new scale and the update briefly explodes. Lowering beta2 to 0.95 shortens the memory to about 20 steps, so the denominator tracks sudden changes. This is why Llama 2 and many other LLM recipes use betas of (0.9, 0.95).
Epsilon has a more subtle role than “avoid division by zero”. When √v̂ is much smaller than eps, the update becomes eta·m̂/eps, which is momentum SGD with a learning rate of eta/eps. With eps = 1e-8 that regime almost never triggers. In large models, though, some gradient coordinates are genuinely tiny, and the regime can matter. Llama 2 raised eps to 1e-5, which caps the amplification for near-zero-variance coordinates.
Gradient clipping rescales the whole gradient vector when its global L2 norm exceeds a threshold, preserving direction. Llama 2 clips at 1.0. QLoRA clips at 0.3 (Dettmers et al., 2023). With Adam, clipping matters less for the step size than with SGD, because the division by √v̂ already normalises scale. Its real job is protecting the moment buffers. One huge gradient enters v as its square and stays there for hundreds of steps, suppressing learning on those coordinates long after the spike itself.
Optimizer memory: where the bytes go
Optimizer state is often the largest single block of training memory. The ZeRO paper gives the canonical accounting for mixed-precision Adam: 2 bytes per parameter for fp16 weights, 2 for fp16 gradients, and K = 12 for optimizer state, which covers an fp32 master copy plus fp32 momentum and variance (Rajbhandari et al., 2019). That totals 16 bytes per parameter. The paper’s own example is a 7.5B model needing 120 GB under standard data parallelism.
The table applies the same accounting to an illustrative 7B-parameter model. It counts only weights, gradients and optimizer state, using 1 GB = 10⁹ bytes. Activations, temporary buffers and fragmentation come on top. Treat the rows as arithmetic, not measurements.
| Optimizer (mixed precision) | State per param | Bytes per param, total | 7B model, illustrative |
|---|---|---|---|
| SGD, no momentum | fp32 master only | 2 + 2 + 4 = 8 | ~56 GB |
| SGD + momentum, Lion | master + 1 fp32 buffer | 2 + 2 + 8 = 12 | ~84 GB |
| Muon on matrices | master + 1 fp32 momentum | ≈12 (AdamW on the rest) | ~84 GB |
| Adam / AdamW | master + m + v | 2 + 2 + 12 = 16 | ~112 GB |
| 8-bit AdamW | master + 8-bit m and v | 2 + 2 + 4 + 2 = 10 | ~70 GB |
| Adafactor, no momentum | master + factored v | slightly over 8 | ~56–60 GB |
Three conclusions follow. First, the step from AdamW to Lion or Muon saves a quarter of this memory, which is meaningful but not transformative. Second, 8-bit optimizer states, introduced by Dettmers et al. in 8-bit Optimizers via Block-wise Quantization, save a similar amount with almost no algorithmic change. The method uses block-wise quantization, a dynamic non-linear quantization map and a stable embedding layer to keep 32-bit performance. Third, the big wins come from sharding. ZeRO stage 1 partitions exactly these optimizer bytes across data-parallel ranks; our explainer on distributed training with FSDP, ZeRO and NCCL shows how.
For LoRA and QLoRA, the picture flips. Optimizer state exists only for the adapter parameters, often well under 1% of the model, so the optimizer’s memory share becomes negligible. That is why optimizer choice barely matters for memory in adapter fine-tuning, while it dominates in full pretraining.
Lower-precision formats such as those in our NVFP4 vs MXFP4 explainer shrink weights and activations, not the fp32 master weights or moment buffers. Those masters exist because an update is often smaller than the bf16 spacing of the weight it modifies and would round away.
Fused and foreach implementations
PyTorch’s AdamW has three code paths with identical maths: a per-tensor loop, a multi-tensor foreach path tried first on CUDA, and a fused path you must request with fused=True. For models with thousands of tensors, fewer kernel launches matter; our guide to CUDA Graphs and torch.compile explains why.
Lion: sign momentum from program search
Lion, short for EvoLved Sign Momentum, came out of a Google program-search effort described in Symbolic Discovery of Optimization Algorithms (Chen et al., 2023). Its update interpolates the momentum with the current gradient using beta1, takes the sign, adds decoupled weight decay and scales by the learning rate. It then updates momentum with a different coefficient, beta2. The defaults in the paper are beta1 = 0.9 and beta2 = 0.99.
Because sign() gives every coordinate a magnitude of exactly one, Lion’s update has a larger norm than Adam’s. The paper therefore says Lion needs a smaller learning rate and, in turn, a larger decoupled weight decay. Its concrete ViT-B/16 example uses lr = 1e-4 and lambda = 10 for Lion against lr = 1e-3 and lambda = 1.0 for AdamW. Notice that the product eta·lambda is 1e-3 in both cases. That matches the forgetting-timescale view above: the paper keeps the decay timescale constant while changing the step size.
The paper reports up to 2% higher ViT accuracy on ImageNet, up to 5x compute savings on JFT pretraining and up to 2.3x less compute for diffusion. For language modeling it reports similar or better results than Adam, with gains growing with batch size.
Muon: orthogonalised momentum for weight matrices
Muon, which stands for MomentUm Orthogonalized by Newton-Schulz, was described by Keller Jordan in a December 2024 post. It is an optimizer for the 2D weight matrices of hidden layers only. It computes ordinary SGD momentum, then replaces the momentum matrix with an approximately orthogonal matrix that has the same singular vectors but all singular values pushed toward one. That replacement is computed with five iterations of a quintic Newton-Schulz polynomial with tuned coefficients (3.4445, −4.7750, 2.0315).
The intuition is that momentum updates for transformer weight matrices tend to be dominated by a few directions. Orthogonalising them amplifies the rare directions that would otherwise barely move. Jordan reports that the Newton-Schulz overhead is below 1% of FLOPs for typical language model training, estimating about 0.7% for the NanoGPT speedrun setup. Switching the NanoGPT speedrun to Muon improved training speed by 35% at that time, and Muon cut CIFAR-10 time-to-94% from 3.3 to 2.6 A100-seconds.
Scale was the open question until 2025. Moonshot AI’s Muon is Scalable for LLM Training identified two required changes: adding weight decay and adjusting the per-parameter update scale so its RMS matches AdamW’s. With those, their scaling-law experiments indicated roughly 2x the compute efficiency of AdamW under compute-optimal training. They trained Moonlight, a 16B-total, 3B-active Mixture-of-Experts model, on 5.7T tokens. The Kimi K2 technical report went further. It introduced MuonClip, which adds QK-Clip to rescale query and key projection weights after each update and bound attention logit growth. Kimi K2, a 1.04-trillion-parameter MoE with 32B activated parameters, was pretrained on 15.5T tokens with what the authors report as zero loss spikes.
Muon also reached the mainstream framework. The PyTorch documentation (from the 2.9 docs onward) includes torch.optim.Muon with defaults lr = 1e-3, weight_decay = 0.1, momentum = 0.95, Nesterov on and five Newton-Schulz steps. Its adjust_lr_fn option offers Jordan’s original scaling or Moonshot’s “match_rms_adamw” scaling, which multiplies the learning rate by 0.2·√max(A, B) so AdamW learning rates and weight decay can be reused. Embeddings, the output head, biases and norm gains still go to AdamW, so every Muon run is really a hybrid.
Shampoo, SOAP and the matrix-preconditioning line
Shampoo (Gupta, Koren and Singer, 2018) keeps left and right preconditioner matrices per layer and applies their inverse roots. A distributed Shampoo implementation won the external-tuning track of AlgoPerf 2024; the ICLR 2025 analysis says it showed non-diagonal preconditioning beating Adam even on wall-clock time.
SOAP (Vyas et al., 2024) shows that Shampoo with the 1/2 power is equivalent to running Adafactor in the eigenbasis of Shampoo’s preconditioner. SOAP runs Adam in that slowly rotating basis, adding only a preconditioning-frequency hyperparameter. On 360M and 660M language models in the large-batch regime, it reported over 40% fewer iterations and over 35% less wall-clock time than AdamW.
The reality check: speedups shrink against a tuned AdamW
Here is the thesis most optimizer articles skip. In Fantastic Pretraining Optimizers and Where to Find Them, Wen, Hall, Ma and Liang (2025) benchmarked ten optimizers across four model scales from 0.1B to 1.2B parameters, at 1–8x the Chinchilla-optimal data ratio. They found that speedups over a well-tuned AdamW fall from about 1.4x at 0.1B parameters to merely 1.1x at 1.2B. They also found that the fastest optimizers, including Muon and SOAP, all use matrix preconditioners.
Put that next to Moonshot’s roughly 2x claim and Kimi K2’s stable trillion-parameter run, and a consistent picture emerges. Matrix-aware optimizers are real progress, and their benefits at frontier scale may lie as much in stability as in raw speed. But the headline speedup you see in a paper depends heavily on how hard the AdamW baseline was tuned. For a team that has not tuned learning rate, warmup, beta2 and weight decay, the largest available improvement is usually a better-tuned AdamW optimizer, not a new optimizer.
Sophia (Liu et al., 2023) is a cautionary example of the same pattern. It uses a lightweight diagonal Hessian estimate with element-wise clipping and reported about 2x fewer steps than Adam on GPT models from 125M to 1.5B parameters. It has not, to our knowledge, become a default in published frontier recipes, which is a reminder that small-scale wins need independent replication.
LARS, LAMB and Adafactor in brief
LARS and LAMB address very large batches. LAMB (You et al., 2019) computes a per-layer trust ratio, the weight norm divided by the update norm, and scales each layer’s Adam update by it. The paper reports BERT training at a batch size of 32,868 and a cut from about 3 days to 76 minutes on a TPUv3 Pod. It notes that LARS performs poorly on attention models such as BERT, which is why LAMB rather than LARS is the transformer choice.
Adafactor (Shazeer and Stern, 2018) stores only row and column sums of the squared-gradient average for each matrix. It reconstructs per-element estimates from them, giving sublinear memory. It also adds update clipping, a rising beta2 schedule, relative step sizes and, in the reported Transformer setup, no momentum. It matched published Adam results on WMT 2014 English-German. Adafactor remains a sensible choice when optimizer memory is the binding constraint, and its factoring trick reappears inside SOAP.
Walk-through: The AdamW Optimizer in PyTorch, From Param Groups to a Scratch Implementation
The snippets below cover what teams most often get wrong: which parameters receive weight decay, how the schedule is wired, and what an AdamW step computes.
Param groups: no decay on biases, norms and embeddings
Weight decay pulls weights toward zero. That makes sense for large weight matrices, where it controls their norm. It makes little sense for LayerNorm or RMSNorm gains, which should sit near one, or for biases, which carry offsets rather than features. A widely used convention, seen in nanoGPT and many training libraries, applies decay only to tensors with two or more dimensions. Embeddings are a judgement call; some recipes exclude them, others do not.
import math
import torch
from torch import nn
def build_param_groups(model: nn.Module, weight_decay: float = 0.1):
decay, no_decay = [], []
for name, p in model.named_parameters():
if not p.requires_grad:
continue
# 2D+ tensors (Linear / attention / MLP weights) get decay.
# 1D tensors (biases, LayerNorm/RMSNorm gains) do not.
if p.ndim >= 2 and "embed" not in name:
decay.append(p)
else:
no_decay.append(p)
return [
{"params": decay, "weight_decay": weight_decay},
{"params": no_decay, "weight_decay": 0.0},
]
model = MyTransformer() # your model
optimizer = torch.optim.AdamW(
build_param_groups(model, weight_decay=0.1),
lr=3e-4,
betas=(0.9, 0.95),
eps=1e-8,
fused=True, # CUDA only; identical maths, fewer kernels
)
The betas and decay here follow the Llama 2 pattern. Llama 2 itself used eps = 1e-5; treat eps as a value to test rather than copy.
Warmup plus cosine decay with a floor
The simplest robust schedule is a LambdaLR multiplier on the peak learning rate. The function below warms up linearly, then decays along a cosine to a floor of 10% of peak, which is the shape Llama 2 reported.
def warmup_cosine(step, warmup_steps, total_steps, min_ratio=0.1):
if step < warmup_steps:
return (step + 1) / warmup_steps
progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)
progress = min(progress, 1.0)
cosine = 0.5 * (1.0 + math.cos(math.pi * progress))
return min_ratio + (1.0 - min_ratio) * cosine
scheduler = torch.optim.lr_scheduler.LambdaLR(
optimizer,
lr_lambda=lambda s: warmup_cosine(s, warmup_steps=2000, total_steps=100_000),
)
for step, batch in enumerate(loader):
loss = model(**batch).loss
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
scheduler.step() # once per optimizer step, not per epoch
optimizer.zero_grad(set_to_none=True)
Two details cause silent bugs. Call scheduler.step() once per optimizer step; with gradient accumulation, that means once per accumulated update, not per micro-batch. And clip after backward() but before step(); if you use a gradient scaler for fp16, unscale first so the clip threshold applies to real gradients.
A minimal AdamW step from scratch
Writing the update by hand is the fastest way to internalise it and to debug a custom optimizer. This version mirrors the PyTorch pseudocode, including the order of decay and moment updates.
@torch.no_grad()
def adamw_step(params, grads, state, lr, betas=(0.9, 0.95),
eps=1e-8, weight_decay=0.1):
beta1, beta2 = betas
state["t"] = state.get("t", 0) + 1
t = state["t"]
bc1 = 1 - beta1 ** t
bc2 = 1 - beta2 ** t
for i, (p, g) in enumerate(zip(params, grads)):
m = state.setdefault(("m", i), torch.zeros_like(p, dtype=torch.float32))
v = state.setdefault(("v", i), torch.zeros_like(p, dtype=torch.float32))
g32 = g.float()
p.mul_(1 - lr * weight_decay) # decoupled decay first
m.mul_(beta1).add_(g32, alpha=1 - beta1) # first moment
v.mul_(beta2).addcmul_(g32, g32, value=1 - beta2) # second moment
denom = (v / bc2).sqrt_().add_(eps)
p.addcdiv_(m / bc1, denom, value=-lr) # adaptive step
To turn this into L2-regularised Adam, delete the p.mul_ line and add weight_decay * p to g32 instead. Running both on a small model with a large weight decay is a quick way to see Adam vs AdamW diverge: the L2 version leaves high-variance weights almost unregularised.
A Muon hybrid in PyTorch
Muon only touches 2D hidden matrices, so a real setup builds two optimizers. The sketch below uses torch.optim.Muon with Moonshot’s RMS-matching option so the AdamW learning rate can be shared. Check that your installed PyTorch includes the class; it appears in the 2.9 documentation.
muon_params, adamw_params = [], []
for name, p in model.named_parameters():
is_matrix = p.ndim == 2
is_edge = ("embed" in name) or ("lm_head" in name)
(muon_params if is_matrix and not is_edge else adamw_params).append(p)
opt_muon = torch.optim.Muon(
muon_params, lr=3e-4, weight_decay=0.1,
momentum=0.95, nesterov=True, adjust_lr_fn="match_rms_adamw",
)
opt_adamw = torch.optim.AdamW(adamw_params, lr=3e-4,
betas=(0.9, 0.95), weight_decay=0.1)
optimizers = [opt_muon, opt_adamw] # step and schedule both
The core of Muon is short enough to read. This is the Newton-Schulz routine as published in Jordan’s post, lightly commented:
def zeropower_via_newtonschulz5(G, steps=5, eps=1e-7):
a, b, c = (3.4445, -4.7750, 2.0315)
X = G.bfloat16()
X /= (X.norm() + eps) # scale so singular values are <= 1
transposed = G.size(0) > G.size(1)
if transposed:
X = X.T
for _ in range(steps):
A = X @ X.T
B = b * A + c * A @ A
X = a * X + B @ X # quintic pushes singular values toward 1
return X.T if transposed else X
At scale, the systems cost is that these matmuls need the full matrix even when it is sharded across devices.
AdamW Optimizer Hyperparameter Defaults by Training Regime
The table collects published settings, each from its primary source. They are starting points from specific papers, not universal truths, and they assume the batch sizes used in those papers.
| Regime (source) | Optimizer | Peak LR | Betas / eps | Weight decay | Schedule | Clip |
|---|---|---|---|---|---|---|
| LLM pretraining, Llama 2 7B/13B | AdamW | 3.0e-4 | 0.9, 0.95 / 1e-5 | 0.1 | 2,000 warmup, cosine to 10% | 1.0 |
| LLM pretraining, Llama 2 34B/70B | AdamW | 1.5e-4 | 0.9, 0.95 / 1e-5 | 0.1 | same | 1.0 |
| QLoRA fine-tune, 7B/13B | paged AdamW | 2e-4 | beta2 0.999 | not specified here | constant | 0.3 |
| QLoRA fine-tune, 33B/65B | paged AdamW | 1e-4 | beta2 0.999 | not specified here | constant | 0.3 |
| ViT, DeiT-B on ImageNet | AdamW | 5e-4 × batch/512 | library defaults | 0.05 | 5 warmup epochs, cosine, 300 epochs | n/a |
| Diffusion, DiT | AdamW | 1e-4 | library defaults | 0 | constant, no warmup, EMA 0.9999 | n/a |
| ViT-B/16, Lion paper | Lion | 1e-4 | 0.9, 0.99 | 10.0 | per paper | n/a |
| Muon, PyTorch defaults | Muon | 1e-3 | momentum 0.95, Nesterov | 0.1 | user-defined | n/a |
Sources: Llama 2 (Touvron et al., 2023), QLoRA (Dettmers et al., 2023, Table 9 and appendix), DeiT (Touvron et al., 2020, Table 9), DiT (Peebles and Xie, 2022, training setup), Lion (Chen et al., 2023, Section 5), and the PyTorch Muon documentation.
Two gaps deserve honesty. First, full-parameter supervised fine-tuning of a pretrained LLM has no single canonical primary source. Practitioners usually start an order of magnitude or more below the pretraining peak, with short warmup and linear or cosine decay; treat that as a heuristic to sweep, not a sourced default. Second, peak learning rates do not transfer across batch sizes or widths without adjustment. DeiT’s formula scales linearly with batch size; for transformers at scale, a small sweep beats any formula.
The DiT row is a useful counterexample to LLM habits. Its authors found warmup and regularisation unnecessary and relied on a weight EMA, so LLM stability machinery is not automatically required elsewhere.
Choosing an Optimizer: A Decision Matrix
The right choice depends more on your constraint than on which optimizer is newest. The flowchart and matrix below encode the reasoning in this post.

Figure 4: A practical decision path for picking a deep learning optimizer based on regime, memory and schedule constraints.
Adapter fine-tuning goes straight to AdamW because its state is tiny. Other regimes branch on memory pressure, then on scale and engineering budget, then on whether the training horizon is known.
| Situation | First choice | Alternative | Why |
|---|---|---|---|
| LoRA / QLoRA fine-tune | AdamW (paged or 8-bit) | none needed | State is tiny; LR and data matter far more |
| Full fine-tune, memory-bound | 8-bit AdamW | Adafactor | Cuts state from 12 to about 6 bytes per param |
| LLM pretraining, standard team | Tuned AdamW, betas 0.9/0.95 | WSD schedule | Best-understood stability; tuning yields most gain |
| LLM pretraining, frontier or MoE | Muon hybrid with QK-Clip | SOAP, Shampoo | Reported efficiency and stability at scale |
| Unknown training length | Schedule-Free AdamW | WSD | No need to fix T up front |
| Vision transformers, diffusion | AdamW | Lion | Lion reports gains in vision and diffusion |
| Very large batch, BERT-style | LAMB | AdamW with longer warmup | Layer-wise trust ratio |
| Small CNNs | SGD + Nesterov | AdamW | Strong generalisation, minimal state |
If you are training reasoning policies with reinforcement learning, the same AdamW defaults usually apply to the policy update. Our explainer on GRPO and RLVR for reasoning models covers the objective that sits on top of the optimizer.
Trade-offs, Gotchas, and What Goes Wrong
Optimizer failures rarely look like optimizer failures; they look like data bugs or a model that stopped learning.
Loss spikes. A sudden jump in loss that partially recovers is usually a large gradient meeting a stale second-moment estimate. Lower beta2 toward 0.95, confirm global-norm clipping is active, and check whether spikes align with specific data shards. If spikes come from attention logits growing without bound, QK normalisation or Kimi’s QK-Clip address the cause, not the symptom. Restarting from a checkpoint and skipping the offending batches is a common operational fix in large runs.
Divergence after warmup. If loss explodes right as the learning rate peaks, the peak is too high for that batch size and width, or warmup was too short for v̂ to stabilise. Halving the peak learning rate is a faster diagnostic than any other change. Long runs on spot capacity add another trap: if a resumed job restores weights but not optimizer state and scheduler step, it restarts with zero moments at peak learning rate. Our guide to interruption-tolerant GPU training and checkpointing covers saving all three.
Too much weight decay. With decoupled decay, lambda times eta sets the forgetting timescale. Copying lambda = 0.1 into a recipe whose learning rate is 10x larger gives 10x stronger decay. Symptoms are a weight norm that keeps falling and a loss that plateaus early. When you change the learning rate, re-check the product.
Precision traps. Applying updates directly to bf16 weights rounds away small steps, so training stalls late in a cosine schedule when steps are smallest. Keep fp32 master weights, or use an optimizer that implements compensated summation, and keep moment buffers in fp32 unless you deliberately use 8-bit state.
Mismatched hyperparameters after switching optimizers. Lion needs a smaller learning rate and larger decay. Muon needs either its own learning rate or the RMS-matching scale. Schedule-Free needs train and eval mode switches. Most “new optimizer did worse” reports trace back to one of these.
Benchmark illusions. As the 2025 Stanford study shows, speedups measured against an untuned AdamW baseline overstate real gains, and gains measured at 0.1B parameters can largely vanish at 1B. Treat any optimizer claim without a tuned baseline as unverified for your setting.
Practical Recommendations
Start with the AdamW optimizer and invest in tuning it before reaching for anything newer. For transformers, use betas of (0.9, 0.95), weight decay around 0.1 on matrices only, global-norm clipping at 1.0, and a warmup followed by cosine or WSD decay. Sweep the peak learning rate on a small proxy run first; that single knob usually dominates every other choice in this post.
Treat memory as a separate decision from convergence, because the AdamW optimizer’s state is the largest optimizer bill. If optimizer state does not fit, shard it with ZeRO or FSDP before changing algorithms, then try 8-bit states. Changing to Lion or Muon saves roughly a quarter of the bytes, which helps but rarely solves an out-of-memory problem alone.
Adopt Muon when you control the training stack, train mostly dense or MoE transformer matrices, and can run a fair comparison against a tuned AdamW baseline. Use the RMS-matching scale so you can reuse AdamW settings, keep AdamW for embeddings and heads, and monitor attention logit magnitudes. If your horizon is unknown or you run continual pretraining, prefer WSD or Schedule-Free so you are not locked into a cosine tail.
Finally, log per-layer update RMS, pre-clip gradient norm and the fraction of clipped steps; they explain most instabilities early.
A short checklist before launching a run:
- Decay applies only to 2D weight matrices; norms and biases are excluded.
- beta2 is 0.95 for LLM pretraining; clipping at 1.0 is active and logged.
- Warmup length and peak LR were validated on a proxy run.
- The product of learning rate and weight decay was re-checked after any LR change.
- Checkpoints save optimizer state and scheduler step, not just weights.
- Any new optimizer is compared against a tuned AdamW baseline at the target scale.
Frequently Asked Questions
What is the difference between Adam and AdamW?
Adam with weight decay adds the decay term to the gradient, so it passes through Adam’s per-coordinate division by the root of the second moment. Coordinates with large or noisy gradients then receive less regularisation. AdamW subtracts learning rate times decay times the weight directly from the parameters, outside the adaptive scaling, so every coordinate shrinks by the same fraction. Loshchilov and Hutter showed this decouples the best weight decay from the learning rate and improves generalisation. For transformer training, AdamW is the correct default.
Why do LLMs use beta2 = 0.95 instead of 0.999?
The second-moment average with beta2 = 0.999 remembers about 1,000 steps, so it reacts slowly when a rare, very large gradient appears. The denominator stays small, and the resulting update can spike and destabilise training. With beta2 = 0.95 the memory is about 20 steps, so the denominator tracks sudden changes in gradient scale. Llama 2 and many other LLM recipes use (0.9, 0.95) for exactly this stability reason. Smaller models and fine-tunes often keep the 0.999 default without problems.
What is a good learning rate for the AdamW optimizer?
It depends on model size, batch size and regime. Llama 2 used a peak of 3e-4 for its 7B and 13B models and 1.5e-4 for 34B and 70B, with cosine decay. QLoRA used 2e-4 for 7B and 13B adapters and 1e-4 for larger ones with a constant schedule. DiT used a constant 1e-4. DeiT scaled 5e-4 by batch size over 512. Use these as starting points and run a short sweep, since peak learning rate is the most sensitive AdamW hyperparameter.
Is the Muon optimizer better than AdamW?
For hidden-layer weight matrices, Muon has strong evidence behind it. Moonshot reported roughly 2x the compute efficiency of AdamW in scaling-law experiments, and Kimi K2 trained a 1.04-trillion-parameter model on 15.5T tokens with MuonClip without loss spikes. However, an independent 2025 study found that matrix optimizers’ advantage over a well-tuned AdamW shrank to about 1.1x at 1.2B parameters. Muon also still needs AdamW for embeddings, heads and 1D parameters. It is a strong option at scale, not a free win.
Why is learning rate warmup needed with Adam?
Bias correction fixes the average of early moment estimates, but not their variance. In the first few hundred steps, the second-moment estimate is built from very few gradients, so some coordinates get a tiny denominator and a disproportionately large update. Those early large steps can push a transformer into unstable regions, such as saturated attention, from which it does not recover. Warmup keeps the learning rate small while the estimates settle. Llama 2 used 2,000 warmup steps, while DiT reported that it did not need warmup at all.
How much GPU memory does the Adam optimizer use?
In standard mixed-precision training, Adam keeps an fp32 master copy of the weights plus fp32 momentum and variance, which is 12 bytes per parameter. Adding 2-byte weights and 2-byte gradients gives the 16 bytes per parameter that the ZeRO paper uses. For an illustrative 7B model that is about 112 GB before activations. 8-bit optimizer states cut the state from 12 to roughly 6 bytes, Lion and Muon keep one buffer instead of two, and ZeRO or FSDP shard the state across GPUs.
Further Reading
- GRPO and RLVR explained for reasoning models — the RL objective that runs on top of the optimizer in reasoning-model post-training.
- LoRA vs QLoRA vs full fine-tuning vs distillation — how adapter choice changes optimizer memory and learning-rate ranges.
- FlashAttention 3 and 4: IO-aware attention on the GPU memory hierarchy — the activation-memory side of the training memory budget.
- Distributed training with data, tensor and pipeline parallelism, FSDP and ZeRO — how optimizer state gets sharded.
- Keller Jordan, “Muon: An optimizer for hidden layers in neural networks” — the original Muon write-up.
- PyTorch
torch.optim.AdamWdocumentation — authoritative pseudocode and defaults.
References
- Kingma, D. P., and Ba, J. “Adam: A Method for Stochastic Optimization.” ICLR 2015. https://arxiv.org/abs/1412.6980
- Loshchilov, I., and Hutter, F. “Decoupled Weight Decay Regularization.” ICLR 2019. https://arxiv.org/abs/1711.05101
- Duchi, J., Hazan, E., and Singer, Y. “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.” JMLR 12, 2011. https://www.jmlr.org/papers/v12/duchi11a.html
- Shazeer, N., and Stern, M. “Adafactor: Adaptive Learning Rates with Sublinear Memory Cost.” 2018. https://arxiv.org/abs/1804.04235
- You, Y., et al. “Large Batch Optimization for Deep Learning: Training BERT in 76 minutes.” 2019. https://arxiv.org/abs/1904.00962
- Rajbhandari, S., et al. “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.” 2019. https://arxiv.org/abs/1910.02054
- Dettmers, T., et al. “8-bit Optimizers via Block-wise Quantization.” 2021. https://arxiv.org/abs/2110.02861
- Chen, X., et al. “Symbolic Discovery of Optimization Algorithms.” 2023. https://arxiv.org/abs/2302.06675
- Liu, H., et al. “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training.” 2023. https://arxiv.org/abs/2305.14342
- Touvron, H., et al. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” 2023. https://arxiv.org/abs/2307.09288
- Dettmers, T., et al. “QLoRA: Efficient Finetuning of Quantized LLMs.” 2023. https://arxiv.org/abs/2305.14314
- Touvron, H., et al. “Training data-efficient image transformers and distillation through attention” (DeiT). 2020. https://arxiv.org/abs/2012.12877
- Peebles, W., and Xie, S. “Scalable Diffusion Models with Transformers” (DiT). 2022. https://arxiv.org/abs/2212.09748
- Hu, S., et al. “MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies.” 2024. https://arxiv.org/abs/2404.06395
- Defazio, A., et al. “The Road Less Scheduled.” 2024. https://arxiv.org/abs/2405.15682
- Vyas, N., et al. “SOAP: Improving and Stabilizing Shampoo using Adam.” 2024. https://arxiv.org/abs/2409.11321
- Jordan, K. “Muon: An optimizer for hidden layers in neural networks.” December 2024. https://kellerjordan.github.io/posts/muon/
- Liu, J., et al. “Muon is Scalable for LLM Training.” Moonshot AI, 2025. https://arxiv.org/abs/2502.16982
- Kimi Team. “Kimi K2: Open Agentic Intelligence.” 2025. https://arxiv.org/abs/2507.20534
- Wen, K., Hall, D., Ma, T., and Liang, P. “Fantastic Pretraining Optimizers and Where to Find Them.” 2025. https://arxiv.org/abs/2509.02046
- MLCommons AlgoPerf authors. “Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition.” ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/6bdde0373d53d4a501249547084bed43-Abstract-Conference.html
- PyTorch documentation: torch.optim.AdamW and torch.optim.Muon. https://docs.pytorch.org/docs/main/generated/torch.optim.AdamW.html and https://docs.pytorch.org/docs/2.9/generated/torch.optim.Muon.html
By Riju — about
