Fire or don't: a hard yes/no decisionMax depth, FFN without residuals: 1 layer
X
· Wᵀ
ŷ
H(·)
p̂
∂L/∂p̂
p̂ = H(ŷ) ={
1,ŷ > 00,ŷ ≤ 0
∂p̂/∂ŷ ={
0,ŷ > 00,ŷ < 0
∂L/∂ŷ = ∂L/∂p̂ · ∂p̂/∂ŷ = ∂L/∂p̂ · 0 = 0
A flat step gives gradient descent nothing to follow: no signal reaches W.
Sigmoid: a differentiable step
Max depth, FFN without residuals: ≈ 10 layers (quasi-linear region)
X
· Wᵀ
ŷ
σ(·)
p̂
∂L/∂p̂
p̂ = σ(ŷ) = 1 / (1 + e−ŷ)
╍ H(ŷ) for comparison
∂p̂/∂ŷ = σ(ŷ)(1 − σ(ŷ)) ≤ 0.25
positive everywhere, but small
Problem 1: the gradient shrinks. ∂p̂/∂ŷ ≤ 0.25, so every sigmoid layer cuts the back-propagated gradient by at least 4×, and by far more once |ŷ| is large.
Problem 2: the mean is not zero. σ(0) = 0.5. With weights initialised for the quasi-linear region, the mean's share of the second moment, E[p̂²] = Var[p̂] + E[p̂]², grows layer over layer.
Sigmoid fixes zero gradients but brings two new problems: shrinking gradients and a non-zero mean.
ReLU: a gradient of exactly 1
Max depth, FFN without residuals: ≈ 50 layers
X
· Wᵀ
ŷ
ReLU
p̂
∂L/∂p̂
p̂ = max(0, ŷ) ={
ŷ,ŷ > 00,ŷ ≤ 0
∂p̂/∂ŷ ={
1,ŷ > 00,ŷ < 0
Fixed: the gradient. ∂p̂/∂ŷ = 1 for every ŷ > 0, so active units pass the back-propagated gradient through unchanged, however many layers deep.
Still there: the non-zero mean. p̂ ≥ 0, so E[p̂] > 0, and the mean's share of E[p̂²] = Var[p̂] + E[p̂]² still grows layer over layer. That leads to correlated representations: a shared positive offset makes features move together. It also leads to dying neurons: a unit pushed to ŷ ≤ 0 for every input gets zero gradient and never recovers.
ReLU fixes the shrinking gradient but keeps the non-zero mean.
Can you build ReLU from these two?
Talk it over with a neighbour before we move on
Ingredient
H(ŷ)
Ingredient
ŷ
Target: ReLU
max(0, ŷ)
Using only H(ŷ) and ŷ, how would you write max(0, ŷ)?
ReLU is a gated identity
Zoom in on max(0, ŷ): it is the input, multiplied by a threshold
ŷ
H(ŷ)
×
p̂
scaled
p̂ = max(0, ŷ)=ŷ·H(ŷ)
ŷ
the stream, unchanged
×
H(ŷ)
the perceptron's threshold
=
ŷ · H(ŷ)
ReLU
The step function we dropped from perceptrons returns in ReLU. The stream is multiplied by the threshold.
The threshold is back, so why is it fine now?
Applied to the stream, the step kills the gradient; multiplied into it, the step becomes the gradient
∂L/∂ŷ = ∂L/∂p̂ · ∂p̂/∂ŷ = ∂L/∂p̂ · g(ŷ) + ∂L/∂p̂ · ŷ · ∂g(ŷ)/∂ŷ← chain rule: the two dashed paths into ŷ
Gradients from paths that meet at ŷ add. That sum is exactly the product rule.
The scaled path works like a skip connection, but the gate scales the gradient instead of passing it unchanged.
Each gate sees only its own unit
x₁
Σ
ŷ₁
g(ŷ₁)
×
p̂₁
x₂
Σ
ŷ₂
g(ŷ₂)
×
p̂₂
x₃
Σ
ŷ₃
g(ŷ₃)
×
p̂₃
x₄
Σ
ŷ₄
g(ŷ₄)
×
p̂₄
inputs
Linear · Wᵀ: every input → every unit
activation: one unit at a time
Linear layer mixes. Each ŷₖ = Σⱼ Wₖⱼ xⱼ reads all four inputs. The dark lines are everything feeding unit 2.
The gate does not. Unit 2's gate reads only ŷ₂. No arrow crosses between units after the Σ.
p̂ₖ = ŷₖ · g(ŷₖ)
The linear layer mixes features, then an elementwise gate acts on each unit alone.
What if the gate were a sigmoid?
Replace H(ŷ) with σ(ŷ): a soft threshold
ŷ
σ(ŷ)
×
p̂
scaled
p̂ = SiLU(ŷ)=ŷ·σ(ŷ)
ŷ
the stream, unchanged
×
σ(ŷ)
a soft threshold ╍ H(ŷ)
=
ŷ · σ(ŷ)
SiLU ╍ ReLU
SiLU: Hendrycks & Gimpel, 2016; Elfwing et al., 2017; Ramachandran et al., 2017
Swap the hard step for a sigmoid and you get SiLU: ReLU with a soft threshold.
Why is SiLU an improvement on ReLU?
Talk it over with a neighbour before we move on
ReLU
ŷ · H(ŷ)
SiLU
ŷ · σ(ŷ)
The two curves look almost the same. What does SiLU give a unit that ReLU does not? Think about the derivative and dying ReLU...
Why is SiLU better?
ReLU's gradient is exactly zero for every ŷ < 0; SiLU's is small but not zero
ReLU
ŷ · H(ŷ)
∂/∂ŷ
ReLU′
SiLU
ŷ · σ(ŷ)
∂/∂ŷ
SiLU′
ReLU: ∂p̂/∂ŷ = H(ŷ)
SiLU: ∂p̂/∂ŷ = σ(ŷ) + ŷ · σ(ŷ)(1 − σ(ŷ))
ŷ
3
1
0
−1
−3
ReLU′
1
1
0
0
0
SiLU′
1.088
0.928
0.500
0.072
−0.088
ReLU′(0) taken as 0. Both are ≈ 1 on the positive side.
In later layers, correlated activations push many ReLU units below zero on every input. Those units are always off: ReLU′ = 0, so no gradient reaches their weights and they stop learning. SiLU still passes a small gradient for moderate ŷ < 0, so an off unit can recover.
SiLU keeps a small gradient where ReLU has none, so switched-off neurons can still learn.
The gated-identity family
One recipe, p̂ = ŷ · g(ŷ), with a different gate g each time
ŷ
g(ŷ)
×
p̂
scaled
The gates, g(ŷ)
━ H━ σ━ Φ
The activations, p̂ = ŷ · g(ŷ)
╍ ŷ━ ReLU━ SiLU━ GELU
Mean for ŷ ~ N(0, 1)
Activation
E[p̂]
E[p̂]²/E[p̂²]
ŷ, no gate
0
0
σ(ŷ) alone
0.500
0.85
ReLU
0.399
0.32
GELU
0.282
0.19
SiLU
0.207
0.12
Right column: the mean's share of the second moment. Smaller means less of the activation is a shared offset.
ReLU = ŷ · H(ŷ) · SiLU = ŷ · σ(ŷ) · GELU = ŷ · Φ(ŷ), Φ the standard normal CDF (BERT, GPT-2)
Whatever the gate, once it is open the gradient is about 1, which keeps deep networks learning.
These gates only look at one feature
Jacobians ∂P̂/∂Ŷ for the same input, ŷ = [−3, −1, 0, 1, 3]
SiLUp̂ₖ = ŷₖ · σ(ŷₖ)
i \ j
ŷ=−3
ŷ=−1
ŷ=0
ŷ=1
ŷ=3
p̂₁
∂p̂₁/∂ŷ₁−0.088
00
00
00
00
p̂₂
00
∂p̂₂/∂ŷ₂0.072
00
00
00
p̂₃
00
00
∂p̂₃/∂ŷ₃0.500
00
00
p̂₄
00
00
00
∂p̂₄/∂ŷ₄0.928
00
p̂₅
00
00
00
00
∂p̂₅/∂ŷ₅1.088
Diagonal only: p̂ₖ depends on ŷₖ alone. The gate for feature k never sees feature j.
Softmaxp̂ₖ = eŷₖ / Σⱼ eŷⱼ
i \ j
ŷ=−3
ŷ=−1
ŷ=0
ŷ=1
ŷ=3
p̂₁
p̂₁(1−p̂₁)0.002
−p̂₁p̂₂−3.1e−5
−p̂₁p̂₃−8.5e−5
−p̂₁p̂₄−2.3e−4
−p̂₁p̂₅−0.002
p̂₂
−p̂₂p̂₁−3.1e−5
p̂₂(1−p̂₂)0.015
−p̂₂p̂₃−6.3e−4
−p̂₂p̂₄−0.002
−p̂₂p̂₅−0.013
p̂₃
−p̂₃p̂₁−8.5e−5
−p̂₃p̂₂−6.3e−4
p̂₃(1−p̂₃)0.040
−p̂₃p̂₄−0.005
−p̂₃p̂₅−0.034
p̂₄
−p̂₄p̂₁−2.3e−4
−p̂₄p̂₂−0.002
−p̂₄p̂₃−0.005
p̂₄(1−p̂₄)0.100
−p̂₄p̂₅−0.093
p̂₅
−p̂₅p̂₁−0.002
−p̂₅p̂₂−0.013
−p̂₅p̂₃−0.034
−p̂₅p̂₄−0.093
p̂₅(1−p̂₅)0.142
Dense: every p̂ₖ depends on every ŷⱼ, with off-diagonal −p̂ᵢp̂ⱼ (from the softmax lecture).
GLU: gate k has its own row of weights and reads all four inputs. The dark lines are everything feeding gate 2.
p̂ₖ = xₖ · σ(Σⱼ Wₖⱼ xⱼ)
GLU moves the mixing into the gate: every gate reads every feature, while each value passes straight through.
GLU
Gated linear unit: compute the gate from a linear projection of all features
X
e × f
· Wᵀ W: f × f
σ(·)
⊙
P̂
e × f
XWᵀ
W, f × fcolumn j = input feature j
1
W11
W12
⋯
W1f
2
W21
W22
⋯
W2f
⋮
⋮
⋮
⋮
k
Wk1
Wk2
⋯
Wkf
⋮
⋮
⋮
⋮
f
Wf1
Wf2
⋯
Wff
Row k holds the weights that gate feature k: one row per feature, f rows.
gate, e × f
scaled path
P̂ = σ(XWᵀ) ⊙ X
Gate on feature k: σ(Σⱼ xⱼ Wₖⱼ), a weighted vote of every feature.
Gated linear units: Dauphin et al., 2017 (their form adds the value projection on the next slide)
Set W = I and the gate is σ(X): that is SiLU. GLU learns which features open each gate.
GLU is SiLU whose gate reads every feature, like softmax, with each feature having its own gate weight matrix.
· Woutᵀ f × n
e × f
Adding a value projection
Gate a linear projection XVᵀ instead of X itself
f dimensionsn dimensions
X
e × f
· Wᵀ W: n × f
XWᵀ
XVᵀ
σ(·)
· Vᵀ V: n × f
⊙
P̂
e × n
gate, e × n
value, e × n
P̂ = σ(XWᵀ) ⊙ XVᵀ then P̂ Woutᵀ
n is our choice, so mixing and gating can happen in a wider or narrower space than the f input features.
Inside a transformer feed-forward block (T5 base, Shazeer 2020): f = 768, n = 2048, then Wout (f × n) maps n back to f.
V lets us choose the dimension where mixing and gating happen.
The value projection
Unit k multiplies its own gate σ((XWᵀ)ₖ) by its own value (XVᵀ)ₖ
x₁
Σ
Σ
(XWᵀ)₁
(XVᵀ)₁
σ(·)
×
p̂₁
h₁
x₂
Σ
Σ
(XWᵀ)₂
(XVᵀ)₂
σ(·)
×
p̂₂
h₂
x₃
Σ
Σ
(XWᵀ)₃
(XVᵀ)₃
σ(·)
×
p̂₃
h₃
x₄
Σ
Σ
(XWᵀ)₄
(XVᵀ)₄
σ(·)
×
p̂₄
h₄
inputs
W and V: every input
gate
· Woutᵀ: n → f
W rows → gates · V rows → values · dashed Wout maps the n units back to f outputs
p̂ₖ = σ(Σⱼ Wₖⱼ xⱼ) · Σⱼ Vₖⱼ xⱼ H = P̂ Woutᵀ
Here f = n = 4. In T5 base, f = 768, n = 2048.
Linear W n × f
Linear V n × f
σ
⊙
Linear Wout f × n
X
e×f
XVᵀ, e×n
e×n
P̂
e×n
H
e×f
Now the gate and the value both mix every input; each unit multiplies its own pair.
SwiGLU: SiLU as the gate
Put a gated identity inside the gated linear unit (the name comes from Swish, another name for SiLU)
X
· Wᵀ
· Vᵀ
SiLU(·)
⊙
XWᵀ
gate
XVᵀ
P̂
P̂ = SiLU(XWᵀ) ⊙ XVᵀSiLU(u) = u · σ(u)
Swap the gate function in GLU (Shazeer, 2020)
Gate function
Decomposed into H, σ, Φ
GLU
σ(XWᵀ) ⊙ XVᵀ
σ(XWᵀ) ⊙ XVᵀ
ReGLU
ReLU(XWᵀ) ⊙ XVᵀ
H(XWᵀ) ⊙ XWᵀ ⊙ XVᵀ
GEGLU
GELU(XWᵀ) ⊙ XVᵀ
Φ(XWᵀ) ⊙ XWᵀ ⊙ XVᵀ
SwiGLU
SiLU(XWᵀ) ⊙ XVᵀ
σ(XWᵀ) ⊙ XWᵀ ⊙ XVᵀ
Each of these has three weight matrices (W, V and the output layer), so n is cut to 2/3 to keep parameters equal: 3072 → 2048 in the T5 experiments.
The default feed-forward activation in today's LLMs
PaLM · LLaMA family · Mistral · Qwen
Two gates stacked:
SiLU gates XWᵀ by itself, so its gradient stays ≈ 1 where open.
⊙ XVᵀ gates the value with a signal built from every feature.
SwiGLU is a gated identity inside a gated linear unit.
Do we really need the nonlinear activation?
SwiGLU already multiplies two projections of the input
X
· Wᵀ
· Vᵀ
σ(·)
needed?
⊙
⊙
XWᵀ
XVᵀ
P̂
P̂ = σ(XWᵀ) ⊙ XWᵀ ⊙ XVᵀ
explicit nonlinearityproduct of two projections
x is linear. x · x is not.╍ x━ x · x
Multiplying two linear functions of the input already bends the output.
So drop the σ?
P̂ = XWᵀ ⊙ XVᵀ
p̂ₖ = (x · wₖ)(x · vₖ) = Σᵢ Σⱼ Wₖᵢ Vₖⱼ xᵢ xⱼ
x is one row of X; wₖ, vₖ are row k of W, V. Every output is a quadratic in x, built from products of input features xᵢ xⱼ. This is the bilinear layer (Mnih & Hinton, 2007; Dauphin et al., 2017).
If the product is already nonlinear, does the extra σ earn its place?
x · x is already nonlinear, so the σ in SwiGLU may be a nonlinearity we don't need.
How much does the nonlinearity add?
Held-out log-perplexity, T5 base on C4, parameters and compute matched (Shazeer, 2020). Lower is better.
Feed-forward layer
65,536 steps (sd)
524,288 steps
vs ReLU
GEGLU
1.942 (0.004)
1.633
−0.044
SwiGLU
1.944 (0.010)
1.636
−0.041
ReGLU
1.953 (0.003)
1.645
−0.032
Bilinear (no σ)
1.960 (0.005)
1.648
−0.029
GLU
1.982 (0.006)
1.663
−0.014
ReLU (baseline)
1.997 (0.005)
1.677
0
GELU
1.983 (0.005)
1.679
+0.002
SiLU
1.994 (0.003)
1.683
+0.006
Bilinear gets ≈ 70% of SwiGLU's gain over ReLU (0.029 of 0.041) with no elementwise nonlinearity.
Above the line: every gated layer beats every ungated one at 524k steps.
Below the line: swapping ReLU for GELU or SiLU alone changes almost nothing here.
SwiGLU
X
· Wᵀ
· Vᵀ
⊙
P̂
SiLU(·)
Bilinear
X
· Wᵀ
· Vᵀ
⊙
P̂
no gate function
Most of SwiGLU's gain comes from the product of two projections, not from the σ.
Activation functions are still being invented
What we built today, one idea at a time
XWᵀ
Linear: no activation, so stacked layers collapse into one
H(XWᵀ)
Perceptron threshold: no gradient
σ(XWᵀ)
Sigmoid: differentiable, but gradient ≤ 0.25 and a non-zero mean
H(XWᵀ) ⊙ XWᵀ
ReLU: multiply by the threshold instead
σ(XWᵀ) ⊙ XWᵀ
SiLU: soften the threshold
σ(XWᵀ) ⊙ XVᵀ
GLU: gates read every feature
σ(XWᵀ) ⊙ XWᵀ ⊙ XVᵀ
SwiGLU: the current LLM default
XWᵀ ⊙ XVᵀ
Bilinear: almost as good, no σ
Open questions
Why do gated layers beat ungated ones at equal parameters?
How much of the gain is the product, and how much the σ?
What should the gate look at, and in what dimension?
Shazeer (2020), on why GLU variants work: “we attribute their success, as all else, to divine benevolence.”
Activation design is still an open research question; expect today's default to change.
SwiGLU h₁ × din
SwiGLU h₂ × h₁
SwiGLU h₃ × h₂
Linear o × h₃
Softmax
x
e × din
e×h₁
e×h₂
e×h₃
e×o
P̂
e × o
Layer 1
Layer 2
Layer 3
output layer
hidden layers: three SwiGLU layers
Each SwiGLU layer opened up
Linear h₁ × din
Linear h₁ × din
σ
⊙
⊙
e×h₁
e×h₁
e×h₁
Layer 1
Linear h₂ × h₁
Linear h₂ × h₁
σ
⊙
⊙
e×h₂
e×h₂
e×h₂
Layer 2
Linear h₃ × h₂
Linear h₃ × h₂
σ
⊙
⊙
e×h₃
e×h₃
e×h₃
Layer 3
x
e×din
Linear o × h₃
Softmax
P̂
e×o
e×h₁
e×h₂
e×h₃
e×o
SwiGLU in a feed-forward network
e = examples, din = input features, hₖ = width of layer k, o = outputs
Both Linears in layer k are hₖ × hₖ₋₁. With no residual stream, every layer's gradient must pass through all the layers after it.
SwiGLU drops in as the activation of a plain feed-forward network; next, we give it a residual stream.
SwiGLU in a residual MLP
e = examples, f = backbone features, hₖ = hidden width of block k, hₒ = hidden width of the output head, o = outputs
+
+
+
Linear Wout: f × h₁
SwiGLU
Norm
Linear Wout: f × h₂
SwiGLU
Norm
Linear Wout: f × h₃
SwiGLU
Norm
Down projection W: f × din
Norm
SwiGLU
Linear W: o × hₒ
Softmax
e×f
e×h₁
e×f
e×f
e×h₂
e×f
e×f
e×h₃
e×f
Block 1
Block 2
Block 3
X₀ e × f
X₁ e × f
X₂ e × f
X₃ e × f
final norm
output projection
e×f
e×hₒ
e×o
P̂
e × o, class probabilities
x, e × din
Each block normalizes the stream, applies SwiGLU (f → hₖ), projects back with Wout (f × hₖ) and adds the result to the stream. The next slide opens up the SwiGLU boxes.
+
Linear f × h₁
⊙
⊙
σ
Linear h₁ × f
Linear h₁ × f
Norm
Block 1
+
Linear f × h₂
⊙
⊙
σ
Linear h₂ × f
Linear h₂ × f
Norm
Block 2
+
Linear f × h₃
⊙
⊙
σ
Linear h₃ × f
Linear h₃ × f
Norm
Block 3
Norm
Linear hₒ × f
Linear hₒ × f
σ
⊙
⊙
Linear o × hₒ
Softmax
P̂ e × o
final norm
output proj.
Down projection W: f × din
x, e × din
X₀ e × f
X₁ e × f
X₂ e × f
X₃ e × f
e×f
e×h₁
e×h₁
e×h₁
e×h₁
e×f
e×f
e×h₂
e×h₂
e×h₂
e×h₂
e×f
e×f
e×h₃
e×h₃
e×h₃
e×h₃
e×f
e×f
e×hₒ
e×hₒ
e×hₒ
e×o
Inside a modern residual MLP
Block k, with N = Norm(Xₖ₋₁): Xₖ = Xₖ₋₁ + (σ(NWₖᵀ) ⊙ NWₖᵀ ⊙ NVₖᵀ) Wout,kᵀ
━ scaled path inside SiLU━ residual stream
Expert 1
Expert 2
x₁
Σ
Σ
(XWᵀ)₁
(XVᵀ)₁
σ(·)
×
p̂₁
×
h₁
x₂
Σ
Σ
(XWᵀ)₂
(XVᵀ)₂
σ(·)
×
p̂₂
×
h₂
x₃
Σ
Σ
(XWᵀ)₃
(XVᵀ)₃
σ(·)
×
p̂₃
×
h₃
x₄
Σ
Σ
(XWᵀ)₄
(XVᵀ)₄
σ(·)
×
p̂₄
×
h₄
Σ
(x Wrᵀ)₁
Σ
(x Wrᵀ)₂
softmax
g₁ = 0.7
g₂ = 0.3
inputs
· Woutᵀ
each expert owns its units: their rows of W, V and columns of Wout · router: a linear map Wr (one row per expert), then softmax
Linear W n × f
Linear V n × f
Linear Wr, E × f
softmax
σ
⊙
⊙
G
P̂
Linear Wout f × n
X
H
XVᵀ, e×n
Mixture of experts: one gate per expert
Group the hidden units into experts; a router gives each expert one gate
g = softmax(x Wrᵀ) H = Σᵢ gᵢ · Eᵢ(x)
One gate, many units. g₁ scales both units of expert 1.
Gates compete. The softmax makes raising one gate lower the other.
In the block: P̂ ⊙ G, with gᵢ repeated over expert i's units. Shazeer et al., 2017
MoE puts one gate on a whole block of units, and the gates compete through a softmax.
Expert 1
Expert 2
x₁
Σ
Σ
(XWᵀ)₁
(XVᵀ)₁
σ(·)
×
p̂₁
×
h₁
x₂
Σ
Σ
(XWᵀ)₂
(XVᵀ)₂
σ(·)
×
p̂₂
×
h₂
x₃
Σ
Σ
(XWᵀ)₃
(XVᵀ)₃
σ(·)
×
p̂₃
×
h₃
x₄
Σ
Σ
(XWᵀ)₄
(XVᵀ)₄
σ(·)
×
p̂₄
×
h₄
Σ
(x Wrᵀ)₁
Σ
(x Wrᵀ)₂
softmax top-1
g₁ = 0.7 kept
g₂ → 0
inputs
· Woutᵀ
each expert owns its units: their rows of W, V and columns of Wout · router: a linear map Wr (one row per expert), then softmax
Linear W n × f
Linear V n × f
Linear Wr, E × f
top-1
σ
⊙
⊙
G
P̂
Linear Wout f × n
X
H
XVᵀ, e×n
Top-k routing: a hard gate on experts
Keep the k largest gates and skip the rest (here k = 1)
g̃ᵢ = gᵢ · H(gᵢ − τ) τ = k-th largest gate
The Heaviside step is back, on whole experts. Expert 2 is never computed.
Parameters grow with E; compute with k. Huge but sparse: Kimi K2 uses ≈ 3% of its weights per token.
model
k/E
total/active
Mixtral 8x7B
2/8
47B/13B
DeepSeek-V3
8/256
671B/37B
Kimi K2
8/384
1T/32B
E = experts, k = routed per token (+1 shared expert in DeepSeek-V3, Kimi K2)
Without a balancing term
tokens
Expert 110 tokens
Expert 21 token
Expert 31 token
Experts 2 and 3 get almost no tokens, so almost no gradient: the dead-unit problem, per expert.
With a balancing term
tokens
Expert 14 tokens
Expert 24 tokens
Expert 34 tokens
Every expert gets tokens and keeps learning.
Keeping every expert in use
A skipped expert gets no gradient, so routing can collapse onto a few experts
L = Ltask + α · N · Σᵢ fᵢ Pᵢ
fᵢ: fraction of tokens sent to expert i (a count, no gradient) Pᵢ: mean router probability for expert i (has a gradient) Smallest when every fᵢ = Pᵢ = 1/N; α = 0.01.
This term balances load. The sparsity still comes from top-k.
The extra loss term balances load across experts; top-k supplies the sparsity.