AI 4200 · Deep Learning
The transformer encoder
All slides, one per page
© 2026 Dr. Jesse Roberts
Recap: CNNs
x · e×3×4×4Conv 3×3ReLUmax pool 2global avg poolLinearŷ
0
0
0
0
0
0
0
0.22
0.55
0.23
0.28
0
0
0.38
0.37
0.57
0.44
0
0
0.47
0.26
0.36
0.24
0
0
0.31
0.45
0.59
0.35
0
0
0
0
0
0
0
0
-0.25
0
-0.25
2
-0.25
0
-0.25
0
learned
3 × 3 filter
0.2
0.9
0.1
0.4
0.5
0.3
0.8
0.6
0.7
0.1
0.3
0.2
0.4
0.6
0.9
0.5
0.9
0.8
0.7
0.9
0.825
one channel · 4 × 4
max pool · 2 × 2
global avg · 1
· Wᵀ + b
class 1
+1.15
class 2
-0.42
class 3
+0.41
ŷ · scores
illustrative weights
NOW  Filter → channel → max pool → average → class scores
Max pooling as a gate, then softmax
Top-left quadrant: four values, one gate each
?
MAX POOLING
V · hardmax(V)
0.2
0
0.9
1
0.5
0
0.3
0
  • Output 0.9
  • Gradient: winner only
SOFTPOOL, τ = 0.1
V · softmax(V / τ)
0.2
0.001
0.9
0.979
0.5
0.018
0.3
0.002
  • Output 0.891
  • Gradient: ?
Output barely moves
Every value gets a gradient
τ dials max ↔ average
Pooling is a gate: a weighted sum, weights from the values.
Limits of local pooling
Talk it over with a neighbour before we move on
?
Sky or sea?
A: now?
B: now?
TOP-LEFT PATCH · REPRESENTATION (blue, wings, hull)
Picture A
[1, 0, 0]
Picture B
[1, 0, 0]
Identical. Sky or sea?
The representation has no idea.
Each patch's representation has no idea if it is sky or sea.
Local is all we have
Recap model on picture A: each output is built inside its own quadrant
x · e×3×4×4Conv 3×3ReLUmax pool 2global avg poolLinearŷ
quadrant boundaries
OUTPUT: ONE PER QUADRANT
blue
blue + bird
blue
blue
THE PROBLEM
  • Each local representation is built only from its own quadrant
  • No information crosses a boundary until the next layer
  • So the top-left can never use the bird
All the pooling methods we have seen are local gates. They only use local information. 
SiLU → SwiGLU, local → global gating
Remember the activation-functions lecture: we have fixed local only info once already
?
ŷ₁g(ŷ₁)×p̂₁ŷ₂g(ŷ₂)×p̂₂ŷ₃g(ŷ₃)×p̂₃ŷ₄g(ŷ₄)×p̂₄ŷgate reads one unitgatevalue path: each ŷ goes straight to its own ×
BEFORE THE NEXT CLICK
  • What does each SiLU gate get to see?
  • How did we let the gate see more?
ŷ₁Σ(ŷWᵀ)₁SiLU(·)×p̂₁ŷ₂Σ(ŷWᵀ)₂SiLU(·)×p̂₂ŷ₃Σ(ŷWᵀ)₃SiLU(·)×p̂₃ŷ₄Σ(ŷWᵀ)₄SiLU(·)×p̂₄ŷWᵀ: every ŷ, summedgatevalue path: each ŷ goes straight to its own ×
SiLU: gate k reads only ŷₖ
p̂ₖ = ŷₖ · g(ŷₖ)
SwiGLU: gate k reads every ŷ
p̂ = ŷ ⊙ SiLU(ŷWᵀ)
Pooling today: a quadrant's gate reads only its own quadrant
Next: a quadrant's gate reads every quadrant
Same move as SwiGLU: let the gate read everything.
Value projection V not drawn. Full SwiGLU is SiLU(XWᵀ) ⊙ XVᵀ; here the value projection is folded into ŷ = XVᵀ, so the gate reads ŷ through its own W.
SiLU vs SwiGLU, as vectors
Ŷ in, P̂ out. SwiGLU adds one thing: a projection Wᵀ before the gate
SiLUŶσ(·)⊙P̂P̂ = Ŷ ⊙ σ(Ŷ) SwiGLUŶWᵀŶWᵀSiLU(·)⊙P̂P̂ = Ŷ ⊙ SiLU(ŶWᵀ)
The only difference: a projection Wᵀ before the gate, so each gate reads every unit of Ŷ
Attention keeps this pattern: projections feed the gate.
Max pooling is a gated sum
All four windows, each value to its own ×. What gates give the max?
?
x · e×3×4×4Conv 3×3ReLUmax pool 2global avg poolLinearŷ 0.20.90.10.40.50.30.80.60.70.10.30.20.40.60.90.5TLTRBLBRone channel,4 windowsTL0.2?00.9?10.5?00.3?0hardmax0.9Σout TLTR0.1?00.4?00.8?10.6?0hardmax0.8Σout TRBL0.7?10.1?00.4?00.6?0hardmax0.7Σout BLBR0.3?00.2?00.9?10.5?0hardmax0.9Σout BR16 connections, 4 independent hardmaxes0.20.20.90.90.50.50.30.30.10.10.40.40.80.80.60.60.70.70.10.10.40.40.60.60.30.30.20.20.90.90.50.5
NOW  Each window's four values come off the channel, one input each
Max pooling is a gated sum: gate 1 on each window’s max, 0 elsewhere.
Max pooling, vectorized
Same network, one vector operation per window: still four hardmaxes
x · e×3×4×4Conv 3×3ReLUmax pool 2global avg poolLinearŷ TLx0.20.90.50.3⊙ΣoTRx0.10.40.80.6⊙ΣoBLx0.70.10.40.6⊙ΣoBRx0.30.20.90.5⊙Σohardmax(·)a01000.9hardmax(·)a00100.8hardmax(·)a10000.7hardmax(·)a00100.9a = hardmax(x)for each window:o = Σ a ⊙ x
One vector rule per window: o = Σ a ⊙ x, with a = hardmax(x).
Hardmax → softmax pooling
Hardmax gets a gradient only to the winner. Can we soften the gate?
?
x · e×3×4×4Conv 3×3ReLUmax pool 2global avg poolLinearŷ
x · e×3×4×4Conv 3×3ReLUsoftmax pool 2global avg poolLinearŷ
TLx0.20.90.50.3⊙ΣoTRx0.10.40.80.6⊙ΣoBLx0.70.10.40.6⊙ΣoBRx0.30.20.90.5⊙Σohardmax(·)a01000.9hardmax(·)a00100.8hardmax(·)a10000.7hardmax(·)a00100.9a = hardmax(x)softmax(·/τ)a0.0010.9790.0180.0020.891softmax(·/τ)a0.0010.0160.8660.1170.770softmax(·/τ)a0.7040.0020.0350.2590.663softmax(·/τ)a0.0020.0010.9790.0180.891a = softmax(x/τ)for each window:o = Σ a ⊙ x
Softer gates, same problem. Can't see beyond the greater context.
Softmax pooling → attention
?
x · e×3×H×WConv 3×3ReLUsoftmax poolglobal avg poolLinearŷ
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· Vglobal avg poolLinearŷ
TL[1, 0, 0]×gate ?TR[1, 1, 0]×gate ?BL[1, 0, 0]×gate ?BR[1, 0, 0]×gate ?x = (blue, wings, hull)softmaxover the 4 scoresΣout TLs ?s ?s ?s ?WKk [0, 0]WKk [1, 0]WKk [0, 0]WKk [0, 0]·s 0·s 2·s 0·s 0query from TL onlyWQq [2, 2]a 0.096a 0.711a 0.096a 0.096[1, 0.71, 0]blue, wings → sky
Softmax pooling across all four quadrants. The gate needs a score per quadrant. From what?
Key k = x WKᵀ: what each quadrant has. Query q = xTL WQᵀ: what top-left looks for. Score s = q · k
Two projections of the input, multiplied together. Where have we seen that before?
SwiGLU's pattern: projections feed the gate. Here s = (xTLWQᵀ) · (xjWKᵀ): how similar query and key are
Softmax pooling with one change: the gate reads projections, q · k, not raw values.
WV not drawn. The gates are plain numbers, so Σj aj(WVxj) = WV(Σj ajxj): WV can come after the gated sum and merge into the next linear layer. Running example: WV = I.
Every quadrant queries every quadrant
Same network, one query at a time: TL, then TR, BL, BR
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· Vglobal avg poolLinearŷ TL[1, 0, 0]×WKk [0, 0]TR[1, 1, 0]×WKk [1, 0]BL[1, 0, 0]×WKk [0, 0]BR[1, 0, 0]×WKk [0, 0]WQ····softmaxΣx = (blue, wings, hull)WQ reads only blue, and every quadrant is blue: every query is q = [2, 2]query from TLq [2, 2]s 0s 2s 0s 0a 0.096a 0.711a 0.096a 0.096o TL [1, 0.71, 0]query from TRq [2, 2]s 0s 2s 0s 0a 0.096a 0.711a 0.096a 0.096o TR [1, 0.71, 0]query from BLq [2, 2]s 0s 2s 0s 0a 0.096a 0.711a 0.096a 0.096o BL [1, 0.71, 0]query from BRq [2, 2]s 0s 2s 0s 0a 0.096a 0.711a 0.096a 0.096o BR [1, 0.71, 0]
NOW  
Pooling had four local gates. Attention has four gates, each reading every quadrant.
Context, blended into every quadrant
Picture B. Draw each quadrant's (blue, wings, hull) as a picture: the boat's strength is the hull number
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· Vglobal avg poolLinearŷ
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
picture B
conv
→
[1,0,0][1,0,1][1,0,0][1,0,0]
x: each quadrant alone
attention
→
[1,0,0.71][1,0,0.71][1,0,0.71][1,0,0.71]
attention output
+ x
→
[2,0,0.71][2,0,1.71][2,0,0.71][2,0,0.71]
x + attention
Before attention each quadrant knows only itself. Top-left is just blue: [1, 0, 0].
Attention output: a shadow boat everywhere, and all four quadrants are now identical. The boat quadrant lost what it was.
Fix: add each quadrant back, x + attention. This skip is a residual connection; it stays in the network from here on.
Top-left: [1, 0, 0] → [2, 0, 0.71]. It knows it is water, not just blue, because of the boat. No extra layer needed.
This breaks the CNN assumption that only local information matters.
A local representation that depends on global context.
Last two pictures: boat opacity = hull ÷ 1.71, the largest hull value. Running example weights.
Same calculation, picture A
Now the bird. Draw each quadrant's (blue, wings, hull): the bird's strength is the wings number
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
picture A
conv
→
[1,0,0][1,1,0][1,0,0][1,0,0]
x: each quadrant alone
attention
→
[1,0.71,0][1,0.71,0][1,0.71,0][1,0.71,0]
attention output
+ x
→
[2,0.71,0][2,1.71,0][2,0.71,0][2,0.71,0]
x + attention
Same top-left input as picture B: [1, 0, 0]. Predict its output before the next click.
Same steps as the boat: a shadow bird everywhere, and the bird quadrant stays the strongest.
Top-left: [1, 0, 0] → [2, 0.71, 0]. Beside the boat it was [2, 0, 0.71]. Same input, now sky, not water.
The only difference is the other quadrants. The top-left's representation now carries its context.
Same patch, different context, different representation.
Last two pictures: bird opacity = wings ÷ 1.71, the largest wings value. Running example weights.
Projections and the score grid
Picture A. Which quadrant should the top-left one listen to?
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
quad
x
q = x WQᵀ
k = x WKᵀ
TL
[1, 0, 0]
[2, 2]
[0, 0]
TR
[1, 1, 0]
[2, 2]
[1, 0]
BL
[1, 0, 0]
[2, 2]
[0, 0]
BR
[1, 0, 0]
[2, 2]
[0, 0]
x = (blue, wings, hull) · WQ: “is there an object nearby?” (2 × blue) · WK: wings, hull
scores q · k
k TL
k TR
k BL
k BR
q TL
0
2
0
0
q TR
0
2
0
0
q BL
0
2
0
0
q BR
0
2
0
0
TL ROW, SOFTMAX
TL 0.096
TR 0.711
BL 0.096
BR 0.096
bird quadrant: 71%
NOW  Each quadrant's features x come from the conv's three channels
NOW  Query: q = x W_Qᵀ, what each quadrant looks for
NOW  Key: k = x W_Kᵀ, what each quadrant has
NOW  Score = q · k for every pair: a 4 × 4 grid
NOW  Softmax each row: the gates. The top-left listens to the bird.
Projections and the score grid
Picture A. Which quadrant should the top-left one listen to?
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
quad
x
q = x WQᵀ
k = x WKᵀ
TL
[1, 0, 0]
[2, 2]
[0, 0]
TR
[1, 1, 0]
[2, 2]
[1, 0]
BL
[1, 0, 0]
[2, 2]
[0, 0]
BR
[1, 0, 0]
[2, 2]
[0, 0]
x = (blue, wings, hull) · WQ: “is there an object nearby?” (2 × blue) · WK: wings, hull
scores q · k
k TL
k TR
k BL
k BR
q TL
0
2
0
0
q TR
0
2
0
0
q BL
0
2
0
0
q BR
0
2
0
0
TL ROW, SOFTMAX
TL 0.096
TR 0.711
BL 0.096
BR 0.096
bird quadrant: 71%
NOW  Each quadrant's features x come from the conv's three channels
NOW  Query: q = x W_Qᵀ, what each quadrant looks for
NOW  Key: k = x W_Kᵀ, what each quadrant has
NOW  Score = q · k for every pair: a 4 × 4 grid
NOW  Softmax each row: the gates. The top-left listens to the bird.
Projections and the score grid
Picture A. Which quadrant should the top-left one listen to?
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
quad
x
q = x WQᵀ
k = x WKᵀ
TL
[1, 0, 0]
[2, 2]
[0, 0]
TR
[1, 1, 0]
[2, 2]
[1, 0]
BL
[1, 0, 0]
[2, 2]
[0, 0]
BR
[1, 0, 0]
[2, 2]
[0, 0]
x = (blue, wings, hull) · WQ: “is there an object nearby?” (2 × blue) · WK: wings, hull
scores q · k
k TL
k TR
k BL
k BR
q TL
0
2
0
0
q TR
0
2
0
0
q BL
0
2
0
0
q BR
0
2
0
0
TL ROW, SOFTMAX
TL 0.096
TR 0.711
BL 0.096
BR 0.096
bird quadrant: 71%
NOW  Each quadrant's features x come from the conv's three channels
NOW  Query: q = x W_Qᵀ, what each quadrant looks for
NOW  Key: k = x W_Kᵀ, what each quadrant has
NOW  Score = q · k for every pair: a 4 × 4 grid
NOW  Softmax each row: the gates. The top-left listens to the bird.
Projections and the score grid
Picture A. Which quadrant should the top-left one listen to?
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
quad
x
q = x WQᵀ
k = x WKᵀ
TL
[1, 0, 0]
[2, 2]
[0, 0]
TR
[1, 1, 0]
[2, 2]
[1, 0]
BL
[1, 0, 0]
[2, 2]
[0, 0]
BR
[1, 0, 0]
[2, 2]
[0, 0]
x = (blue, wings, hull) · WQ: “is there an object nearby?” (2 × blue) · WK: wings, hull
scores q · k
k TL
k TR
k BL
k BR
q TL
0
2
0
0
q TR
0
2
0
0
q BL
0
2
0
0
q BR
0
2
0
0
TL ROW, SOFTMAX
TL 0.096
TR 0.711
BL 0.096
BR 0.096
bird quadrant: 71%
NOW  Each quadrant's features x come from the conv's three channels
NOW  Query: q = x W_Qᵀ, what each quadrant looks for
NOW  Key: k = x W_Kᵀ, what each quadrant has
NOW  Score = q · k for every pair: a 4 × 4 grid
NOW  Softmax each row: the gates. The top-left listens to the bird.
Projections and the score grid
Picture A. Which quadrant should the top-left one listen to?
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
quad
x
q = x WQᵀ
k = x WKᵀ
TL
[1, 0, 0]
[2, 2]
[0, 0]
TR
[1, 1, 0]
[2, 2]
[1, 0]
BL
[1, 0, 0]
[2, 2]
[0, 0]
BR
[1, 0, 0]
[2, 2]
[0, 0]
x = (blue, wings, hull) · WQ: “is there an object nearby?” (2 × blue) · WK: wings, hull
scores q · k
k TL
k TR
k BL
k BR
q TL
0
2
0
0
q TR
0
2
0
0
q BL
0
2
0
0
q BR
0
2
0
0
TL ROW, SOFTMAX
TL 0.096
TR 0.711
BL 0.096
BR 0.096
bird quadrant: 71%
NOW  Each quadrant's features x come from the conv's three channels
NOW  Query: q = x W_Qᵀ, what each quadrant looks for
NOW  Key: k = x W_Kᵀ, what each quadrant has
NOW  Score = q · k for every pair: a 4 × 4 grid
NOW  Softmax each row: the gates. The top-left listens to the bird.
Attention output and query, key, value
Mix what each quadrant hands over, weighted by the top-left row of gates
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
quad
gate
v (blue, wings, hull)
gate × v
TL
0.096
[1, 0, 0]
[0.10, 0.00, 0.00]
TR
0.711
[1, 1, 0]
[0.71, 0.71, 0.00]
BL
0.096
[1, 0, 0]
[0.10, 0.00, 0.00]
BR
0.096
[1, 0, 0]
[0.10, 0.00, 0.00]
TL output = sum
[1.00, 0.71, 0.00]
blue, with wings nearby → sky
A
B
B · TL OUTPUT
[1.00, 0.00, 0.71]
blue + hull → water
NAME THE PARTS
looks for → query Q
has → key K
hands over → value V
output = softmax(QKᵀ) V
NOW  The top-left row of gates, from the score grid
The usual attention diagram
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual XLinear WQLinear WKLinear WVQKVMatMul QKᵀSoftMaxMatMul ·V
BEFORE THE NEXT CLICK
This is the diagram in every paper. Which of our parts is each box?
NAME THE PARTS
Q = XWQᵀ
query: what each patch looks for
K = XWKᵀ
key: what each patch has
V = XWVᵀ
value: what each patch hands over
MatMul QKᵀ
Every query against every key: the 4 × 4 grid of similarities.
SoftMax
One gate per row: softmax pooling, but over every patch.
MatMul ·V
Each row's gated sum of values. WV is the projection we left out of our diagrams.
Every box is a part we built: softmax pooling, with projections feeding the gate.
Redrawn after Vaswani et al. (2017), Fig. 2. Scale ÷√d added next. Its optional mask is for decoders; an encoder does not use one.
Variance through attention
Assume every input x has variance σ². Follow it through the usual diagram.
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
x · e×3×H×WConv 3×3ReLUQKVQKᵀ× 1/√dsoftmax· V+global avg poolLinearŷresidual
XLinear WQLinear WKLinear WVQKVMatMul QKᵀScale ÷√dSoftMaxMatMul ·VVar σ²w: Var 1/fan_inVar σ² eachVar d·σ²·σ² = d·σ⁴gates → hardmaxgates stay soft÷√d: Var σ⁴ = 1
QUESTION
x has variance σ². What variance should each weight have so the projection keeps σ²?
QUESTION
With that init, what is the variance of each element of Q, K, V?
QUESTION
What is the variance of each score after MatMul QKᵀ?
QUESTION
What comes out of the softmax as d grows?
QUESTION
σ⁴ = 1 when σ² = 1, but d keeps growing. How would you fix it?
ANSWER
Var(w) = 1/fan_in, std 1/√fan_in: LeCun (fan-in) init. Each output sums fan_in terms w·x.
ANSWER
Var(qi) = fan_in · (1/fan_in) · σ² = σ². Same for K and V.
ANSWER
d products of σ²·σ², added: Var(s) = d·σ⁴. σ⁴ is fine while σ² = 1; d is not.
ANSWER
High variance forces the gate to act like a hardmax: top gate → 1, the rest → 0.
σ² = 1, largest of 64 gates
d
1
4
16
64
256
top gate
0.14
0.29
0.56
0.79
0.90
÷√d
0.14
0.11
0.10
0.11
0.11
BACK TO LECUN (FAN-IN) INIT
  • LeCun init fixed a sum of fan_in terms with 1/√fan_in
  • Same move here: divide the scores by √d
  • Var(s/√d) = σ⁴: 1 when σ² = 1, whatever d is
  • Toy model, d = 256: accuracy 1.00 with it, 0.51 without
softmax(QKᵀ/√d) V
LeCun init's move again: ÷√d keeps the scores' variance from growing with d.
Two heads, two questions
One score grid asks one question. What if the picture needs two?
?
x · e×3×H×WConv 3×3ReLUQ1K1V1QKᵀ/√dsoftmax· VQ2K2V2QKᵀ/√dsoftmax· Vconcat× WO+global avg poolLinearŷresidual
HEAD 1: WINGS NEARBY?
HEAD 2: HULL NEARBY?
CONCAT 4×2
× WO
= 4 × 1
FOLD BACK
A
sky
B
water
Each head: its own WQ, WK, WV, one 2 × 2 map. Flatten each map to one row per quadrant, stack the heads side by side, mix them with WO.
Each head here has d = 1 (q and k are single numbers), so ÷√d = ÷1. Picture A, head 1: scores [0, 2, 0, 0], softmax puts 0.71 on the bird quadrant.
Multi-head attention: each head asks its own question; WO mixes the answers.
Output projection
Merge the heads' channels back into one model-width map, so it can be added to the input.
WHAT IS FIXED
Q, K and V projections, and the output projection: learned weights, the same for every picture.
The output projection is a 1 × 1 conv across the stacked channels.
WHAT IS COMPUTED
The gates: softmax(QKᵀ/√d). They come from the picture, fresh every time.
A conv kernel is learned once. A head's mixing weights are built on the fly.
MODEL SO FAR
Q, K, V convs × heads
softmax(QKᵀ/√d) V
concat channels
1×1 conv
global avg pool
linear
Fixed projections, gates computed from the input.
Shuffle test
The model was trained to read this block as “Ra”. Swap the two left signs: what does it say?
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀsoftmax· V+global avg poolLinearŷresidual
PICTURE
ATTENTION OUTPUT
AVG POOL → ŷ
mouth over arm: “Ra”
mouth + ctx
sun + ctx
arm + ctx
stroke + ctx
ŷ = “Ra” ✓
trained on this
arm over mouth: not Ra
ŷ = ?
ŷ = “Ra” ✗
same average, same answer
WHY IT CAN'T TELL
  • The swap moves outputs, not their values
  • Average of the same 4 vectors: identical
  • Attention's mixing has no position information
In text: “dog bites man” vs “man bites dog”
Attention mixes by content: position plays no part.
Ra is spelled r (mouth) then a throaty a (forearm), then the sun-disk sign and stroke. Signs drawn simplified.
Position in CNNs
?
CNNs drop absolute position, but only at the very end
x · e×3×4×4Conv 3×3ReLUmax pool 2global avg poolLinearŷ
CONV 3×3
  • Sees left, right, above, below
  • Outputs stay on their grid
MAX POOL
  • Mixes locally
  • Place decides what mixes
GLOBAL AVG POOL
  • Drops absolute position
  • “Ra” anywhere is “Ra”
Mouth over arm spells “Ra”. Does a CNN know the mouth is above the arm?
Mouth over arm spells “Ra”: the middle needs layout, the label doesn't
CNNs keep relative arrangement until the very last step.
Experiment: relative position
Is the square left of the circle? Predict each row.
?
yes
no
MODEL
RELATIVE POSITION
TEST ACCURACY
CNN, pooling at the end
kept
1.00
CNN, pooling early
thrown away early
0.51
Conv stem + attention, no position
never had it
0.58
40 × 40 images · 8,000 train, 2,000 test · mean of 3 seeds · chance 0.50
Attention = a position-blind global mix at layer one
QUESTION
How might we give attention position?
Throw away arrangement too early and “left of” cannot be learned.
Conv stem = two 3×3 conv layers and a 4×4 patch conv before the attention blocks (exp_position.py).
Absolute vs relative position
Two ways to tell attention where a patch is
x · e×3×H×WConv 3×3ReLUQKVQKᵀ/√d+ offset biassoftmax· V+global avg poolLinearŷresidual
ABSOLUTE · ORIGINAL ViT
0,0
0,1
0,2
0,3
1,0
1,1
1,2
1,3
2,0
2,1
2,2
2,3
3,0
3,1
3,2
3,3
  • A learned vector added to each patch: an address
  • Our toy, 10 epochs: 0.51 to 1.00 across seeds. Shaky.
RELATIVE · WHAT WE WANT
  • Score sees the offset: “2 patches right”
  • What a conv kernel always had
OFFSET BIAS · WHAT WE USE
sij + b[Δrow, Δcol]
One learned number per offset, per head, added to the score. Now attention knows “left of” and “above”.
Give attention offsets, not addresses: one learned bias per offset.
Rotary position encoding (RoPE)
Each pair of query or key numbers is an arrow: turn it by its position × θ
x · e×3×H×WConv 3×3ReLUQKVRoPE on Q, KQKᵀ/√dsoftmax· V+global avg poolLinearŷresidual querykey
POSITIONS 2 AND 5
q at 2: turn 2θ → [0.54, 0.84]
k at 5: turn 5θ → [-0.80, 0.60]
q · k = 0.071
BOTH SHIFTED BY 3: 5 AND 8
q at 5: turn 5θ → [-0.80, 0.60]
k at 8: turn 8θ → [-0.65, -0.76]
q · k = 0.071
WHY
  • The score sees the angle between: (5 − 2)θ = (8 − 5)θ
  • Same offset, same score
  • Only Q and K turn
  • 2D: half the channels by column, half by row
NOW  A query and a key, before turning
RoPE in 2D
Split the query and key channels in half. One half turns by column, the other by row.
SCORE
q · k = column-offset term
+ row-offset term
A dot product is a sum over channels, so the two halves just add.
WHERE IT GOES
On queries and keys only. Never on values, never on the residual stream.
Which channels get which axis doesn't matter: the projections learn to fill them.
Caveat: no single pair turns with both row and column, so diagonals aren't direct. Later variants learn mixed directions.
MODEL SO FAR
conv
Q, K, V convs
RoPE on Q, K
softmax(QKᵀ/√d) V
1×1 conv
global avg pool
linear
2D RoPE: column offset plus row offset.
Offset bias: 1.00 on every seed
Same square-and-circle task, same 10 epochs, 3 seeds
MODEL · 10 EPOCHS
EACH SEED
MEAN
Conv stem + attention, no position
0.58 · 0.57 · 0.59
0.58
+ learned position (absolute)
0.81 · 0.51 · 1.00
0.77
+ offset bias, default lr
0.59 · 0.58 · 0.58
0.58
+ offset bias, 10× lr on the bias table
1.00 · 1.00 · 1.00
1.00
QUESTION
Why did the default learning rate fail?
Next slide: how to spot a stalled parameter.
MODERN MODELS
RoPE rotates q and k so the score depends on the offset. Same idea, built in. (1.00 here too.)
A learned offset bias gets back what convolution kept for free.
At 30 epochs, learned position and the default-lr offset bias also reach 1.00 on all 3 seeds (no position: 0.59). Conv stem = two 3×3 convs and a 4×4 patch conv (exp_position.py).
A stalled parameter: how to spot it
Symptom: offset bias at default lr 0.58, no position at all 0.58. The bias adds nothing.
012345670100200300400500600training step (10 epochs)largest |bias| entryreach 0.63default 0.49needed ≈ 4.6reach 6.310× lr 5.6
CHECK 1 · GRADIENT
Is the bias even learning?
CHECK 1 · GRADIENT
‖∇b‖ from 10−6 to 10−3: never zero. It is learning, just slowly.
CHECK 2 · REACH
How far can Adam move one entry in 630 steps?
CHECK 2 · REACH
Adam steps ≈ lr, whatever the gradient size (Quiz 3). Reach = Σ lrt = 0.63; measured max 0.49.
CHECK 3 · NEEDED SIZE
How big must a bias be to pick 1 patch of 100?
CHECK 3 · NEEDED SIZE
Score gap ≈ ln 100 ≈ 4.6. A bias of 0.49 gives only e0.49 ≈ 1.6× the odds.
FIXES
  • 10× lr for the bias group (reach 6.3) → 1.00
  • Train 3× longer (30 epochs) → 1.00
  • Or rescale the parameter: b = 10·b′
Adam([{"params": rest}, {"params": bias, "lr": 10*lr}])
If a parameter must grow big, check it can: Adam's reach ≈ lr × steps.
Reach is approximate (one Adam step briefly hit 3.7× lr). In practice: log max |w| and ‖Δw‖/‖w‖ per parameter group (diag_bias.py).
Why bother?
We built a lot of machinery to get back what a CNN already had
?
x · e×3×H×WConv 3×3ReLUQKVQKᵀ× 1/√dsoftmax· V+global avg poolLinearŷresidual
FAIR QUESTION
On the square-and-circle task, the CNN scores 1.00 too. So why replace it?
1Context at layer one. Every patch reads every patch in one step: sky or water, settled.
2Input-dependent mixing. Gates computed from each picture; a conv kernel is fixed.
3One shape for any sequence. Patches today, words next: same block.
4The catch: it needs data. Fewer built-in assumptions, more to learn.
Attention trades built-in assumptions for flexibility.
Experiment: number of heads
Square left of circle? Triangle above circle? Both at once: 4 classes
x · e×1×40×40Conv 3×3ReLUConv 3×3ReLUpatch conv 4×4encoder block · h heads · RoPEglobal avg poolLinearŷ
TEST ACCURACY · MEAN OF 2 SEEDS
0.521 head1.002 heads1.004 heads1.008 headschance 0.25
SETUP
  • Conv stem + attention, one block
  • Position: RoPE, offset-based
  • Each head: width 4
  • Two relations to track at once
1 head: 0.52 (seeds 0.43 / 0.61); 2+ heads: 1.00. Head width fixed at 4, so more heads = more width too.
The Vision Transformer (ViT)
Dosovitskiy et al., 2021 · ViT-Base/16
1 · PATCH CUT
conv 16×16, stride 16
  • 224² image → 196 tokens of 768
  • In place of our conv features
2 · POSITION
  • A learned embedding, 196 × 768
  • Added once, at the input
3 · BLOCK × 12
  • Our attention + residual
  • Plus LayerNorm and an MLP
  • Stacked 12 deep
4 · OUTPUT
  • Average the 196 tokens → Linear → ŷ, as we did
Newer models use RoPE for position instead of a learned embedding.
Inside one encoder block
Attention: we built it
q, k, v · ×1/√d · softmax ·
12 heads · Linear out
Residual adds: built
MLP: new
768 → 3072 → 768
GELU, per token
LayerNorm: new
LayerNorm: new · rescales each token
Convolution is position-only attention
Look back at what we built: one restriction removed
CONVOLUTION
●
  • Weights from offset only
  • Neighbours only
ATTENTION + RoPE
●
  • Weights from offset and content
  • Any patch
Multi-head attention with relative position can express any convolution layer: one head per kernel offset (Cordonnier et al., 2020).
Convolution is position-only attention; a transformer adds content.
Residual, norm, feed-forward
Attention, heads and position were the new parts. The rest is CNN parts.
x · e×3×H×WConv 3×3ReLUpatch conv[ norm → attention → + → norm → feed-forward → + ] × Nglobal avg poolLinearŷ
Residual add
= ResNet skip connection
Norm
= batch norm's idea, per position
Feed-forward net
= two 1 × 1 convs, or a SwiGLU
Stack it
= 6 blocks (2017) · 12 (ViT-Base, BERT-Base)
Nothing in the encoder block is new except attention.
Hybrid ViT
You've already built this · original Vision Transformer paper (Dosovitskiy et al., 2021)
x · e×3×H×WConv 3×3ReLUpatch convencoder block × Nglobal avg poolLinearŷ
OURS
  • One conv layer
  • Four quadrants
  • Encoder blocks on top
THE PAPER'S
  • ResNet front end
  • Many patches
  • Encoder blocks on top
A CNN front end plus encoder blocks is the hybrid ViT.
Class token as attention pooling
The CNN's last step, global average pooling, becomes attention too.
GLOBAL AVERAGE POOLING
weights: 1/N each
Attention where every gate is equal. Ignores content.
CLASS TOKEN
weights: softmax(q_cls Kᵀ/√d)
A learned query added as one extra token. It attends over every patch; its output goes to the linear head.
MODEL SO FAR
conv front end
patches
+ class token
encoder block × N
global avg pool
linear on class token
The class token is attention pooling over the whole picture.
Pure ViT: removing the conv front end
One deletion from the hybrid. Everything after it stays the same.
x · e×3×H×WConv 3×3ReLUpatch conv 16×16, stride 16encoder block × Nglobal avg poolLinearŷ
THE PATCH PROJECTION
  • Raw pixels → 16 × 16 patches
  • One shared projection:
conv, kernel 16 × 16, stride 16
  • No nonlinearity, no overlap
WHAT IS LOST
  • Stacked local filters
  • Even neighbours must be learned through attention
Pure ViT = hybrid minus the convolutions.
Attention on a 1D image
One long row of pixels, cut into 768-pixel windows; attention runs over the windows
………………x · 1 × 4608 pixelscut into windows1 × 4608 → 6 × 768each window → its own row………………6 × 768WLinear 768 × 768same W on every row= 1D conv, kernel 768, stride 7686 tokensEncoderBlock × 12attentionMLPevery token reads every tokenout: 6 × 768LayerNormaverage overwindows768Linear1000 × 768logits1000
A 1D image is a row of 768-number tokens, one per window; attention mixes the windows.
768 = 16 × 16 × 3, a ViT-Base patch; ViT-Base and BERT-Base are 768 wide. Illustrative: 6 windows; only a few of each window's 768 pixels drawn.
One learned segment per word
What if each 768-pixel window described a word? No camera for words: where do its pixels come from?
?
now: [MASK] row → Linear 768 → 30,000
………………thecatsatonthematcut into windows1 × 4608 → 6 × 768each window → its own row………………6 × 768WLinear 768 × 768same W on every row= 1D conv, kernel 768, stride 7686 tokensEncoderBlock × 12attentionMLPevery token reads every tokenout: 6 × 768LayerNormaverage overwindows768Linear1000 × 768logits1000…[MASK]…6 × 768masked-word predictionone row in, one word outLinear768 → 30,000softmaxsat?30,000 probstarget: satgrey rows: not used for the loss
Each word gets its own segment; the model learns what to paint in it. For text, the patches are learned.Hide a word, predict it from the others: that's how BERT learns, on raw text.
768 = 16 × 16 × 3, a ViT-Base patch; ViT-Base and BERT-Base are 768 wide. Illustrative: 6 windows; only a few of each window's 768 pixels drawn.
Embeddings
Same network, new front end: each word's ID looks up its own row of a learned table
now: [MASK] row → Linear 768 → 30,000
now: embedding lookup · E 30,000 × 768
…the…cat…[MASK]…on…the…matcut into windows1 × 4608 → 6 × 768each window → its own row………………6 × 768WLinear 768 × 768same W on every row= 1D conv, kernel 768, stride 7686 tokensEncoderBlock × 12attentionMLPevery token reads every tokenout: 6 × 768LayerNorm6 × 768masked-word predictionone row in, one word outLinear768 → 30,000softmaxsat?30,000 probstarget: satgrey rows: not used for the loss⋮…103⋮…1996⋮…2006⋮…4937⋮…13523⋮thecat[MASK]onthematembedding E · 30,000 × 768row = word's ID; both “the” → row 199600…1…0one-hot · 30,000×E30,000 × 768=+0.12−0.40…+0.07row 103 ([MASK]) · 768a lookup:multiply by one-hot= copy one row
WHERE DO THE ROWS COME FROM?
Random at first, learned in pretraining
No pre-made word2vec / GloVe vectors
Position (512 × 768): learned too
WHO PICKS THE PIECES?
WordPiece tokenizer, built first from piece counts: not learned
playing → play + ##ing, one row of E each
An embedding is a learned patch, one per word piece. The tokenizer picks the windows; training paints them.
BERT-Base: 30,000-piece WordPiece vocabulary, 768 wide. Illustrative: word IDs and row values; only a few of E's rows and of each row's 768 numbers drawn.
BERT and its pretraining
?
original BERT:
post-norm
LN(x + Attn(x))
ours / ModernBERT:
pre-norm
x + Attn(LN(x))
QUESTION
ModernBERT (2024) went pre‑norm. Why?
Post-norm puts a LayerNorm on every skip path, so the gradient to early layers crosses them all; pre-norm keeps a clean skip path: deep stacks train stably.
Warner et al. 2024; Xiong et al. 2020
11WHY REUSE E AT THE OUTPUT?
Input gradient is sparse: a word's row learns only when it appears. The output softmax touches every row, every step. Tying gives every row that signal.
But the jobs differ (read vs. predict). Many large models untie (more data; dense signal not needed). Tying saves ≈23 M parameters here.
BERT = our model, on text, pretrained.
BERT-Base, Devlin et al. 2019: ≈ 110 M parameters. Pretrained on raw text: fill in hidden words (15%). Tying saves ≈ 23 M.
ImageNet accuracy by pretraining data
ImageNet top-1 accuracy (%) after fine-tuning
PRETRAINED ON
CNN (BiT ResNet)
VIT
ImageNet
1.3 M images
81.3
77.9
ViT-B/16
ImageNet-21k
14 M images
85.4
85.2
ViT-L/16
JFT-300M
300 M images
87.5
88.0
ViT-H/14
Small data: the CNN leads. More data: ViT catches up, then edges ahead.
ViT: Dosovitskiy et al., 2021 (ViT paper), App. Table 5, best ViT size per row. CNN: BiT ResNet152x4, Kolesnikov et al., 2020 (BiT paper), Tables 1–2.
Multi-modality: one token language
Our ViT turns a picture into one 768-number token; BERT turns each word into one
?
photo Aphoto Bthecat[MASK]onthematword tokens(BERT's lookup)our ViTpatch conv 16×16196 × 768EncoderBlock × 12196 × 768average over tokens1 × 768L[image]learned Linearimage token → text spaceour ViT: one 768-number tokenper pictureone sequence · 7 tokens · 768 eachEncoderBlock × 12[MASK] reads the image tokenLinear768 → ≈30ksatwith photo Astoodwith photo Bsoftmax over ≈30k (30,522) wordsQUESTIONText alone can't tell:sat or stood?HOW IT'S LEARNEDTrained together on pictures with captionsby hiding words, as before:the only way to guess rightis to read the imageANYTHING YOU CAN CUT INTO TOKENStextthecatsatonthematwordsimage16×16 patchesaudioshort sound slicesvideopatches × framesEncoderBlock × None model shapeAnything cut into tokens → one model shape → transformers everywhere → foundation models
Same format makes it possible; training together makes it a shared language.
Early examples (2019): VisualBERT, VL-BERT, ViLBERT. Real systems pass many image tokens (one per patch); our single averaged token keeps the idea.
What the picture adds
Our trained model fills a hidden word, first from the text alone, then with the photo
?
Validation photo: a boy in a Hawaiian shirt and denim shorts jumping in a grassy back yard, a wooden fence behind him and the siding of a house above the fence.
“a child in hawaiian clothing jumps and poses in a low cut yard nearby a fence and [MASK] .”
What fills the blank?
No photo
text alone: top 5 guesses
1treesnot in the photo
2waternot in the photo
3grassin the photo
4watchnot in the photo
5snownot in the photo
With photo
one image token + text: top 5 guesses
1grassin the photo
2treesnot in the photo
3fencein the photo
4fieldin the photo
5it
The answer was building (the house behind). Neither got it, but with the photo, water, snow and watch dropped out and fence and field came in.
One image token is enough to steer the guesses toward the scene.
Our model: pretrained ViT-B/16 → one image token → 6-block BERT, fine-tuned on Flickr30k. One validation example.
What about RNNs?
RNN CELL (TORCH CANVAS), ONE WORD PER STEP
press Play
ht needs ht−1: 6 steps in a row
TRANSFORMER, ONE ATTENTION LAYER
press Play
all 6 positions at once: one step
loss at all 6 positions in one pass
Sequential → can’t use a GPU’s thousands of cores across the sequence
Parallel → training on billions of words becomes feasible
Parallel training is what made massive pretraining — and foundation models — practical.
RNNs (e.g. LSTMs) were the standard sequence models before 2017; this course skips them.
Foundation models
Three ingredients, all needed at once
1
Extreme data
  • Images: ~300 M (JFT-300M, the original ViT)
  • Text: billions of words (BERT); no labels needed, so nearly unlimited
2
Just enough flexibility
CNN
locality + weight sharing
little data: great
at scale: blinders
Transformer
attention: global, input-dependent mixing
weights shared across positions
sweet spot for extreme data
MLP
everything wired to everything
nothing shared
needs even more data and parameters
3
A task that forces the relationships
Hide a word, predict it
BERT: must read context
(images: masked patches)
Captioned pictures, hide words
must read the image (multi-modality)
The task is
the teacher
Extreme data × just-flexible-enough model × the right task: pretrain once, adapt everywhere.
Adapting a pretrained model: Project 3 and Tue, Nov 17.
Cost of attention
Every patch scores every patch. How does the score grid grow?
?
INPUT
PATCHES / WORDS N
SCORES N²
our 4 quadrants
4
16
ViT-Base, 224 × 224 image
196
38,416
512 words
512
262,144
4,096 words
4,096
16,777,216
PER HEAD, PER LAYER
Double the input, four times the scores.
A conv layer's cost grows only linearly.
Global gates cost N² scores.
Takeaways
  • Attention: projections decide the gates and mixture based on context
  • Summation needed a variance correction again ÷√d
  • Heads ask several questions; concat × WO mixes them
  • Attention ignores position: add it back with a learned offset bias (RoPE in modern models)
  • ViT = patches + EncoderBlock (pre-norm, residual); BERT = same block on words, MLM
  • Cut anything into tokens: any modality joins in
  • Trains in parallel, unlike RNNs → foundation models
Next: what the encoder hands the decoder.
Quiz details
Thursday, Nov 12: 25 minutes, equation sheet provided
ACTIVATIONS AND DROPOUT
  • ReLU, SiLU and SwiGLU as block diagrams
  • Dropout in train() vs eval()
  • Why kept units are scaled by 1/(1 − p)
SCHEDULES AND OPTIMIZERS
  • Step learning-rate schedules
  • Adam vs SGD, and momentum
  • Bonus: Adam's bias correction
Noah runs the quiz, then Daniel lectures on decoders.