Max pooling is a gate

Variance through convs and pools, and the first step from pooling toward attention

01

The rest of the semester

02

Variance through convs and pools

03

Max pooling is a gated activation

04

Soft and learned gates: SoftPool, attention pooling

05

Fixed vs. input-dependent mixing

All slides, one per page

The rest of the semester

Thu Oct 29

Today: max pooling is a gate

P2, P3 out

Tue Nov 3

No class (Election Day)

Thu Nov 5

Quiz 2

P2 groups set

Tue Nov 10

Transformer encoder: attention

P3 proposal

Thu Nov 12

Quiz 3 (25 min, run by Noah), then decoders and decoder-only models (Daniel)

Tue Nov 17

Foundation models, meta-learning, fine-tuning, PEFT

Thu Nov 19

Project 2 presentations

P2 due

Tue Nov 24

Quiz 4

Thu Nov 26

No class (Thanksgiving)

Tue Dec 1

Book discussion

Thu Dec 3

Computational processes built from neural parts

Tue Dec 8

Final, 3:30

P3 due

QUIZ 2 · THU NOV 5
The full hour. Covers residual connections and gradient flow, normalization, shapes in a residual MLP, and softmax. An equation sheet is provided.

Project 2 · built-in symmetry. Three groups. Choose a network built for a symmetry, implement it, train it on the HPC against a baseline, present it in 18 minutes.
Oct 29 → Nov 19

Project 3 · specialist vs. generalist. Design a network for your problem and pit it against an adapted foundation model.
Proposal Nov 10 · due Tue Dec 8, 3:30

Quiz 3 · Thu Nov 12. 25 minutes on activation functions, dropout, learning-rate schedules and Adam, then Daniel’s lecture.

Project 2: Built-in symmetry

Take a network built for a symmetry of its data, build it, and show what the symmetry buys

1

Form a group and choose a network within a week: at most 3 people, and at most two 5000-level students in a group. For example, two 5000-level students and one 4000-level student is fine.

2

Build it in PyTorch (libraries are fine) and draw it in Torch Canvas style.

3

Train it and a baseline on the HPC. Show learning curves, data efficiency and a symmetry test.

4

Publish a pip-installable repo with both models’ weights and a small test sample.

5

Present: 18 minutes, strictly, then 7 minutes of round-robin questions. Everyone presents and everyone answers.

GRADING
Only the presentation is graded, but the models must load and run. We install your repo with pip and run one short script; untested code that doesn't run gets no points.

5000-level: a group with a 5000-level student picks from Going beyond.
4000-level: working on a Going beyond network earns +15 extra credit on the project.

Spend your minutes on what's new; give what we already know one sentence.

Project 2: the networks

Each group takes a different one; first to tell me gets it

Standard

DeepSets

Invariant to the order of a set

PointNet

Invariant to the order of points in a point cloud

GIN

Invariant to relabeling a graph’s nodes

LSTM or GRU

Roughly equivariant to a shift in time

Going beyond harder, with ideas we haven't covered

Set Transformer

Invariant to the order of a set

EGNN

Equivariant to rotation, translation and reflection

SchNet

Invariant energy under rotation, translation and atom indexing

Hamiltonian NN

Conserves energy

G-CNN

Equivariant to 90° rotations and reflections

Spherical CNN

Equivariant to 3D rotations on a sphere

Same lesson as the CNN: name the symmetry, build it into the weights.

Project 3: A specialist vs. a generalist

Design a model for your own problem, then pit it against an adapted foundation model

1 · PROPOSE, WITHIN 12 DAYS

One paragraph: the problem and its data source. Email it to Jtroberts@tntech.edu, cc ntherron42@tntech.edu; we reply with feedback.

An ongoing project can be acceptable.

2 · BUILD THE SPECIALIST

Design it for the data, task and invariances, informed by the literature. Train it on the HPC with intentional augmentation, regularization, learning-rate schedule and early stopping.

3 · COMPARE, LAST

Adapt the most appropriate foundation model for your domain with full fine-tuning or PEFT, and compare fairly.

We cover foundation models and adaptation on Nov 17, so this comes last.

Why now? Finding a problem and data takes time, and HPC runs take longer. Starting today gives you six weeks.
You don't know foundation models yet, and that's fine. Nothing before step 3 needs them, and we'll cover them before you get there.

You don't have to win. How you design and train is what's graded.

Variance through a conv layer: forward

p̂ always means the output of an activation function. Boxed: p̂ = ReLU(GroupNorm(x)), then a 3 × 3 conv.

1

GroupNorm sets the scale. Its output x̂ has mean 0 and variance 1, so its second moment is also 1.

Var = 1.00

TUESDAY'S CNN · BLOCK 1

σ²x any

σ²x̂ = 1, E[x̂²] = 1

The more terms you sum, the more variance: fan-in = C_in · k².

Variance through a conv layer: forward

p̂ always means the output of an activation function. Boxed: p̂ = ReLU(GroupNorm(x)), then a 3 × 3 conv.

1

GroupNorm sets the scale. Its output x̂ has mean 0 and variance 1, so its second moment is also 1.

Var = 1.00

2

ReLU cuts the second moment in half: E[p̂²] = ½. Its mean is no longer 0, so, as we have seen before, its second moment is not its variance.

E[p̂²] = 0.50
mean 0.40, Var 0.34

TUESDAY'S CNN · BLOCK 1

σ²x any

σ²x̂ = 1, E[x̂²] = 1

E[p̂²] = .5, σ²p̂ = .34

The more terms you sum, the more variance: fan-in = C_in · k².

Variance through a conv layer: forward

p̂ always means the output of an activation function. Boxed: p̂ = ReLU(GroupNorm(x)), then a 3 × 3 conv.

1

GroupNorm sets the scale. Its output x̂ has mean 0 and variance 1, so its second moment is also 1.

Var = 1.00

2

ReLU cuts the second moment in half: E[p̂²] = ½. Its mean is no longer 0, so, as we have seen before, its second moment is not its variance.

E[p̂²] = 0.50
mean 0.40, Var 0.34

3

Multiply: The weights have mean 0, so their second moment is entirely variance: E[w²] = σ_W². Each term w · p̂ (convolutions · p̂) has variance E[w²] · E[p̂²].

E[w²] = 0.0070

TUESDAY'S CNN · BLOCK 1

σ²x any

σ²x̂ = 1, E[x̂²] = 1

E[p̂²] = .5, σ²p̂ = .34

The more terms you sum, the more variance: fan-in = C_in · k².

Variance through a conv layer: forward

p̂ always means the output of an activation function. Boxed: p̂ = ReLU(GroupNorm(x)), then a 3 × 3 conv.

1

GroupNorm sets the scale. Its output x̂ has mean 0 and variance 1, so its second moment is also 1.

Var = 1.00

2

ReLU cuts the second moment in half: E[p̂²] = ½. Its mean is no longer 0, so, as we have seen before, its second moment is not its variance.

E[p̂²] = 0.50
mean 0.40, Var 0.34

3

Multiply: The weights have mean 0, so their second moment is entirely variance: E[w²] = σ_W². Each term w · p̂ (convolutions · p̂) has variance E[w²] · E[p̂²].

E[w²] = 0.0070

4

Sum: ŷ adds up fan-in such terms, and the variance scales by the number summed, just as in a matrix multiply. Fan-in = C_in · k² = 32 · 9 = 288. Choose σW² to be 2/288

σ²ŷ = k²Cin · E[w²] · E[p̂²] = 288 · (2/288) · ½ = 1

measured: 0.97

TUESDAY'S CNN · BLOCK 1

σ²x any

σ²x̂ = 1, E[x̂²] = 1

E[p̂²] = .5, σ²p̂ = .34

σ²ŷ = .97

The more terms you sum, the more variance: fan-in = C_in · k².

Padding shrinks the variance at the edges

σ²_ŷ at each of the 14 × 14 positions, with the ring of padding zeros around it. Slide the same 3 × 3 filter to two places.

σ²ŷ 0.4
2.1white = 1

IN THE MIDDLE

All 9 filter weights land on real inputs, in each of the 32 channels.

σ²ŷ = Cin · k² · σW² · E[p̂²]
    = 32 · 9 · (2/288) · ½ = 1

measured 1.00

ON THE CORNER

5 of the 9 filter weights land on padding zeros, and a zero adds nothing to the sum. Only (k − 1)² = k² − 2k + 1 = 4 real terms remain.

σ²ŷ = Cin · (k² − 2k + 1) · σW² · E[p̂²]
    = 32 · 4 · (2/288) · ½ = 4/9 ≈ 0.44

measured 0.44

Along an edge, 3 weights land on zeros: k² − k = 6 real terms, σ²_ŷ = 6/9 ≈ 0.67 (measured 0.66).

You could correct it: scale each output by (k²/N)^1/2, with N the real inputs under the filter. At a corner, ×1.5.

Fewer real terms in the sum, less variance. PyTorch doesn't correct for it.

Convolution in 1D

Slide the filter, multiply what it covers, sum. Nothing about that needs two dimensions.

Convolution in 1D

Slide the filter, multiply what it covers, sum. Nothing about that needs two dimensions.

2-D: a 3 × 3 filter on an image

At the boxed spot, each of the 3 filter rows adds (−1)·0 + 0·0 + 1·1 = 1, so 9 products sum to 3.

ŷij = Σa,b wab p̂i+a, j+b

Convolution in 1D

Slide the filter, multiply what it covers, sum. Nothing about that needs two dimensions.

2-D: a 3 × 3 filter on an image

At the boxed spot, each of the 3 filter rows adds (−1)·0 + 0·0 + 1·1 = 1, so 9 products sum to 3.

ŷij = Σa,b wab p̂i+a, j+b

1-D: a 3-weight filter on one row

The same slide, multiply and sum: (−1)·0 + 0·0 + 1·1 = 1. Same edges found, same weights everywhere.

ŷi = Σa wa p̂i+a

1-D keeps everything that matters, local windows and shared weights, and is small enough to write out.

A conv is a local linear layer with tied weights

Unroll a 1-D, one-channel, conv with a 3-weight filter on 8 positions into the matrix it really is.

Forward: ŷ = p̂ Wᵀ

Local: zeros off the band.

Tied: one color per diagonal, the same 3 weights in every column.

Each column of Wᵀ, dotted with all of p̂, fills one entry of ŷ. The zeros wipe out 5 products, leaving 3: the fan-in.

Backward: ∂L/∂p̂ = ∂L/∂ŷ W

Same shapes: each gradient has the shape of its variable.

A column of W is a row of Wᵀ: the 3 outputs that p̂ fed, the fan-out.

Read down it: w₃ w₂ w₁, a conv with the filter flipped.

Same multiply and sum as a linear layer, so the same variance rules. Backprop just multiplies by W instead of Wᵀ.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

The conv sits in the network. Backprop brings ∂L/∂ŷ = g to its output, and ∂ŷ/∂W is its input, p̂.

One weight, used in many places: its gradient is the sum over every place it was used.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

Chain rule: ∂L/∂W = (∂L/∂ŷ)ᵀ p̂ = gᵀ p̂, an 8 × 8 matrix. Entry (j, i) is g_j p̂_i.

One weight, used in many places: its gradient is the sum over every place it was used.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

Gradient descent subtracts α times it from the weights, entry by entry: W ← W − α ∂L/∂W.

One weight, used in many places: its gradient is the sum over every place it was used.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

But 50 entries of W are the conv's zeros. They must stay 0, so their gradients are dropped.

One weight, used in many places: its gradient is the sum over every place it was used.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

Every w₁ is the same weight, so each copy's gradient says how w₁ should move. Add them, then make one update.

One weight, used in many places: its gradient is the sum over every place it was used.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

The same for w₂: 8 copies, summed into one gradient and one update.

One weight, used in many places: its gradient is the sum over every place it was used.

Tied weights add up their gradients

How the 1-D conv's three weights get updated.

p̂

1 × 8

1-D conv

weights W

1 × 8

⋯ rest

of the network

loss

∂L/∂w₁ = g₂p̂₁ + g₃p̂₂ + … + g₈p̂₇

7 copies, summed

w₁ ← w₁ − α ∂L/∂w₁

∂L/∂w₂ = g₁p̂₁ + g₂p̂₂ + … + g₈p̂₈

8 copies, summed

w₂ ← w₂ − α ∂L/∂w₂

∂L/∂w₃ = g₁p̂₂ + g₂p̂₃ + … + g₇p̂₈

7 copies, summed

w₃ ← w₃ − α ∂L/∂w₃

And w₃. Three weights move, each by the sum of the gradients from every place it was used.

One weight, used in many places: its gradient is the sum over every place it was used.

Variance through a conv layer: backward

Backprop: the gradient flows down, from ŷ back through the conv and the ReLU.

1

Count the fan-out. Each input pixel feeds 9 positions in each of 32 output channels: fan-out = C_out · k² = 32 · 9 = 288.

TUESDAY'S CNN · BLOCK 1

Forward counts fan-in; backward counts fan-out.

σ²∂L/∂ŷ = 1

Variance through a conv layer: backward

Backprop: the gradient flows down, from ŷ back through the conv and the ReLU.

1

Count the fan-out. Each input pixel feeds 9 positions in each of 32 output channels: fan-out = C_out · k² = 32 · 9 = 288.

2

The gradient is a conv too: ∂L/∂p̂ = ∂L/∂ŷ ⋆ W_flipped. Each p̂ collects the gradient of every ŷ it fed, through the flipped filter: the same multiply and sum, so the same rule.

TUESDAY'S CNN · BLOCK 1

Forward counts fan-in; backward counts fan-out.

σ²∂L/∂ŷ = 1

Variance through a conv layer: backward

Backprop: the gradient flows down, from ŷ back through the conv and the ReLU.

1

Count the fan-out. Each input pixel feeds 9 positions in each of 32 output channels: fan-out = C_out · k² = 32 · 9 = 288.

2

The gradient is a conv too: ∂L/∂p̂ = ∂L/∂ŷ ⋆ W_flipped. Each p̂ collects the gradient of every ŷ it fed, through the flipped filter: the same multiply and sum, so the same rule.

3

Through the conv: σ²_∂L/∂p̂ = k²C_out · E[w²] · σ²_∂L/∂ŷ = 288 · (2/288) · 1 = 2.

measured: 2.01

TUESDAY'S CNN · BLOCK 1

σ²∂L/∂p̂ = 2.01

Forward counts fan-in; backward counts fan-out.

σ²∂L/∂ŷ = 1

Variance through a conv layer: backward

Backprop: the gradient flows down, from ŷ back through the conv and the ReLU.

1

Count the fan-out. Each input pixel feeds 9 positions in each of 32 output channels: fan-out = C_out · k² = 32 · 9 = 288.

2

The gradient is a conv too: ∂L/∂p̂ = ∂L/∂ŷ ⋆ W_flipped. Each p̂ collects the gradient of every ŷ it fed, through the flipped filter: the same multiply and sum, so the same rule.

3

Through the conv: σ²_∂L/∂p̂ = k²C_out · E[w²] · σ²_∂L/∂ŷ = 288 · (2/288) · 1 = 2.

measured: 2.01

4

Through the ReLU: ∂L/∂x̂ = ∂L/∂p̂ ⊙ H(x̂). H is 0 or 1, and 0 half the time, so it halves the gradient's second moment back to 1.

measured: 1.01

σ²∂L/∂ŷ = 1  →  2 after the conv  →  1 after the ReLU

Here C_in = C_out, so one init balances both directions. When they differ, He init has to pick which one to count.

TUESDAY'S CNN · BLOCK 1

σ²∂L/∂p̂ = 2.01

σ²∂L/∂x̂ = 1.01

Forward counts fan-in; backward counts fan-out.

σ²∂L/∂ŷ = 1

Variance through a pool: forward

2 × 2 windows, stride 2. The input has mean 0 and σ² = 1.

Average: a sum of 4 terms with every weight ¼, so the conv rule applies: σ² = 4 · (¼)² · 1 = ¼.

Max: the winner had to beat three others, so it sits higher and spreads less. The largest of 4 standard normals has mean 1.03 and σ² 0.49, computed from its distribution, 4φ(x)Φ(x)³, and matched by measurement.

Both pools shrink the variance, and max also shifts the mean. The GroupNorm after each pool resets both.

Where the max pool's 0.49 comes from

The max of a 2 × 2 window is the largest of 4 independent standard normals.

1

One value. Each of the 4 values in the window is N(0, 1). The chance one is at most x is Φ(x), the area under the bell curve to the left of x.

2

The max is at most x exactly when all four are at most x. The four are independent, so multiply:
P(max ≤ x) = Φ(x) · Φ(x) · Φ(x) · Φ(x) = Φ(x)⁴

3

Its density is the derivative: f(x) = 4 Φ(x)³ φ(x). Read it as: one of the 4 values lands at x (4 choices, density φ), and the other 3 fall below it (Φ³).

4

Mean and variance are the usual integrals, done numerically:
μ = ∫ x f(x) dx = 1.03 σ² = ∫ (x − μ)² f(x) dx = 0.49

At x = 1, one value is below it 84% of the time, but all four only 50% of the time: the max of 4 sits around 1.

A bigger window, n values: P(max ≤ x) = Φ(x)ⁿ, so the max moves further right and spreads even less.

Variance through a pool: backward

The gradient g arrives at one output. Where does it go in the window?

Max: only the winner gets g; the other three get 0. Over the four inputs, E[(∂L/∂x)²] = ¼ · E[g²].

Average: all four get g/4. E[(∂L/∂x)²] = (¼)² · E[g²].

The residual skips route around the convs, not around the pools, so these cuts compound: two max pools, ÷16.

Max pooling sends gradient to only the winner. Remember that.

Variance through the whole network: forward

Tuesday's CNN at init (stem σ_W² = 1/25, He for the block convs). Blue: ideal σ² (E[p̂²] after a ReLU) and distribution.

Every branch starts with a GroupNorm: N(0, 1). ReLU keeps half, E[p̂²] = ½, with a spike at 0; each He-init conv brings σ² back to 1.

Each branch adds 1 to the stream. A max pool of independent values shifts the distribution right and keeps σ² × 0.49: .49 → 1.47, pool → .74 → 1.71 → 2.68.

The head's GroupNorm resets to N(0, 1); the average pool then narrows it to a tight bump.

The stream grows by about 1 per block, but every branch and the head see σ² = 1, because each starts with a norm.

Variance through the whole network: forward

Tuesday's CNN at init (stem σ_W² = 1/25, He for the block convs). Blue: ideal σ² (E[p̂²] after a ReLU) and distribution. Orange: measured on 512 digits.

Max pools barely shrink the real stream (.89 → .90, 1.88 → 2.02): neighboring pixels are nearly equal, so the max of 4 is close to any one of them.

Each branch still adds about 1: .90 → 1.88, 2.02 → 3.16 → 4.41. The real stream is right-skewed, not normal.

The first ReLU in each block keeps about 0.64, not ½: its GroupNorm normalizes that skewed stream. A norm fixes the mean and variance, not the shape.

The stream grows by about 1 per block, but every branch and the head see σ² = 1, because each starts with a norm.

Variance through the whole network: backward

Same network, gradients flowing back. Numbers: E[(∂L/∂·)²] per element, with ∂L/∂out of variance 1; µ = ×10⁻⁶. Blue: ideal.

Head: Linear ×10/32 → 0.31; the avg pool spreads it over 49 positions, ÷49²; ReLU ×½ (a spike of zeros); GroupNorm ÷σ²(X₃) = ÷2.68.

Each + passes the gradient straight back, and its branch adds its own share, which GroupNorm scales by 1/σ² of the stream: big where the pools shrank the stream. 24.3µ → 38.5µ → 90.3µ.

Each max pool: ÷4, with 3 of every 4 gradients exactly 0 (90.3µ → 22.6µ). The stem conv ×32 back to the image.

Gradients flow back along the stream almost unchanged. The pools shrink them; the norms set each branch's share.

Variance through the whole network: backward

Same network, gradients flowing back. Numbers: E[(∂L/∂·)²] per element, with ∂L/∂out of variance 1; µ = ×10⁻⁶. Blue: ideal. Orange: measured.

The real stream has a bigger σ² (the pools barely shrank it), so GroupNorm divides by more and the branches add less: 13.3µ → 16.7µ → 22.2µ, not 90.3µ.

The pool still zeroes 3 in 4 (75% exactly 0 measured) and cuts E² by 4: 22.2µ → 5.5µ.

Shapes: after a ReLU or a pool, most of the gradient is a spike at 0; elsewhere it is close to normal.

Gradients flow back along the stream almost unchanged. The pools shrink them; the norms set each branch's share.

Variance through the whole network: backward

Same network, gradients flowing back. Numbers: E[(∂L/∂·)²] per element, with ∂L/∂out of variance 1; µ = ×10⁻⁶. Blue: ideal. Orange: measured.

At the weights, the sum brings it back. ∂L/∂W = Σ g · p̂ over every position of every image: 512 × 14 × 14 = 100,352 products per weight in block 1, 25,088 in blocks 2 and 3.

Per-entry g (rms) → weight gradient (rms): block 1: 0.0022 → 1.21, 0.0024 → 1.37; block 2: 0.0038 → 1.18, 0.0041 → 1.26; block 3: 0.0032 → 1.05, 0.0036 → 1.30.

‖∂L/∂W‖ / ‖W‖ ≈ 10–16 in every conv, stem to block 3: no layer is starved. (Large overall because this loss sums over the 512 images.)

Gradients flow back along the stream almost unchanged. The pools shrink them; the norms set each branch's share.

Max pooling is a gated activation

V

hardmax(V)

·

out

scaled

Tuesday's top-edge map of the 7, and its strongest 2 × 2 window

ReLU(ŷ), 24 × 24

WINDOW ROW 1, COLUMN 6

V (1 × 4)

3.4

4.1

8.4

9.1

hardmax(V) (4 × 1)

0

0

0

1

= 9.1

= 3.4·0 + 4.1·0 + 8.4·0 + 9.1·1

max pool = V · hardmax(V)

YOU'VE SEEN THIS SHAPE
ReLU = H(ŷ) · ŷ: a step gates the value.
Max = V · hardmax(V): a hard max gates the values.

Each value scores itself, and only the winner passes.

Max pooling is an activation times the values, just like ReLU.

Does max pooling learn?

max pool = V · hardmax(V). Talk it over with a neighbour.

Does it have anything to learn?

Which inputs get a gradient?

Does max pooling learn?

max pool = V · hardmax(V). Talk it over with a neighbour.

Does it have anything to learn?

No weights at all. The gate, hardmax(V), is computed from the values themselves. It only routes.

Which inputs get a gradient?

Only the winner: ∂out/∂V = hardmax(V) = [0, 0, 0, 1]. The other three hear nothing, as on the backward pool slide.

Does max pooling learn?

max pool = V · hardmax(V). Talk it over with a neighbour.

Does it have anything to learn?

No weights at all. The gate, hardmax(V), is computed from the values themselves. It only routes.

Which inputs get a gradient?

Only the winner: ∂out/∂V = hardmax(V) = [0, 0, 0, 1]. The other three hear nothing, as on the backward pool slide.

We have softened a hard gate before

ReLU = H(ŷ)·ŷ

dashed: the gate H(ŷ)

→

SiLU = σ(ŷ)·ŷ

dashed: the gate σ(ŷ)

Swapping the step H for the smooth σ let every input get a gradient.

Does max pooling learn?

max pool = V · hardmax(V). Talk it over with a neighbour.

Does it have anything to learn?

No weights at all. The gate, hardmax(V), is computed from the values themselves. It only routes.

Which inputs get a gradient?

Only the winner: ∂out/∂V = hardmax(V) = [0, 0, 0, 1]. The other three hear nothing, as on the backward pool slide.

We have softened a hard gate before

ReLU = H(ŷ)·ŷ

dashed: the gate H(ŷ)

→

SiLU = σ(ŷ)·ŷ

dashed: the gate σ(ŷ)

Swapping the step H for the smooth σ let every input get a gradient.

COULD WE DO THIS AGAIN?

H(ŷ) · ŷ → σ(ŷ) · ŷ

V · hardmax(V) → V · softmax(V) ?

Same move: replace the hard gate with a smooth one.

A hard gate learns little and teaches little. Let's make this one soft.

Fix 1: make the gate soft

Keep the gated identity; swap the hard gate for a smooth one, as SiLU did.

V

softmax(V)

·

out

scaled

THE 7, TOP-EDGE MAP

v₁

v₂

v₃

v₄

the 2 × 2 window: these four pixels are V

V · hardmax(V)→V · softmax(V)
like H(ŷ) · ŷ → σ(ŷ) · ŷ

v₁ = 3.44

ev₁ = 31

g₁ = 0.002

v₂ = 4.11

ev₂ = 61

g₂ = 0.004

v₃ = 8.44

ev₃ = 4,647

g₃ = 0.337

v₄ = 9.11

ev₄ = 9,054

g₄ = 0.656

Σ ev = 13,794

WINDOW VALUES V

SOFTMAX

GATE softmax(V)

THEN THE SAME DOT PRODUCT

out = V · softmax(V)

= 3.44·0.002 + 4.11·0.004
 + 8.44·0.337 + 9.11·0.656

= 8.85 (max pool: 9.11)

Every gate is positive and they sum to 1. The winner still dominates, but every value gets a share, so every value gets a gradient. This is SoftPool.

Same gated identity, smooth gate: exponentiate, share one sum, divide.

One dial from max to average

V

hardmax(V)

·

out

scaled

Add a temperature to the soft gate: softmax(V / τ). τ = 1 is SoftPool.

gi = evi/τ / Σk evk/τ

max pool

← a softmax mix →

average pool

τ = 0.01

τ = 0.5

τ = 1

τ = 2

τ = 100

0.00

v = 3.44

0.00

v = 4.11

0.00

v = 8.44

1.00

v = 9.11

out = V · softmax(V/τ) = 9.11

τ → 0: one-hot, out = 9.11. Max pooling.
τ → ∞: every g = ¼, out = 6.28. Average pooling: the gates add to 1, so it is the mean, not the sum.

Cold: the softmax is a hard max. Only 9.11 passes.

Max and average pooling are the two ends of one dial.

One dial from max to average

V

softmax(V/0.5)

·

out

scaled

Add a temperature to the soft gate: softmax(V / τ). τ = 1 is SoftPool.

gi = evi/τ / Σk evk/τ

max pool

← a softmax mix →

average pool

τ = 0.01

τ = 0.5

τ = 1

τ = 2

τ = 100

0.00

v = 3.44

0.00

v = 4.11

0.21

v = 8.44

0.79

v = 9.11

out = V · softmax(V/τ) = 8.97

τ → 0: one-hot, out = 9.11. Max pooling.
τ → ∞: every g = ¼, out = 6.28. Average pooling: the gates add to 1, so it is the mean, not the sum.

Warmer: 8.44 starts to get a share of the gate.

Max and average pooling are the two ends of one dial.

One dial from max to average

V

softmax(V)

·

out

scaled

Add a temperature to the soft gate: softmax(V / τ). τ = 1 is SoftPool.

gi = evi/τ / Σk evk/τ

max pool

← a softmax mix →

average pool

τ = 0.01

τ = 0.5

τ = 1

τ = 2

τ = 100

0.00

v = 3.44

0.00

v = 4.11

0.34

v = 8.44

0.66

v = 9.11

out = V · softmax(V/τ) = 8.85

τ → 0: one-hot, out = 9.11. Max pooling.
τ → ∞: every g = ¼, out = 6.28. Average pooling: the gates add to 1, so it is the mean, not the sum.

τ = 1: the two strong values share it, 0.34 and 0.66.

Max and average pooling are the two ends of one dial.

One dial from max to average

V

softmax(V/2)

·

out

scaled

Add a temperature to the soft gate: softmax(V / τ). τ = 1 is SoftPool.

gi = evi/τ / Σk evk/τ

max pool

← a softmax mix →

average pool

τ = 0.01

τ = 0.5

τ = 1

τ = 2

τ = 100

0.03

v = 3.44

0.04

v = 4.11

0.39

v = 8.44

0.54

v = 9.11

out = V · softmax(V/τ) = 8.45

τ → 0: one-hot, out = 9.11. Max pooling.
τ → ∞: every g = ¼, out = 6.28. Average pooling: the gates add to 1, so it is the mean, not the sum.

Hotter still: even the weak values get a few percent.

Max and average pooling are the two ends of one dial.

One dial from max to average

V

¼ each: mean

·

out

scaled

Add a temperature to the soft gate: softmax(V / τ). τ = 1 is SoftPool.

gi = evi/τ / Σk evk/τ

max pool

← a softmax mix →

average pool

τ = 0.01

τ = 0.5

τ = 1

τ = 2

τ = 100

0.24

v = 3.44

0.24

v = 4.11

0.26

v = 8.44

0.26

v = 9.11

out = V · softmax(V/τ) = 6.34

τ → 0: one-hot, out = 9.11. Max pooling.
τ → ∞: every g = ¼, out = 6.28. Average pooling: the gates add to 1, so it is the mean, not the sum.

Very hot: all four share the gate almost equally.

Max and average pooling are the two ends of one dial.

Every input gets a gradient

∂out/∂v_j on the same window: the hard gate against the soft one at τ = 1

v = 3.44

v = 4.11

v = 8.44

v = 9.11

Hard max

0

0

0

1

Softmax, τ = 1

−0.010

−0.017

0.200

0.827

positivenegative

∂out/∂vj = gj (1 + (vj − out) / τ)

The product rule again: v_j is both the value and its own gate's score. Below-average values get a negative push: raising them would pull the output down.

SAME STORY AS RELU → SILU
A hard gate passes gradient only where it is open. Smooth the gate, and every input learns, even the ones it mostly shuts.

A soft gate: every value hears how it should change.

When is the biggest value the wrong one?

Talk it over with a neighbour before we move on

Soft or hard, the pool still scores each value by its own size. When would you want the gate to favour something other than the loudest value?

When loud isn't relevant

A thick, bright stroke fires hard, but a 7 and a 1 both have one. The faint top bar is what tells them apart. Relevance belongs to the task, not to the size of the value.

When relevance depends on context

Whether an edge matters depends on what else is in the image: a short bar at the top means one thing above a diagonal and another above a loop.

Split the job in two: one learned projection scores, another supplies the value. The second card needs more: that's Nov 10.

"Biggest" was only ever a stand-in. Let the network learn the score.

Fix 2: learn the score

You have seen this split before: a gate path and a value path, each with its own weights

gate pathvalue path

ACTIVATIONS LECTURE · SWIGLU

X

· Wᵀ

· Vᵀ

SiLU(·)

⊙

XWᵀ

gate

value XVᵀ

P̂

P̂ = SiLU(XWᵀ) ⊙ XVᵀ

Gate: W reads the token and decides how far each channel opens.

Value: V decides what passes through.

The gate opens and closes the token's own channels.

ATTENTION POOLING: A LEARNED SCORE

X

· wᵀ

· Vᵀ

softmaxj(·)

scores s

gate

value XVᵀ

out

out = softmax(Xwᵀ)ᵀ · XVᵀ

Gate: a learned w scores each position j. Its size no longer decides.

Value: V decides what each position contributes.

The gate chooses between positions: one g_j per position.

Same split as SwiGLU, new direction: gate across positions instead of channels.

The same evolution, twice

ReLU became SwiGLU in three moves. Max pooling can take the same three.

Hard gate
a step decides

Soft gate
a smooth version

Learned gate
gate and value projections

One unit's value
activations lecture

ReLU

H(ŷ) · ŷ

SiLU

σ(ŷ) · ŷ

GLU / SwiGLU

σ(XWᵀ) ⊙ XVᵀ

A window's 4 values
today

Max pooling

V · hardmax(V)

SoftPool

V · softmax(V)

Attention pooling

softmax(Xwᵀ)ᵀ · XVᵀ

hardmax(V) is one-hot at the largest value, the way H(ŷ) is 1 where ŷ > 0. Swap it for softmax and every value gets a share. Then give the gate its own weights w and the value its own V.

Max pooling is an activation times the values, just like ReLU.

What is this called in the CNN world?

Talk it over with a neighbour before we move on

We mix a window's 4 pixels with weights that sum to 1. Depending on what sets those weights, this already has a name. What is it?

EACH VALUE SCORES ITSELF

SoftPool

V · softmax(V)

A drop-in replacement for max pooling; every value gets gradient.

Stergiou, Poppe & Kalliatakis (ICCV 2021)

A LEARNED PROJECTION SCORES

Attention pooling

softmax(Xwᵀ)ᵀ · XVᵀ

The weights depend on the pixels, through w. Our first attention.

FIXED LEARNED WEIGHTS

A 2 × 2, stride-2 conv

V · w

Non-overlapping windows, 4 pixels in, 1 out: a learned reduction. The all-convolutional net replaced pooling this way.

Springenberg et al. (ICLR workshop 2015)

Same window, same mix. What differs is who sets the weights.

Fixed mixing vs. input-dependent mixing

Both are learned. The question is whether the mixing weights change from image to image.

Convolution

out = V · w

w_j is set by where pixel j sits in the window: top-left always gets the same weight.

Learned once in training, then the same for every image and every window.

Linear in V.

Gated pooling (SoftPool, attention pooling)

out = V · g(V)

g(V) is set by what the pixels contain: computed fresh for each window of each image.

The learned part is the rule for making weights (w), not the weights themselves.

Not linear in V: the input multiplies itself through the gate.

Gated pooling is still local and still uses the same w and V in every window. On Nov 10 we let a window's gate depend on the other windows too.

A conv mixes by position; a gate mixes by content.

Does it matter on MNIST?

Tuesday's network with both pools swapped. Full MNIST, 6 epochs, 3 seeds each.

Circles: one seed each. Bar: the mean. Gradient share measured at initialization.

LET EACH CHANNEL LEARN τ

Started at τ = 1, ended between 0.89 and 1.12. Adam could have moved it 10× either way. The push kept flipping: the network has no preference.

All six land within 0.3 points, about the spread between seeds. Max learns as well as the soft gates while only a quarter of its inputs ever hear a gradient.

Every input learns, but MNIST doesn't need a smarter gate. A learned score needs a harder question.

What to take away

Max pooling is a gate, in three lines

01

Convs keep variance with fan-in and fan-out of k² × channels. Pools change it, and max sends gradient to only the winner.

02

Max pooling is hardmax times the values. Soften it: SoftPool. Give it gate and value projections: attention pooling.

03

A stride-2 conv mixes with fixed weights; a gate mixes with weights the input chooses.

Pooling is a gate over positions, and gates can be learned.

Quiz details

Thursday, Nov 5: the full hour, equation sheet provided

RESIDUALS AND NORMALIZATION

Gradient flow through plain and residual networks
Variance of the residual stream
BatchNorm and LayerNorm

SHAPES AND SOFTMAX

Shapes through a residual network
Softmax and cross-entropy
Bonus: scaling residual branches

Activations, dropout and optimizers move to Quiz 3.

Next time

No class Tuesday (Election Day)

THU NOV 5

Quiz 2

The full hour. Meanwhile: choose your Project 2 network, and start on your Project 3 proposal (due Nov 10).

TUE NOV 10

Attention

Today's gate only sees its own window. Split W into two projections, one for this window and one for every other window, and each window's gate is set by everything around it.

Today: a gate inside the window. Next: a gate across windows.

Four gates, one empty cell

Every gate in this course so far, sorted two ways. What belongs in the last cell?

Within one token
the gate opens channels

Across positions
the gate chooses tokens

The value gates itself
no gate weights

A learned gate
its own projection

SiLU

ŷ · σ(ŷ)

Each unit's own value sets its gate.

Max / softmax pooling

V · softmax(V/τ)

Each value scores itself within its window.

GLU / SwiGLU

σ(XWᵀ) ⊙ XVᵀ

W reads the token and gates its channels.

Attention

softmax(QKᵀ/√d) XVᵀ

A learned gate between every pair of tokens. Last slide's pool was the first try.

Attention is GLU's learned gate, turned to face across positions.

A learned score, but for whom?

One w gives the whole image one ranking. Different positions need different things.

One learned w

sj = xj wᵀ

The score of position j doesn't depend on who is pooling. If the bar scores highest, it wins for everyone: the corner, the stem, the background.

That's fine for one summary of the whole image, but not for a layer where every position pools.

Split the score in two

sij = qi · kj

Query q_i: what position i, the one pooling, is looking for.

Key k_j: what position j, the one being considered, has to offer.

Context comes in through the query: the same bar can matter to the corner and not to the background.

GLU's gate read one token. This gate reads two: the one asking and the one being asked about.

Split the gate's weights into a query half and a key half.

Fix 3: split the gate into query and key

How position i pools: one query, and a key and value from every position j

xi

xj

the one asking

each position

· WQᵀ

· WKᵀ

· Vᵀ

qi

kj

xjVᵀ

q · k / √d

sij

softmax over j

gate gij

×

Σj

p̂i

p̂i = Σj softmaxj(qi·kj/√d) xjVᵀ

W_Q, W_K: d × f. V: n × f, the same value projection as GLU. Every token uses the same three weight matrices.

GLU's gate weights W split into W_Q and W_K. The value path is unchanged; only now the value comes from position j.

The gate is set by a pair of tokens: who's asking and who's offering.

Every position asks its own question

All t queries at once give a t × t gate matrix G. Six tokens of the 7, illustrative numbers.

key: the position offering (j)

bar, left

bar, right

corner

stem, middle

stem, bottom

background

query: the position asking (i)

bar, left

0.30

0.34

0.14

0.08

0.06

0.08

bar, right

0.33

0.29

0.20

0.07

0.05

0.06

corner

0.10

0.18

0.12

0.24

0.30

0.06

stem, middle

0.06

0.12

0.28

0.20

0.28

0.06

stem, bottom

0.12

0.30

0.32

0.12

0.08

0.06

background

0.17

0.17

0.16

0.17

0.16

0.17

Row i is where position i looks. It is a softmax, so each row sums to 1: one soft pool per position.

The corner and the stem ask different questions, so they pool different tokens. One w could never do that.

One gate per pair, not per channel. g_ij scales all n channels of x_jVᵀ alike. GLU had a gate for every channel.

t positions, t soft pools, each with its own gate.

Fix 4: drop the window

Let j run over every position in the image, not just a 2 × 2 neighbourhood

CNN: global only through depth

receptive field = 1 + L(k − 1)

With 5 × 5 filters at stride 1, a unit needs 7 layers to see all 28 pixels. Pooling speeds that up, but parts of the digit far apart only meet late.

Attention: global in one layer

j = 1 … t, every position

Every token can pool from every other token in a single step. The bottom of the stem can look straight at the top bar.

MEASURED · ONE CONV LAYER (32 FILTERS) → 12 × 12 TOKENS → GLOBAL AVERAGE POOL → LINEAR

Conv stem only: 58.0%

+ one attention layer: 90.0%

Test accuracy, 8,000 training and 2,000 test MNIST digits, 20 epochs, seed 0. A deliberately weak stem: the point is reach, not a record.

No window: one layer sees the whole image.

Attention in one line

All four fixes, for all t positions at once. X is one image's tokens: t × f.

P̂ = softmax(QKᵀ / √d) XVᵀ

Q = XWQᵀ
K = XWKᵀ

t × d each

Every token makes a query and a key with the same weights.

QKᵀ / √d

t × t

Every score s_ij at once. √d is a temperature (next slide).

G = softmax(·)

t × t, rows sum to 1

Row by row: the gate. Fixes 1, 3 and 4.

P̂ = G · XVᵀ

(t × t)(t × n) = t × n

Each row: a gated sum of values. Then P̂ W_outᵀ maps n back to f.

In papers (Vaswani et al., 2017, "scaled dot-product attention"): softmax(QKᵀ/√d_k) V. Their V is the value matrix, our XVᵀ; their value weights are usually written W_V. Same computation, different letters.

A t × t gate times t values: t soft pools in one matrix product.

Why divide by √d? It's the temperature.

q · k sums d products. With unit-variance entries its variance is d, so wider queries make colder softmaxes.

d

std of q · k

largest gate, softmax(QKᵀ)

largest gate, softmax(QKᵀ/√d)

1

1.02

0.063

0.063

4

2.04

0.209

0.065

16

4.03

0.527

0.063

64

7.97

0.774

0.063

256

15.95

0.891

0.064

Random unit-variance queries and keys, t = 144 keys per query, mean over 500 queries. 1 would be a hard max; 1/144 = 0.007 would be an even split.

BACK TO THE DIAL

softmax(s / √d) is the dial from Fix 1 with τ = √d.

Without it, a 64-wide query gives its largest token 0.77 of the gate: almost back to max pooling, and almost no gradient for the rest.

√d keeps the gate soft at any width, so Fix 1 survives.

Today's tokens: positions of a feature map

A token is one position's vector of channel values: a pixel with 32 numbers instead of 1

the 7, 28 × 28 × 1

Conv2d, 32 filters, 5 × 5
ReLU, max pool 2 × 2

Tuesday's stem

12 × 12 × 32

X: 144 × 32

t = 144 tokens (rows), f = 32 features each. Flatten the grid; keep each position's channels together.

Shading: how strongly each token responds (the length of its 32-number vector). The 7 is still there at a quarter of the resolution.

Part 2 swaps this stem for patches, which turn out to be a convolution too. The attention layer doesn't change.

To attention, an image is a set of t vectors.

Attention on the 7

One row of the trained gate G: where one query token looks among all 144

QUERY TOKEN: ROW 2, COLUMN 6 (12 × 12 GRID)

the query token

large gate g_ij

the 7 (token strength)

Query on the top bar

It pools from the bar itself and the right end where the bar turns down. Tokens that look like it.

Top 3 tokens hold 0.31 of the gate. An even split would give them 0.021.

Conv stem + one single-head attention layer (d = 16) + global average pool, trained on 8,000 MNIST digits; 90.0% test accuracy, p̂ = 0.97 for this 7.

Each token pools by content, from anywhere in the image.

Attention on the 7

One row of the trained gate G: where one query token looks among all 144

QUERY TOKEN: ROW 3, COLUMN 8 (12 × 12 GRID)

the query token

large gate g_ij

the 7 (token strength)

Query at the corner

It looks 7 rows down, to the bottom of the stem. Three pooling stages would be needed for a 2 × 2 window to reach that far.

Top 3 tokens hold 0.45 of the gate. An even split would give them 0.021.

Conv stem + one single-head attention layer (d = 16) + global average pool, trained on 8,000 MNIST digits; 90.0% test accuracy, p̂ = 0.97 for this 7.

Each token pools by content, from anywhere in the image.

Attention on the 7

One row of the trained gate G: where one query token looks among all 144

QUERY TOKEN: ROW 10, COLUMN 3 (12 × 12 GRID)

the query token

large gate g_ij

the 7 (token strength)

Query at the bottom of the stem

It looks back up, mostly at the corner and the right end of the bar. The two ends of the stroke find each other in one layer.

Top 3 tokens hold 0.23 of the gate. An even split would give them 0.021.

Conv stem + one single-head attention layer (d = 16) + global average pool, trained on 8,000 MNIST digits; 90.0% test accuracy, p̂ = 0.97 for this 7.

Each token pools by content, from anywhere in the image.

What a global gate costs

G is t × t: the number of gates grows with the square of the number of tokens

tokens

t

gates t² per head, per layer (log scale)

Today: 12 × 12 tokens
after the conv stem and pool

144

2.1 × 10⁴

Tuesday’s 24 × 24 map
no pooling

576

3.3 × 10⁵

224 × 224, 16 × 16 patches
a standard ViT (part 2)

196

3.8 × 10⁴

224 × 224, every pixel
one token per pixel

50,176

2.5 × 10⁹

One token per pixel of a 224 × 224 photo: 2.5 × 10⁹ gates, about 10 GB at 4 bytes each, for one image, one head, one layer. Grouping pixels into 16 × 16 patches cuts that by a factor of about 65,000.

Global costs t². That is why a ViT starts by cutting the image into patches.

Shuffle the tokens

Talk it over with a neighbour before we move on

the 7's tokens

shuffle →

the same 144 tokens, scrambled

Feed the scrambled tokens through the attention layer.

What happens to P̂? And to the prediction after the global average pool?

Tuesday: shuffling pixels didn't bother the MLP but sank the CNN. Which one does attention behave like? Look back at the equation: where does position appear in it?

Predict first, then we'll check.

Shuffle in, shuffle out

Attention is equivariant to every reordering of the tokens, not just shifts

P̂ reorders the same way

attn(shuffle(X)) = shuffle(attn(X))

Measured on the trained layer with the 7's tokens: largest difference 1.1 × 10⁻⁵, which is float rounding.

After the global average pool the order is gone entirely: the prediction is identical (invariant).

WHY
Look at s_ij = q_i · k_j. Nothing in it says where token j sits, only what it contains. The weights are shared by every token, so a token's place in the list never enters the computation.

Like the CNN, no weight has an address. Unlike the CNN, nothing knows who its neighbours are.

To this layer, the 7 and its scrambled tokens are the same input. Our stem is still local, so the inside of each token keeps its order; a ViT has nothing like that beyond the patch. Part 2 gives back the "where": positional embeddings.

Attention scores what a token is, never where it is.

Less built in, more data needed

Tuesday's trade-off, one more rung: what each layer assumes about images

MLP

Every weight has an address. Assumes nothing about neighbours or shifts.

Pixel shuffle: doesn't care.

CNN

Same filter everywhere, and locality: neighbours belong together. Shift-equivariant.

Pixel shuffle: breaks it.

Attention

Same weights for every token, but no locality and no position. Equivariant to every reordering.

Token shuffle: can't even tell.

VISION TRANSFORMER vs. RESNET, IMAGENET TOP-1 · DOSOVITSKIY ET AL. (ICLR 2021)

Pretrain on ImageNet (1.3 M images): the ResNets win.

ImageNet-21k (14 M): about even.

JFT-300M (300 M): ViT-H/14 88.55% vs. ResNet (BiT-L) 87.54%.

"Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality."

Attention trades locality for reach. The data has to supply what the layer doesn't assume.

Our lens and the field's words

"Attention is a gate across positions" is a way of seeing it. Here is how papers say the same things.

IN THIS COURSE

IN PAPERS

the gate G

attention weights / attention pattern

a soft pool over positions

a weighted average of values

the query and key halves of the gate

queries and keys

value projection V; values XVᵀ

W_V; values V

√d as the temperature

scaled dot-product attention

tokens of a feature map

a sequence of tokens / patches

WHERE THE ANALOGY STRETCHES

Max pooling had no query. The value scored itself. The query is the new idea, not a detail.

One gate per pair, not per channel. GLU gated each channel; attention gives each pair one number (one per head, in part 2).

No shrinking. Strided pooling makes the map smaller; attention gives back t tokens for t tokens.

Prior work on this view: Stand-Alone Self-Attention (Ramachandran et al., 2019); Gated Attention Unit (Hua et al., 2022).

Same computation, two vocabularies. Know both.

Preview: the transformer block

Two residual branches you already know how to build, stacked L times

X

t × f

X′

Norm

Attentioncombines across tokens

+

Norm

SwiGLU MLPuntangles each token

+

residual stream: each branch adds to X

Attention is the only place tokens exchange information. Today's lecture.

The MLP sees one token at a time with shared weights: a 1 × 1 convolution over the token grid.

Norm and + are the residual block from earlier in the course; transformers use LayerNorm or RMSNorm instead of GroupNorm.

Attention combines across tokens; the MLP untangles each one. Repeat.

What to take away

From max pooling to attention, in three lines

01

Pooling is a gate over positions. Attention makes it soft, learned, and set by who is asking.

02

GLU's gate weights split into a query and a key. √d is the temperature that keeps the gate soft.

03

Attention is global in one layer and blind to position: it costs t² and needs data for what it doesn't assume.

Attention: a soft, learned pool that every token runs for itself.

Next time: ViT, part 2

Three loose ends from today, three pieces of a Vision Transformer

Loose end: one gate per pair

Multi-head attention

Several smaller gates side by side, each free to look for something different.

Loose end: t² gates

Patches are a convolution

16 × 16 patches at stride 16: a conv layer whose windows don't overlap. 196 tokens instead of 50,176.

Loose end: blind to position

Positional embeddings

Give every token its "where" back. No CNN equivalent, so we'll take our time.

Today: the gate. Next time: everything a ViT builds around it.