Steering what a network learns
Inductive bias: building a task's invariances into the data or into the architecture
01
Learning isn't enough: which solution?
02
Building shift into the weights: convolution
03
Does the bias pay? The shift test and data efficiency
04
The price of an assumption
05
Choosing the bias your data has
All slides, one per page
© 2026 Dr. Jesse Roberts
Quiz next Thursday
A heads-up before we start
THIS THURSDAY · OCT 29
Details released
What the quiz covers and how it will run.
NEXT THURSDAY · NOV 5
Quiz
Full details come out this Thursday.
Details this Thursday; quiz next Thursday.
Learning isn't enough
Three questions about deep networks, and the one we start on today
1
Can a network represent it?
Almost anything. Your MLP even memorized 8,000 random labels.
2
Can we make it learn?
Activations that pass gradients, softmax and cross-entropy, backpropagation, optimizers.
3
Which solution does it learn?
Many fits exist. We want one that respects the task, and we want it from as little data as possible.
Also correct
under shift
your MLP
All possible models
Fits the training data
Also fits the test data
less likelymore likely · MLP
Thursday's map, shaded by where an MLP is likely to land. It landed on a fit that passes the test set but reads a 7 shifted 3 px as a 3.
Today's job is adjusting which fit the network lands on.
Fast to generalize, slow to memorize
Where SGD takes the MLP through model space, epoch by epoch, on 8,000 training digits
All possible models
less likelymore likely · where training lands · MLP
Fits the training data
Also fits the test data
random init
SGD starts from random weights. Which region does it reach first: fits the training data, or also fits the test data?
Fast to generalize, slow to memorize
Where SGD takes the MLP through model space, epoch by epoch, on 8,000 training digits
All possible models
less likelymore likely · where training lands · MLP
Fits the training data
Also fits the test data
< 1 epoch: 90.0% on test
What we measured
0%
25%
50%
75%
100%
0
20
40
60
0
0.5
1
epoch
zoom: epoch 0–1
fast
test accuracy, true labels
test accuracy, 20% of training labels flipped
flipped training labels memorized
Fast to generalize, slow to memorize
Where SGD takes the MLP through model space, epoch by epoch, on 8,000 training digits
All possible models
less likelymore likely · where training lands · MLP
Fits the training data
Also fits the test data
< 1 epoch: 90.0% on test
epoch 60: every training label fit, flipped ones too
What we measured
0%
25%
50%
75%
100%
0
20
40
60
0
0.5
1
epoch
zoom: epoch 0–1
fast
slow
test accuracy, true labels
test accuracy, 20% of training labels flipped
flipped training labels memorized
Not the classical picture: with true labels the MLP fits every training digit and test accuracy still rises (96.0%). With flipped labels it peaks at 92.8% (epoch 6), then falls to 82.3% as it memorizes them.
SGD finds what generalizes first; memorizing what doesn't transfer takes many more epochs.
Data augmentation is expensive
Same MLP, same 8,000 digits, retrained with random shifts of up to ±3 px
100%
75%
50%
25%
0%
0
1
2
3
4
5
6
shift right (pixels)
MLP + shift augmentation
plain MLP
range shown in training
60%
The bill
Every case must be shown. Shift × scale × rotation is 1,715 versions of every digit: 1.0 × 10⁸ images for MNIST.
Capacity spent on copies. The MLP still stores each stroke separately at every position it was shown.
Only the range shown. 94% at 3 px, but 60% at 6 px, a shift it never saw.
Augmentation teaches an invariance one example at a time.
Invariances: changes that keep the label
What the network should learn to ignore, written as transformations of the input
Keep the label: still a 7
original
shift 3 px
rotate 15°
scale 1.2×
thicker strokes
Change the label: not an invariance
a 6
now a 9
Which changes are invariances depends on the task: turning a photo of a cat upside down leaves a cat; turning a 6 leaves a 9.
INVARIANCE OF THE TASK
A transformation T is an invariance if it never changes the true label:
y(T(x)) = y(x) for every x
We want the network to share it:
ŷ(T(x)) = ŷ(x)
An invariance is knowledge about the task: what can change without changing the answer.
Two paths to learning invariances
Same space of models, same digits: where does training land?
Fits the training data
Fits the training data
Fits the training data
MLP
MLP + augmentation · Thursday
CNN · today
Invariance · of the task
A change that leaves the label alone: a shifted 7 is still a 7.
Inductive bias · of the learner
Which models training is likely to land on, before any data.
Two paths to learning invariances
Same space of models, same digits: where does training land?
Fits the training data
Also correct under shift
Fits the training data
Also correct under shift
Fits the training data
Also correct under shift
MLP
MLP + augmentation · Thursday
CNN · today
Invariance · of the task
A change that leaves the label alone: a shifted 7 is still a 7.
Inductive bias · of the learner
Which models training is likely to land on, before any data.
Two paths to learning invariances
Same space of models, same digits: where does training land?
less likelymore likely · MLP
Fits the training data
Also correct under shift
No help: training lands anywhere that fits, mostly where shifted digits fail.
Fits the training data
Also correct under shift
Fits the training data
Also correct under shift
MLP
MLP + augmentation · Thursday
CNN · today
Invariance · of the task
A change that leaves the label alone: a shifted 7 is still a 7.
Inductive bias · of the learner
Which models training is likely to land on, before any data.
Two paths to learning invariances
Same space of models, same digits: where does training land?
less likelymore likely · MLP
Fits the training data
Also correct under shift
No help: training lands anywhere that fits, mostly where shifted digits fail.
Fits augmented data
less likelymore likely · MLP + aug.
Fits the training data
Also correct under shift
Lever 1, the data. Shifted copies shrink what fits, so training can only land on models that handle them.
Fits the training data
Also correct under shift
MLP
MLP + augmentation · Thursday
CNN · today
Invariance · of the task
A change that leaves the label alone: a shifted 7 is still a 7.
Inductive bias · of the learner
Which models training is likely to land on, before any data.
Two paths to learning invariances
Same space of models, same digits: where does training land?
less likelymore likely · MLP
Fits the training data
Also correct under shift
No help: training lands anywhere that fits, mostly where shifted digits fail.
Fits augmented data
less likelymore likely · MLP + aug.
Fits the training data
Also correct under shift
Lever 1, the data. Shifted copies shrink what fits, so training can only land on models that handle them.
less likelymore likely · CNN
Fits the training data
Also correct under shift
Lever 2, the architecture. Same data as the MLP; tied local weights make the shift-correct fits far likelier.
MLP
MLP + augmentation · Thursday
CNN · today
Invariance · of the task
A change that leaves the label alone: a shifted 7 is still a 7.
Inductive bias · of the learner
Which models training is likely to land on, before any data.
Both levers change where training is likely to land, not what a network could represent.
A CNN makes two assumptions
The network we will build today. What makes it a CNN is two bets about images.
e × 1 × 28 × 28e × 32 × 28 × 28e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 1 × 1e × 32e × 10x, e × 1 × 28 × 28Conv 5×532 × 1max pool 2GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+max pool 2GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+GroupNorm (1)ReLUglobal avg poole a b c → e (a b c)32 × 1² → 32Linear10 × 32out e × 10
Most of this you know: ReLU, the residual +, a Linear layer at the end. The new pieces are the conv layers (outlined), plus pooling and GroupNorm later.
A CNN makes two assumptions
The network we will build today. What makes it a CNN is two bets about images.
e × 1 × 28 × 28e × 32 × 28 × 28e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 1 × 1e × 32e × 10x, e × 1 × 28 × 28Conv 5×532 × 1max pool 2GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+max pool 2GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+GroupNorm (1)ReLUglobal avg poole a b c → e (a b c)32 × 1² → 32Linear10 × 32out e × 10
Assumption 1: locality. What a pixel means depends on its neighbours. So each conv unit looks through a small window, 5 × 5 or 3 × 3, not the whole image.
A CNN makes two assumptions
The network we will build today. What makes it a CNN is two bets about images.
e × 1 × 28 × 28e × 32 × 28 × 28e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 14 × 14e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 7 × 7e × 32 × 1 × 1e × 32e × 10x, e × 1 × 28 × 28Conv 5×532 × 1max pool 2GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+max pool 2GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+GroupNorm (1)ReLUConv 3×332 × 32GroupNorm (1)ReLUConv 3×332 × 32+GroupNorm (1)ReLUglobal avg poole a b c → e (a b c)32 × 1² → 32Linear10 × 32out e × 10
Assumption 2: the same thing anywhere. A stroke is a stroke wherever it sits. So each conv slides one set of weights over every position.
Assumption 1: locality
Every neuron sees all of the input
input X (28 × 28)
W₁ · 784 weights
Σ
ŷ₁ = 5.0
ReLU
5.0
neuron 1 · h₁
W₂ · 784 weights
Σ
ŷ₂ = 4.7
ReLU
4.7
neuron 2 · h₂
ŷn = Σi Wn,i xi + bn,  hn = ReLU(ŷn)
× on each line: each pixel times its weight (b = 0)
Assumption 1: locality
Some input to some neurons: each neuron sees only half
input X (28 × 28)
W₁ · 392 weights, rest 0
Σ
ŷ₁ = 1.9
ReLU
1.9
neuron 1 · h₁
W₂ · 392 weights, rest 0
Σ
ŷ₂ = 1.0
ReLU
1.0
neuron 2 · h₂
ŷn = Σi Wn,i xi + bn,  hn = ReLU(ŷn)
× on each line: each pixel times its weight (b = 0)
Assumption 1: locality
Smaller still: four neurons, each sees a quarter
input X (28 × 28)
each W: 196 weights on its quarter, the rest 0
Σ
ŷ₁ = 1.6
ReLU
1.6
neuron 1 · h₁
Σ
ŷ₂ = 0.9
ReLU
0.9
neuron 2 · h₂
Σ
ŷ₃ = 2.3
ReLU
2.3
neuron 3 · h₃
Σ
ŷ₄ = 1.4
ReLU
1.4
neuron 4 · h₄
Assumption 1: locality
Smallest: 5 × 5 windows, stride 2, one neuron per window
input X (28 × 28)
NEURON 1 OF 144 · WINDOW ROW 0, COLUMN 0
−1
+0.3
+0.2
+0.4
+0.8
+0.7
−0.3
−0.9
+0.9
+0.2
−0.3
+0.7
−0.3
−0.8
−0.3
+0.4
−0.2
+0.2
−0.6
0
−0.5
−0.8
+0.7
+1
−0.6
×
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
its own 25 weights
pixels under it
Σ → ReLU →
0.0
Each window is its own neuron with its own 25 weights. Stride 2: the next neuron's window starts 2 pixels over.
output (12 × 12): one value per neuron
Assumption 1: locality
Smallest: 5 × 5 windows, stride 2, one neuron per window
input X (28 × 28)
NEURON 2 OF 144 · WINDOW ROW 0, COLUMN 1
−0.8
−0.2
0
−0.5
−0.4
−0.6
+1
+1
−0.6
−0.9
−0.3
−0.3
−0.5
+0.7
+0.1
+0.5
+0.3
+0.7
−0.9
+0.3
−0.3
−0.4
−0.4
+0.6
+0.2
×
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
its own 25 weights
pixels under it
Σ → ReLU →
0.0
Each window is its own neuron with its own 25 weights. Stride 2: the next neuron's window starts 2 pixels over.
output (12 × 12): one value per neuron
Assumption 1: locality
Smallest: 5 × 5 windows, stride 2, one neuron per window
input X (28 × 28)
NEURON 3 OF 144 · WINDOW ROW 0, COLUMN 2
+0.4
−0.3
+0.9
−0.5
−0.5
0
−0.7
+0.1
+0.9
+0.5
+0.5
+0.3
0
+0.4
+0.8
−0.4
−0.1
+0.8
−0.8
0
+0.9
+0.3
+0.3
−0.4
+0.6
×
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
its own 25 weights
pixels under it
Σ → ReLU →
0.0
Each window is its own neuron with its own 25 weights. Stride 2: the next neuron's window starts 2 pixels over.
output (12 × 12): one value per neuron
Assumption 1: locality
Smallest: 5 × 5 windows, stride 2, one neuron per window
input X (28 × 28)
STRONGEST: NEURON 103 OF 144 · WINDOW ROW 8, COLUMN 6
0
+0.6
+0.5
+0.8
+0.5
+0.2
+0.3
+0.3
−0.4
+0.7
−0.9
−0.1
+0.6
+0.1
+0.8
+0.9
+0.9
+0.7
+0.4
+0.7
+0.4
−0.8
0
+0.7
+0.5
×
0.0
0.0
0.3
1.0
1.0
0.0
0.0
0.8
1.0
1.0
0.0
0.6
1.0
0.9
0.4
0.3
1.0
1.0
0.9
0.0
0.9
1.0
0.9
0.2
0.0
its own 25 weights
pixels under it
Σ → ReLU →
5.0
Each window is its own neuron with its own 25 weights. Stride 2: the next neuron's window starts 2 pixels over.
output (12 × 12): one value per neuron
Still has addresses. 144 neurons, 144 separate sets of 25 weights. A stroke one neuron learns is no help to its neighbour.
Locality: some input to some neurons. Fewer weights each, but every neuron still learns on its own.
Assumption 2: Features are spatially invariant
A stroke anywhere is the same stroke, so every unit can use the same 25 weights
unit (6, 9)
unit (8, 15)
unit (17, 12)
Three units, three separate sets of 25 weights, each learned only from ink at its own spot.
WEIGHTS FOR ONE 24 × 24 MAP OF UNITS
Fully connected
4.5 × 10⁵
Local 5 × 5 patches
1.4 × 10⁴
Assumption 2: Features are spatially invariant
A stroke anywhere is the same stroke, so every unit can use the same 25 weights
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
unit (6, 9)
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
unit (8, 15)
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
unit (17, 12)
One set of 25 weights, W, used by every unit. A stroke detector learned anywhere works everywhere.
WEIGHTS FOR ONE 24 × 24 MAP OF UNITS
Fully connected
4.5 × 10⁵
Local 5 × 5 patches
1.4 × 10⁴
Local and shared
25
Assumption 2: Features are spatially invariant
A stroke anywhere is the same stroke, so every unit can use the same 25 weights
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
unit (6, 9)
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
unit (8, 15)
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
unit (17, 12)
One set of 25 weights, W, used by every unit. A stroke detector learned anywhere works everywhere.
WEIGHTS FOR ONE 24 × 24 MAP OF UNITS
Fully connected
4.5 × 10⁵
Local 5 × 5 patches
1.4 × 10⁴
Local and shared
25
THIS IS A CONVOLUTION
ŷij = Σa Σb Wab Xi+a, j+b
a, b = 0…4: one filter W, slid to every position (i, j). Deep learning calls it convolution; strictly it is cross-correlation, since W is not flipped.
Sharing isn't a trick to save weights. It is the assumption that a stroke means the same thing everywhere.
Shared weights for every neuron
The same windows as before, but now every neuron shares one set of 25 weights
input X (28 × 28)
NEURON 1 OF 144 · WINDOW ROW 0, COLUMN 0
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
×
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
the same 25 weights W
pixels under it
Σ → ReLU →
0.0
Every window is a neuron, and every neuron uses the same W: −1 above, +1 below, a top-edge detector. Same windows and stride as the last slide.
feature map (12 × 12): one value per neuron
Shared weights for every neuron
The same windows as before, but now every neuron shares one set of 25 weights
input X (28 × 28)
NEURON 2 OF 144 · WINDOW ROW 0, COLUMN 1
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
×
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
the same 25 weights W
pixels under it
Σ → ReLU →
0.0
Every window is a neuron, and every neuron uses the same W: −1 above, +1 below, a top-edge detector. Same windows and stride as the last slide.
feature map (12 × 12): one value per neuron
Shared weights for every neuron
The same windows as before, but now every neuron shares one set of 25 weights
input X (28 × 28)
NEURON 3 OF 144 · WINDOW ROW 0, COLUMN 2
−1
−1
−1
−1
−1
−1
−1
−1
−1
−1
0
0
0
0
0
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
×
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
the same 25 weights W
pixels under it
Σ → ReLU →
0.0
Every window is a neuron, and every neuron uses the same W: −1 above, +1 below, a top-edge detector. Same windows and stride as the last slide.
feature map (12 × 12): one value per neuron
Shared weights for every neuron
The same windows as before, but now every neuron shares one set of 25 weights
input X (28 × 28)