What did your MLP actually learn?
Generalization, the shift test, and data augmentation
01
Over- and underfitting, as curves and a map
02
Double descent and the deep learning twist
03
A stress test your 97% model fails
04
Why it fails, and an experiment that confirms it
05
Data augmentation: the standard fix, and its cost
All slides, one per page
© 2026 Dr. Jesse Roberts
Recap: what over- and underfitting look like
Error on training and test digits as training runs; one-hidden-layer MLPs, real MNIST
Underfit · 2 hidden units, 8,000 digits
Overfit · 512 hidden units, 500 digits
100%
50%
0%
30%
15%
0%
0
20
40
60
0
20
40
60
80
epoch
epoch
test 39.5%
train 36.8%
test 24.6%
train 0%
best: 19.6%, epoch 4
Both errors stay high and close together.
More training will not help: the model class is not appropriate to the problem.
Training error keeps falling to 0 while test error rises from its best at epoch 4.
The overfitting signal is test error rising while training error keeps improving.
Reading the curves
Judge the trend over training, not a single snapshot
Underfit
Overfit
traintest
Underfitting
Overfitting
Train error
Stays high
Keeps falling, often to 0
Test error
High, tracking train error
Rises while train error improves
Cause
High bias: the model class cannot express the solution
High variance: the model is too flexible for the data
Fix
A more capable or better-suited model class
More data, regularization (dropout, weight decay), early stopping, less capacity
DEFINITION TO CARRY FORWARD
A model generalizes when it has low error on a held-out test set drawn from the same distribution as training.
A high test error alone is ambiguous; the direction it moves tells you which problem you have.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Each dot is one trained line


Press Next run.
One dot per trained model


16 weights, 10 points: many exact fits.
No model in the class fits the training data: train and test error both stay high.
The class contains every model that fits, good and bad. Training data alone may not pick a good one.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 1
train MSE 0.109
test MSE 0.085
High both times: no line bends.
One dot per trained model


16 weights, 10 points: many exact fits.
1
1 run: train error high and test error high, every time. More runs will not help.
The class contains every model that fits, good and bad. Training data alone may not pick a good one.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 2
train MSE 0.085
test MSE 0.070
High both times: no line bends.
One dot per trained model


16 weights, 10 points: many exact fits.
1
2
2 runs: train error high and test error high, every time. More runs will not help.
The class contains every model that fits, good and bad. Training data alone may not pick a good one.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 3
train MSE 0.086
test MSE 0.069
High both times: no line bends.
One dot per trained model


16 weights, 10 points: many exact fits.
1
2
3
3 runs: train error high and test error high, every time. More runs will not help.
The class contains every model that fits, good and bad. Training data alone may not pick a good one.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
One dot per trained model


16 weights, 10 points: many exact fits.
1
2
3
4
4 runs: train error high and test error high, every time. More runs will not help.
The class contains every model that fits, good and bad. Training data alone may not pick a good one.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 1: fails the test
train MSE 0.000
test MSE 0.215
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
4 runs: train error high and test error high, every time. More runs will not help.
1 run: it fits the training data, but not the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 2: fails the test
train MSE 0.000
test MSE 0.256
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
4 runs: train error high and test error high, every time. More runs will not help.
2 runs: every one fits the training data; only 0 also fit the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 3: fails the test
train MSE 0.000
test MSE 0.557
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4 runs: train error high and test error high, every time. More runs will not help.
3 runs: every one fits the training data; only 0 also fit the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 4: fits test too
train MSE 0.000
test MSE 0.037
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
4 runs: train error high and test error high, every time. More runs will not help.
4 runs: every one fits the training data; only 1 also fits the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 5: fails the test
train MSE 0.000
test MSE 0.304
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
5
4 runs: train error high and test error high, every time. More runs will not help.
5 runs: every one fits the training data; only 1 also fits the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 6: fails the test
train MSE 0.000
test MSE 0.834
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
5
6
4 runs: train error high and test error high, every time. More runs will not help.
6 runs: every one fits the training data; only 1 also fits the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 7: fails the test
train MSE 0.000
test MSE 0.227
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
5
6
7
4 runs: train error high and test error high, every time. More runs will not help.
7 runs: every one fits the training data; only 1 also fits the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 8: fails the test
train MSE 0.000
test MSE 1.079
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
5
6
7
8
4 runs: train error high and test error high, every time. More runs will not help.
8 runs: every one fits the training data; only 1 also fits the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 9: fits test too
train MSE 0.000
test MSE 0.055
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
5
6
7
8
9
4 runs: train error high and test error high, every time. More runs will not help.
9 runs: every one fits the training data; only 2 also fit the test data.
An overparameterized class contains models that generalize and models that do not.
Why does this happen? Model space

Classical overfitting explanation: Training picks a well-fit, arbitrary model from the class with all models being possible.
Underfit: the class misses the data
Overfit risk: the class is too large
All possible models
All possible models
Fits the training data
Fits the training data
Also fits the test data
Also fits the test data
Chosen model class
Chosen model class
fits train,
fails test
A straight line,
fit to curved data
A degree-15 polynomial,
fit to straight-line data
train test model
train test model
Run 4
train MSE 0.083
test MSE 0.066
High both times: no line bends.
Run 10: fails the test
train MSE 0.000
test MSE 0.297
A plain line: 0.031. Dashed: true line.
1
2
3
4
1
2
3
4
5
6
7
8
9
10
4 runs: train error high and test error high, every time. More runs will not help.
10 runs: every one fits the training data; only 2 also fit the test data.
An overparameterized class contains models that generalize and models that do not.
Deep nets: a class that covers almost everything
?
Same map, but now the chosen model class is a large neural network
All possible models
Fits the training data
Also fits the test data
Chosen class: deep net
A
B
C
How big is this class?
A wide enough network can approximate any continuous function on a bounded input space.
Universal approximation: Cybenko 1989; Hornik et al. 1989
And the data is small by comparison
Our residual MLP: 4.7 × 10⁵ parameters, 8.0 × 10³ training images.
YOUR PREDICTION
Where in the model space do you expect a trained deep neural net to be?
AFits the training data, not the test data
BFits both the training and the test data
CFits neither
Yet on real labels, it works
4.7 × 10⁵
parameters
8.0 × 10³
training images: about 58 parameters per image
96%
test accuracy, with 100% training accuracy
The answer to the prediction is B: out of a class that covers almost everything, training landed among the models that also fit the test data.
Most models in this class that fit the training data would fail the test. Training somehow avoided them.
Far more parameters than training images, and it still generalizes.
Reaction image: a man in surgical scrubs and a hair cap, pulling down his mask, looking puzzled. Caption: But why?
A hypothesis: the data is too hard to memorize
Maybe the class is smaller in practice than it looks on the map
label 3
label 8
label 1
label 0
label 4
label 0
6 of the 8,000 training images. Each is 784 pixel values, and every one must be mapped to its label.
HYPOTHESIS
Storing 8,000 separate image-to-label answers is too hard for this network. The only way it can fit the training data is to find patterns shared across many images, and shared patterns also work on new images.
If it is true
The huge class is not really a problem: in practice the network can only reach models that capture structure, so it lands in B.
A hypothesis is only useful if some experiment could prove it wrong.
A test: shuffle the labels
?
Same 8,000 images, same number of each label; only the pairing of image and label is scrambled
true 3 given 9
true 8 given 7
true 1 given 8
true 0 given 3
true 4 given 7
true 0 given 4
Why this test tells us something
No rule links image to label anymore: the shuffled labels match the true ones on 9.6% of images, which is chance.
That leaves training accuracy as a pure measure of memorization: can the class store 8,000 arbitrary answers?
Two possible outcomes
Training accuracy stays low: the class cannot memorize. Its limited size could be what makes it generalize.
Training accuracy reaches ~100%: the class can memorize anything. Its size cannot be what makes it generalize.
Predict before the next slide: which outcome will we see?
It can memorize noise
Same MLP, same 8,000 images; one run with true labels, one with labels shuffled at random
100%
75%
50%
25%
0%
0
20
40
60
epoch
true labels: train 100%, test 96%
random labels: train
random labels: test ≈ chance
Solid: training accuracy · dashed: test accuracy · dotted: chance (10%)
99%
of 8,000 meaningless labels memorized by epoch 60. Test accuracy: 9.7%, chance level.
Memorizing noise takes longer: 70% at epoch 10, where true labels are already at 99%. But it gets there.
Standard image networks memorize random labels on CIFAR-10 and ImageNet, and even pure-noise images on CIFAR-10.
Zhang et al. (2017), "Understanding deep learning requires rethinking generalization"
The hypothesis was wrong. So, why do deep nets generalize?
All possible models
Fits the training data
Also fits the test data
Class: p = 50 hidden units
Hypothesis 2: maybe the classical picture is wrong
An experiment where we control the size of the class: one hidden layer, more and more hidden units
Each added hidden unit adds new functions the model can express: the chosen class grows to take in more of the space.
The model
V: random, never trainedW: fit by OLS784 pixelsp = 50 hidden units10 scores
The first layer is random and never trained. Only W is fit, exactly, by ordinary least squares (OLS): the optimal linear combination of the p hidden units.
Classical picture: training picks any model that fits. A bigger class holds more models that fit the training data but fail the test, so past some size test error only rises.
Hypothesis 2: the classical picture is wrong. Training does not pick arbitrarily, so test error could fall even as the class grows.
All possible models
Fits the training data
Also fits the test data
Class: p = 100 hidden units
Hypothesis 2: maybe the classical picture is wrong
An experiment where we control the size of the class: one hidden layer, more and more hidden units
Each added hidden unit adds new functions the model can express: the chosen class grows to take in more of the space.
The model
V: random, never trainedW: fit by OLS784 pixelsp = 100 hidden units10 scores
The first layer is random and never trained. Only W is fit, exactly, by ordinary least squares (OLS): the optimal linear combination of the p hidden units.
Classical picture: training picks any model that fits. A bigger class holds more models that fit the training data but fail the test, so past some size test error only rises.
Hypothesis 2: the classical picture is wrong. Training does not pick arbitrarily, so test error could fall even as the class grows.
All possible models
Fits the training data
Also fits the test data
Class: p = 400 hidden units
Hypothesis 2: maybe the classical picture is wrong
An experiment where we control the size of the class: one hidden layer, more and more hidden units
Each added hidden unit adds new functions the model can express: the chosen class grows to take in more of the space.
The model
V: random, never trainedW: fit by OLS784 pixelsp = 400 hidden units10 scores
The first layer is random and never trained. Only W is fit, exactly, by ordinary least squares (OLS): the optimal linear combination of the p hidden units.
Classical picture: training picks any model that fits. A bigger class holds more models that fit the training data but fail the test, so past some size test error only rises.
Hypothesis 2: the classical picture is wrong. Training does not pick arbitrarily, so test error could fall even as the class grows.
All possible models
Fits the training data
Also fits the test data
Class: p = 1,000 hidden units
Hypothesis 2: maybe the classical picture is wrong
An experiment where we control the size of the class: one hidden layer, more and more hidden units
Each added hidden unit adds new functions the model can express: the chosen class grows to take in more of the space.
The model
V: random, never trainedW: fit by OLS784 pixelsp = 1,000 hidden units10 scores
The first layer is random and never trained. Only W is fit, exactly, by ordinary least squares (OLS): the optimal linear combination of the p hidden units.
Classical picture: training picks any model that fits. A bigger class holds more models that fit the training data but fail the test, so past some size test error only rises.
Hypothesis 2: the classical picture is wrong. Training does not pick arbitrarily, so test error could fall even as the class grows.
All possible models
Fits the training data
Also fits the test data
Class: p = 10,000 hidden units
Hypothesis 2: maybe the classical picture is wrong
An experiment where we control the size of the class: one hidden layer, more and more hidden units
Each added hidden unit adds new functions the model can express: the chosen class grows to take in more of the space.
The model
V: random, never trainedW: fit by OLS784 pixelsp = 10,000 hidden units10 scores
The first layer is random and never trained. Only W is fit, exactly, by ordinary least squares (OLS): the optimal linear combination of the p hidden units.
Classical picture: training picks any model that fits. A bigger class holds more models that fit the training data but fail the test, so past some size test error only rises.
Hypothesis 2: the classical picture is wrong. Training does not pick arbitrarily, so test error could fall even as the class grows.
All possible models
Fits the training data
Also fits the test data
Class: p = 100,000 hidden units
Hypothesis 2: maybe the classical picture is wrong
An experiment where we control the size of the class: one hidden layer, more and more hidden units
Each added hidden unit adds new functions the model can express: the chosen class grows to take in more of the space.
The model
V: random, never trainedW: fit by OLS784 pixelsp = 100,000 hidden units10 scores
The first layer is random and never trained. Only W is fit, exactly, by ordinary least squares (OLS): the optimal linear combination of the p hidden units.
Classical picture: training picks any model that fits. A bigger class holds more models that fit the training data but fail the test, so past some size test error only rises.
Hypothesis 2: the classical picture is wrong. Training does not pick arbitrarily, so test error could fall even as the class grows.
The classical picture: a U-curve
?
Test error (orange) and training error (blue) as the model grows
p = n
fewer hidden units than training digits
100%
75%
50%
25%
0%
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · training digits n = 1,000
classical best 17.0%
87.9% at p = n
p = 50 hidden units
train error 24.1%
test error 30.5%
The model: MLP, p hidden units
V: random, never trainedW: trained784 pixelsp = 50 hidden units10 scoresimage xh = ReLU(Vx)Wh → argmax
Training: only W, solved exactly
Ordinary least squares (OLS) finds the optimal linear combination of the p hidden units for the one-hot labels of n = 1,000 digits, in one step, no gradient descent. If many W fit perfectly, OLS returns the one with the smallest weights.
YOUR PREDICTION
What will happen to test error if we keep adding hidden units past p = n?
Real MNIST: 1,000 training and 2,000 test digits; p from 50 to 10,000
The classical picture: a U-curve
?
Test error (orange) and training error (blue) as the model grows
p = n
fewer hidden units than training digits
100%
75%
50%
25%
0%
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · training digits n = 1,000
classical best 17.0%
87.9% at p = n
p = 100 hidden units
train error 11.8%
test error 22.4%
The model: MLP, p hidden units
V: random, never trainedW: trained784 pixelsp = 100 hidden units10 scoresimage xh = ReLU(Vx)Wh → argmax
Training: only W, solved exactly
Ordinary least squares (OLS) finds the optimal linear combination of the p hidden units for the one-hot labels of n = 1,000 digits, in one step, no gradient descent. If many W fit perfectly, OLS returns the one with the smallest weights.
YOUR PREDICTION
What will happen to test error if we keep adding hidden units past p = n?
Real MNIST: 1,000 training and 2,000 test digits; p from 50 to 10,000
The classical picture: a U-curve
?
Test error (orange) and training error (blue) as the model grows
p = n
fewer hidden units than training digits
100%
75%
50%
25%
0%
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · training digits n = 1,000
classical best 17.0%
87.9% at p = n
p = 400 hidden units
train error 0.6%
test error 17.0%
The model: MLP, p hidden units
V: random, never trainedW: trained784 pixelsp = 400 hidden units10 scoresimage xh = ReLU(Vx)Wh → argmax
Training: only W, solved exactly
Ordinary least squares (OLS) finds the optimal linear combination of the p hidden units for the one-hot labels of n = 1,000 digits, in one step, no gradient descent. If many W fit perfectly, OLS returns the one with the smallest weights.
YOUR PREDICTION
What will happen to test error if we keep adding hidden units past p = n?
Real MNIST: 1,000 training and 2,000 test digits; p from 50 to 10,000
The classical picture: a U-curve
?
Test error (orange) and training error (blue) as the model grows
p = n
fewer hidden units than training digits
100%
75%
50%
25%
0%
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · training digits n = 1,000
classical best 17.0%
87.9% at p = n
p = 800 hidden units
train error 0.0%
test error 32.0%
The model: MLP, p hidden units
V: random, never trainedW: trained784 pixelsp = 800 hidden units10 scoresimage xh = ReLU(Vx)Wh → argmax
Training: only W, solved exactly
Ordinary least squares (OLS) finds the optimal linear combination of the p hidden units for the one-hot labels of n = 1,000 digits, in one step, no gradient descent. If many W fit perfectly, OLS returns the one with the smallest weights.
YOUR PREDICTION
What will happen to test error if we keep adding hidden units past p = n?
Real MNIST: 1,000 training and 2,000 test digits; p from 50 to 10,000
The classical picture: a U-curve
?
Test error (orange) and training error (blue) as the model grows
p = n
fewer hidden units than training digits
100%
75%
50%
25%
0%
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · training digits n = 1,000
classical best 17.0%
87.9% at p = n
p = 1,000 hidden units
train error 0.0%
test error 87.9%
The model: MLP, p hidden units
V: random, never trainedW: trained784 pixelsp = 1,000 hidden units10 scoresimage xh = ReLU(Vx)Wh → argmax
Training: only W, solved exactly
Ordinary least squares (OLS) finds the optimal linear combination of the p hidden units for the one-hot labels of n = 1,000 digits, in one step, no gradient descent. If many W fit perfectly, OLS returns the one with the smallest weights.
YOUR PREDICTION
What will happen to test error if we keep adding hidden units past p = n?
Real MNIST: 1,000 training and 2,000 test digits; p from 50 to 10,000
Double descent: past the U-curve
?
Test error (orange) and training error (blue) as the model grows
p = n
fewer hidden units than training digits
more hidden units than training digits
100%
75%
50%
25%
0%
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · training digits n = 1,000
classical best 17.0%
87.9% at p = n
8.2%
The classical assumption
Training picks any model that fits. More parameters mean more ways to fit badly, so past the sweet spot test error only gets worse.
What actually happens
Test error spikes at p = n, then falls below the classical best: 17.0% → 8.2%.
Belkin et al. (2019); in deep nets, by width and by training time: Nakkiran et al. (2019)
QUESTION
Past p = n, every model fits the training data perfectly. So why does test error go down?
The classical assumption is wrong: training does not pick uniformly among the models that fit.
Maybe explicit regularization causes generalization?
Two tests on the same residual MLP: take weight decay away, then turn it up tenfold
TAKE IT AWAY · TRUE LABELS
If regularization made it generalize, removing it should hurt
96.1%
test, weight decay on
96.3%
test, weight decay off
No loss at all. It generalizes without it.
TURN IT UP 10× · RANDOM LABELS
If regularization prevented memorizing, noise should stop fitting
99.5%
train, random labels
10.8%
test (chance: 10%)
It still memorizes the noise.
At scale, the same holds for dropout and data augmentation: on CIFAR-10 and ImageNet they add a few points of test accuracy, but networks fit random labels with them on and generalize with them off.
Zhang et al. (2017): explicit regularization "is neither necessary nor by itself sufficient" for generalization.
What guides training toward fits that generalize?
?
Talk it over with a neighbour before we move on
The class allows bad fits. Past p = n, countless models fit the training data and fail the test.
No penalty asks for good ones. Taking away weight decay changed nothing.
Yet training keeps landing on them. Test error falls as the class grows.
“He intends only his own gain, and he is in this, as in many other cases, led by an invisible hand to promote an end which was no part of his intention.”
Adam Smith, The Wealth of Nations (1776), Book IV, Chapter 2
QUESTION
Each step of training only lowers the training loss. What is the invisible hand that steers it toward fits that generalize?
The optimization method guides the selection
Implicit regularization in its clearest form: size (norm) of the fitted weights in the double descent experiment
p = n
1,000
100
10
1
50
100
400
1,000
3,000
10,000
hidden units p (log scale) · weight norm (log scale) · n = 1,000
1,041 at p = n
5.1
2.7
At p = n exactly one set of weights fits all 1,000 digits. To hit every point it must be huge: norm 1,041, test error 88%.
Past p = n infinitely many weight settings fit exactly. Starting from zero, a weight only grows or shrinks away from zero if that helps performance, so training ends at the smallest-weight fit.
More hidden units make an even smaller fit available: norm 2.7 at p = 10,000, smoother, and test error down to 8.2%.
No penalty term anywhere, yet gradient descent chooses. This is implicit regularization. Ridgeless least squares: Hastie et al. (2022); Bartlett et al. (2020).
Starting from zero, training prefers small-weight solutions. But what about a deep net with random initialization?
Setup: make 20% of the training labels wrong
?
Talk it over with a neighbour before we move on
label 3
label 8
label 1
label 0
label 7
Training set: 8,000 digits. 80% keep their true label; 20% get a random wrong label.
Test set: 2,000 digits, all with true labels.
Model: the same residual MLP, trained for 60 epochs. We track the two training groups separately.
Sketch your prediction: accuracy over 60 epochs
100%
50%
0%
0
60
epoch
train, correct labels train, wrong labels test
QUESTION
The network can memorize anything. Which group does it fit first, and what happens to test accuracy as it trains?
Gradient descent fits shared structure first
The same idea with gradient descent: 8,000 digits, 20% given a wrong label, each group tracked separately
100%
75%
50%
25%
0%
0
20
40
60
epoch
train, correctly labeled digits
train, wrongly labeled digits
test accuracy
Ring: best test accuracy, 92.6% at epoch 3. "Train" accuracy is against the labels the network was given.
Arpit et al. (2017); coherent gradients: Chatterjee (2020); GPT-3: Brown et al. (2020)
EPOCH 1
92% of correctly labeled digits fit, but only 2% of wrongly labeled ones: it predicts the true digit.
EPOCH 3
Test accuracy peaks at 92.6%. Stop here (early stopping) and the noise is never memorized.
EPOCH 60
Wrong labels memorized (99%); test falls to 81%.
Correctly labeled 7s all push the weights the same way, so their gradients add up. A mislabeled digit pushes alone.
Used at scale: large language models are typically trained for about one pass over their data, so most examples are seen once. GPT-3 saw its main web corpus 0.44 times.
Gradient descent follows loud, coherent gradients first. Stop early and the noise is never fit.
Putting it together: implicit regularization
The class is huge, but training does not pick at random from the models that fit
All possible models
Fits the training data
Where training tends to land
Also fits the test data
Chosen class: deep net
your MLP
fits train,
fails test
1
Many models fit. A deep net can fit the training data countless ways, even pure noise.
2
Training picks one. No penalty decides, so the procedure does: architecture, initialization, optimizer.
3
It picks a simple one. Linear case: the smallest weights. Deep nets: shared structure before noise (after epoch 1, 92% of correctly labeled digits fit vs 2% of mislabeled).
4
Simple fits tend to fit the test data too, when test digits come from the same distribution.
Still debated: optimizer or architecture? Full-batch gradient descent can match SGD (Geiping et al. 2022), and even randomly sampled networks that fit tend to generalize (Chiang et al. 2023).
BUT SIMPLE BY WHOSE STANDARD?
Nothing in this preference says a 7 moved three pixels is still a 7. Let's test it.
Generalization comes from which fit training picks, not how many fits the class contains.
Generalize beyond training distribution
?
Forget the network for a moment. You are the classifier.
as in the dataset
1. What digit is this?
Generalize beyond training distribution
?
Forget the network for a moment. You are the classifier.
as in the dataset
moved 3 pixels right
1. What digit is this?
2. Moved 3 pixels right. And now?
Generalize beyond training distribution
?
Forget the network for a moment. You are the classifier.
as in the dataset
moved 3 pixels right
1. What digit is this?
2. Moved 3 pixels right. And now?
YOU NEVER TRAINED ON SHIFTED 7s
You knew anyway, because you learned what a 7 is. A model that truly generalized should know just as you did.
WHAT WE HAVE BEEN MEASURING
Low error on held-out digits from the same distribution as training.
WHAT WE ACTUALLY WANT
Correct on any input a person would label the same way: the task, not just the dataset.
So does our MLP know? First: what has it actually been trained and tested on?
MNIST data is limited to a centered view
MNIST was built so that position barely varies, in training and in testing
Average of 10,000 digits. Dashed: the central 20×20 box.
HOW MNIST WAS BUILT
Each digit was size-normalized to fit a 20×20 box, then centered by its center of mass in the 28×28 frame.
96.7%
of all ink lies inside the central 20×20 box
111
pixels are blank in every one of the 10,000 digits
Train and test digits put their ink in the same place. Whatever the model does with the edges of the frame is almost never trained and almost never tested.
A shifted 7 lands in territory the model has barely seen. What does it do there?
Stress test: move every digit 3 pixels right
?
Your trained residual MLP scores ~97% on the MNIST test set. No retraining: same model, shifted test digits.
test digit, as is
shifted 3 px right
Every test digit moves 3 pixels right. What will the accuracy be?
A~95%
B~80%
C~50%
D~10%
A human reads both images as a 7 without hesitating.
The answer: accuracy collapses
Same trained model, same 2,000 test digits, each shifted right with zero fill
100%
75%
50%
25%
0%
0
1
2
3
4
5
6
shift right (pixels)
95.6%
45%
8%
chance (10%)
One test 7, shifted
0 px
reads 7
2 px
reads 7
3 px
reads 3
4 px
reads 0
Accuracy halves at 3 px and reaches chance by 6 px.
Why it fails: every weight has an address
What training changed in four first-layer units: a row of W, after minus before, as a 28×28 image
unit 12
unit 158
unit 4
unit 255
weight increased
weight decreased
unchanged
Each unit responds to ink at fixed positions. The same stroke two pixels over lands on different weights.
A feature learned at (10, 12) gives no help at (12, 12). The model must learn every feature separately at every position.
THE WHITE BORDER
117 / 784
input pixels never contain ink in the 8,000 training digits.
Their weights moved by at most 2 × 10⁻⁵, against up to 0.2 elsewhere.
A stroke shifted onto them meets untrained, random weights.
The MLP learned "ink here means 7," not "this shape means 7."
If location doesn't matter, shuffle the pixels
?
If the MLP ignores spatial arrangement, pixel order should not matter to it
original
each pixel to a fixed new spot
Unreadable to a person. Train the same MLP from scratch on shuffled images; test on shuffled images.
Normal pixels
95.6%
seeds: 94.8–96.1%
Shuffled pixels
94.7%
seeds: 94.1–95.7%
Same accuracy, within seed-to-seed noise. The explanation holds: the MLP never used which pixels are neighbors.
LEAVE THIS OPEN
Is ignoring pixel order a strength or a weakness?
To an MLP, an image is a list of 784 unrelated numbers.
If location doesn't matter, shuffle the pixels
?
If the MLP ignores spatial arrangement, pixel order should not matter to it
original
each pixel to a fixed new spot
Unreadable to a person. Train the same MLP from scratch on shuffled images; test on shuffled images.
Normal pixels
95.6%
seeds: 94.8–96.1%
Shuffled pixels
94.7%
seeds: 94.1–95.7%
Same accuracy, within seed-to-seed noise. The explanation holds: the MLP never used which pixels are neighbors.
LEAVE THIS OPEN
Is ignoring pixel order a strength or a weakness?
To an MLP, an image is a list of 784 unrelated numbers.
If location doesn't matter, shuffle the pixels
?
If the MLP ignores spatial arrangement, pixel order should not matter to it
original
each pixel to a fixed new spot
The same permutation, applied to every image
Pixel 1 always goes to the same spot, pixel 2 to its spot, and so on, for every training and test image.
Unreadable to a person. Train the same MLP from scratch on shuffled images; test on shuffled images.
Normal pixels
95.6%
seeds: 94.8–96.1%
Shuffled pixels
94.7%
seeds: 94.1–95.7%
Same accuracy, within seed-to-seed noise. The explanation holds: the MLP never used which pixels are neighbors.
LEAVE THIS OPEN
Is ignoring pixel order a strength or a weakness?
To an MLP, an image is a list of 784 unrelated numbers.
Data augmentation: teaching invariance
For failures along a known, systematic axis: shift, crop, rotation, scale, flip, color
x 0, y 0
label 7
x +2, y -1
label 7
x -3, y +1
label 7
x +1, y +3
label 7
x -2, y -2
label 7
x +3, y 0
label 7
x -1, y +2
label 7
x +2, y +2
label 7
What it does
Each time an image is drawn for a batch, apply a random transformation and keep the label. One 7 becomes every shifted 7.
What it says
"The label does not change under this transformation." An invariance, written into the data instead of the model.
When to use it
When you can name the axis a model fails on. Each axis is its own transformation, taught separately.
Shown: one real test 7 under eight random translations, pad 3 and re-crop (±3 px on each axis).
Augmentation encodes an invariance through the data.
Augmentation changes the data, not the class
Original training and test data
With shifted digits in both
All possible models
All possible models
Fits the training data
fit the original data
Fits augmented training data
Also fits the test data
Fit the original test data
Fits the augmented data
Fits the augmented test data
Fits the augmented test data
Chosen model class
Chosen model class: same
Most runs also fit the test data, but almost none fit the augmented data, let alone the augmented test data.
The augmented data is now the training data, so runs can only land there, and most also fit the augmented test data.
Changing the data shrinks the set of models that fit.
Demo: retrain with random shifts
Same MLP, same 8,000 digits; each batch padded 3 px and randomly re-cropped
100%
75%
50%
25%
0%
0
1
2
3
4
5
6
shift right (pixels)
trained with shifts (±3 px)
original
range seen in training
test condition
original
shifts
No shift
95.6%
95.9%
3 px right
45%
94%
Random ±2 px, both axes
72%
96%
6 px right (never seen)
8%
60%
ALSO A REGULARIZER
Train/test gap: 4.2 → 1.6 points
Mean of 3 seeds, 2,000 test digits.
Robust inside the range it was shown, fading beyond it.
What else can vary without changing the digit?
?
Shift was one invariance. Before the next step, name others.
Still a 7 if we…?
YOUR TURN
List transformations that leave the label unchanged. Each one is an axis the MLP has to be taught.
Every invariance you can name is a candidate for augmentation.
What else can vary without changing the digit?
?
Shift was one invariance. Before the next step, name others.
Still a 7 if we…?
YOUR TURN
List transformations that leave the label unchanged. Each one is an axis the MLP has to be taught.
Scale
A bigger or smaller 7
Small rotation
A slanted 7
Stroke thickness
A bolder or thinner pen
Small skew or stretch
Someone else's handwriting
Every invariance you can name is a candidate for augmentation.
Scale: same failure, same fix
Test digits rescaled about the center; then the MLP retrained with random rescaling (0.8–1.2×)
100%
75%
50%
25%
0%
0.6×
0.8×
1.0×
1.2×
1.4×
digit scale
trained with random rescaling
original training
range seen in training
0.8×
0.9×
1.0×
1.1×
1.2×
test digits at
original
rescaled
0.8× (smaller)
79%
94%
1.0×
95.6%
96.0%
1.2× (larger)
86%
95%
1.4× (never seen)
59%
87%
Mean of 3 seeds, 2,000 test digits. Bilinear resampling, zero fill.
Same failure, same fix: one more axis, taught by showing every case.
Proposal: add random horizontal flips
?
Flips are a standard augmentation for photos. Each training digit below, mirrored left to right:
a 7 flipped
a 3 flipped
a 4 flipped
a 8 flipped
QUESTION
Train every digit with its label, sometimes flipped. Is this augmentation appropriate?
Proposal: add random horizontal flips
?
Flips are a standard augmentation for photos. Each training digit below, mirrored left to right:
a 7 not a 7
a 3 not a 3
a 4 not a 4
a 8 still an 8
QUESTION
Train every digit with its label, sometimes flipped. Is this augmentation appropriate?
No. A flipped digit is not the same digit. A mirrored 3 is not a 3, and a mirrored 7 or 4 is no digit at all. Augmentation claims the label survives; here it does not. An 8 happens to survive, but one rule has to hold for every class.
Contrast: on CIFAR-10 a mirrored cat is still a cat and a mirrored truck still a truck, so flips are standard there.
The catch: how many training images is that?
A shifted, shrunken, slightly rotated 7 is still a 7, so the axes must be covered in combination
Axis
Settings
Count
Original
MNIST training set
60,000
× Shift (crop)
−3 to +3 px, each axis (7 × 7)
× 49
× Scale
0.8× to 1.2×, steps of 0.1
× 5
× Rotation
−15° to +15°, steps of 5° (small only: 180° turns a 6 into a 9)
× 7
Total
1,715 versions of every digit
1.0 × 10⁸
Combinations multiply, they don't add. Add 3 stroke widths and it is 3.1 × 10⁸. Sampling one random version per image each epoch avoids storing them, but the model still has to see enough of this space: many more epochs.
What else it costs
Capacity spent on copies. The MLP still stores each feature separately at every position, size and angle it was shown.
Only the range shown. Trained on ±3 px: 94% at 3 px, 60% at 6 px. Trained on 0.8–1.2×: 87% at 1.4×.
One axis at a time. Every invariance you name is another factor in the count.
Augmentation works, but it teaches by showing every case rather than knowing the rule.
What this means for your project
Residual MLP on MNIST, CIFAR-10 and ImageNet: no convolutions, no extra data
ALLOWED
Data augmentation: it does not count as adding data. Expect it to be your biggest lever on CIFAR-10 and ImageNet.
TRY THIS WEEK
Run today's shift plot on your own MNIST model.
Compare train and test curves on CIFAR-10: is test error rising?
Add random crops and re-run both.
How far can MLPs go? Bachmann, Anagnostidis & Hofmann (NeurIPS 2023), "Scaling MLPs: A Tale of Inductive Bias": plain MLPs reach 95% on CIFAR-10, but only with large-scale pretraining and heavy augmentation, which this project rules out. Read it for how much compute it takes, not as a target.
On natural images, augmentation is how an MLP rules out solutions that should be systematically excluded.
Also correct
under shift
Not overfitting: the wrong solution
All possible models
Fits the training data
Also fits the test data
Chosen model class
your MLP
It passed our definition
95.6% on held-out digits from the same distribution. The validation score gave no warning.
The data allowed many solutions
Training picked one that depends on absolute pixel positions. It fits every centered digit we gave it.
The real task is bigger than the test set
A shifted 7 is still a 7. The i.i.d. test set never asks that question, so it cannot catch the failure.
Generalizing on the test set is not the same as learning the task.
Why CIFAR-10 and ImageNet hurt so much more
Only MNIST lines its objects up for you
MNIST
CIFAR-10
ImageNet
Images
28×28, grayscale
32×32, color
Full photos, variable size
Size-normalized
Yes, to a 20×20 box
No
No
Centered
Yes, by center of mass
No
No
What was required
One handwritten digit
"One prominent instance" of the class; occlusion and unusual viewpoints allowed
A photo containing the class, anywhere in the scene
CIFAR-10 labeling criteria: Krizhevsky (2009), Appendix C. "There is no canonical viewpoint or scale at which the objects appear": Hinton et al. (2012).
An MLP must learn each feature at every position and scale where it appears.
That multiplies the data and capacity it needs, and it is the gap you are fighting in the project.
On natural images, the shift problem is not a stress test; it is the test set.