Generative AI: Models that make new stuff
All slides, one per page
© 2026 Dr. Jesse Roberts
Round 1: join
Live activity page: in the exported deck this whole slide is the page itself, and it can be clicked
EMBEDDED PAGE
jessetnroberts.com/gen-vs-descrim/
Students scan the QR code and get a random role: generator (invent 5 new words) or discriminator (judge 10 words). Both learn the same mystery language from the same 16 example words.
Round 1: results
Live activity page: in the exported deck this whole slide is the page itself, and it can be clicked
EMBEDDED PAGE
jessetnroberts.com/gen-vs-descrim/?view=results
Live scores: share of invented words that belong vs. random letters, share of judgements right vs. a coin flip, how hard each group said it felt, the popular inventions, why words failed, and which words fooled the discriminators. Reveal the rule at the end.
Which job was harder?
Talk it over with a neighbour before we move on
?
SAME 16 WORDS, TWO JOBS
Generators
Invent 5 new words that belong to the language
Discriminators
Judge 10 words: belongs or impostor
BEFORE THE NEXT CLICK
Both groups saw exactly the same examples. Why was one job so much harder than the other?
CHECKING VS. PRODUCING
  • To judge, you test one word against what you have seen
  • To generate, you must produce a word that passes every check at once
  • and still isn’t a copy of an example
THE BASELINES
  • Guessing judges 50% right (a coin flip)
  • Random letters almost never make a valid word (under 1%)
Recognizing is easier than producing.
DISCRIMINATIVE
learns p̂(belongs | x): where the line between them is
Telling apart vs. making new
GENERATIVE
learns p̂(x): where the words live, then samples new ones (◇)
A generative model can also judge (is p̂(x) high?), but a discriminative model can’t make anything new.
A rule, or a feel?
Talk it over with a neighbour before we move on
?
BEFORE THE NEXT CLICK
When you invented or judged words, did you follow a rule you could write down, or a feel for what looked right?
THE HIDDEN LANGUAGE AS A FINITE-STATE MACHINE
consonantvowelconsonantbare nconsonantstartCVN
Double circles: a word may end there. After a vowel, an n can start a new syllable (n + vowel) or stand alone. This is follows a similar pattern to japanese!
RUN IT TO CHECK
Follow the arrows letter by letter. End on a double circle? It belongs. A discriminator.
RUN IT TO WRITE
Take any arrow at random, write its letter, stop on a double circle. A generator.
A rule set you can run forwards is already a generative model.
Grading a generator
Look back at the round 1 results: where did points go?
?
VALID
Fails when: a word breaks the rule (“tamekot”)
In real models: confident output that doesn’t fit the data, like an invented citation
NOVEL
Fails when: a word is copied from the examples
In real models: memorization: repeating training text word for word
DIVERSE
Fails when: everyone writes the same safe word
In real models: mode collapse: an image model that draws the same face every time
Each is easy alone; a good generator is valid, novel and diverse at once.
Round 2: join
Live activity page: in the exported deck this whole slide is the page itself, and it can be clicked
EMBEDDED PAGE
jessetnroberts.com/gen-vs-descrim/round2.html
Everyone plays on their own phone: tap to place a +, tap to place an =, press Check, and the computer says ✓ or ✗ from a secret two-part rule. Ten tries; three ✓ in a row wins; then they type their guess.
Round 2: players
Live activity page: in the exported deck this whole slide is the page itself, and it can be clicked
EMBEDDED PAGE
jessetnroberts.com/gen-vs-descrim/round2.html?view=players
Live grid of every player’s latest placement and their ✓ / ✗ so far. Rules stay hidden until Reveal the rules.
Round 2: summary
Live activity page: in the exported deck this whole slide is the page itself, and it can be clicked
EMBEDDED PAGE
jessetnroberts.com/gen-vs-descrim/round2.html?view=summary
The learning curve (share of ✓ by try number against the 25% from placing at random), where the = landed relative to the + for each rule, and everyone’s guesses.
Learning from ✓ and ✗ alone
In round 2, nobody ever showed you a correct example
GeneratorJudgesecret rulea new attemptonly ✓ or ✗
GANS (2014)
A generator network learns to make images only from a discriminator network’s verdicts. The discriminator learns from real images.
CHATBOTS
After learning from text, a model is tuned on people’s ratings of its answers (reinforcement learning from human feedback).
The judge knows the rule. The generator only ever sees the verdicts, and still learns where to put the =.
Feedback can stand in for examples, but it is a much weaker signal.
GANs: Goodfellow et al., 2014, “Generative Adversarial Nets”. RLHF: Ouyang et al., 2022 (InstructGPT).
What does one ✗ tell you?
Talk it over with a neighbour before we move on
?
+=✗
The rule had two parts: up or down, and left or right.
BEFORE THE NEXT CLICK
This try put the = below and to the right of the +, and got ✗. Which part was wrong?
WE CAN’T TELL
  • Up/down, left/right, or both: a ✗ doesn’t say which
  • One bit of feedback for a two-part rule: the credit-assignment problem
  • You had to design tries that test one part at a time
WHY NETWORKS TRAIN WITH GRADIENTS
A gradient says which way to move each weight, not just “wrong”. A GAN’s generator gets the discriminator’s gradient, not only its verdict.
The richer the feedback, the faster the learning.
An LLM plays round 1 at scale
Learn p̂(next token | context) from text, then sample one token at a time
CONTEXT
The cat sat on the ___
mat0.31floor0.18sofa0.12bed0.08roof0.05table0.04
Illustrative numbers. Sampling picks “mat” about 31% of the time, “floor” 18%, …
YOU, TODAY
AN LLM
Round 1: 16 example words → a feel for the language
Pretraining: trillions of tokens of text → a feel for human writing
Invent a word that belongs
Pick each next token by sampling from p̂
Round 2: ✓ / ✗ → found the rule
Tuning: people’s ratings → helpful answers
Same idea, much bigger distribution: learn it, then sample from it.
Takeaways
  • Discriminative: tell apart (p̂(belongs | x)). Generative: make new (learn p̂(x), then sample).
  • Producing is harder than recognizing: a new sample must pass every check at once.
  • A rule set you can run forwards, like a finite-state machine, is already a generative model.
  • Grade a generator on three things at once: valid, novel, diverse.
  • Feedback alone (✓ / ✗, ratings, a discriminator) can teach a generator, but slowly: it doesn’t say what was wrong.
  • An LLM learns p̂(next token | context) from text and samples one token at a time.
Generative AI: learn the distribution, then sample from it.
A tour of generative systems
PEOPLE WRITE THE RULES
1
Grammar1957
rules → sentences
RULES LEARNED BY COUNTING
2
Counting1948
letter-pair table → words
3
Texture1999
grow an image pixel by pixel
RULES LEARNED BY A NETWORK
4
Energy1982
settle into a stored pattern
5
LLM2018
next token, again and again
Then: the VAE, up close2013
decode random codes
From rules people write, to rules learned by counting, to rules learned by a network.
Every stop samples from a joint distribution
THE CHAIN OF CONDITIONALS
p̂(input, output)
  = p̂(input) · p̂(output | input)
p̂(x1 … xn)
  = p̂(x1) · p̂(x2 | x1) · … · p̂(xn | x1 … xn−1)
  = Π p̂(next piece | what came before)
Each step’s input is what came before; its output is the next piece. Sample it, append it, repeat.
ROUND 1’S LETTER PAIRS: THE COUNTS ARE A JOINT
Letters after “k”: e 4a 2u 2i 1(of 9; 118 pairs in all)
p̂(k, e) = 4/118   p̂(k) = 9/118
p̂(e | k) = p̂(k, e) / p̂(k) = 4/9
ONE WORD, ONE CONDITIONAL PER LETTER
p̂(makuton)
  = p̂(m | start) · p̂(a | m) · p̂(k | a) · …
  = 1/16 · 1/6 · 1/4 · 2/9 · 1/3 · 1/6 · 1/5 · 4/9
  = 1/349,920 ≈ 2.9 × 10⁻⁶
Tiny, and that’s fine: there are a lot of possible words. Generating it was 8 small samples.
SOME STOPS SAMPLE A HIDDEN CODE FIRST
p̂(z, x) = p̂(z) · p̂(x | z)
Input: random numbers z. Output: the example x. (The VAE, at the end)
Generating is sampling from p̂(input, output), one conditional at a time.
GrammarCountingTextureEnergyLLM
Stop 1: a grammar
Generative grammar: where the word “generative” comes from
THE RULES
S → NP VP
NP → Det N | Det Adj N
    | NP that VP ↺ calls itself
VP → V NP | V
Det → the | a
Adj → green | tiny | sleepy
N → cat | robot | student
V → sees | builds | follows | sleeps
ONE RANDOM RUN
  S
⇒ NP VP ×1
⇒ Det Adj N VP ×1/3
⇒ the Adj N VP ×1/2
⇒ the tiny N VP ×1/3
⇒ the tiny cat VP ×1/3
⇒ the tiny cat V NP ×1/2
⇒ the tiny cat builds NP ×1/4
⇒ the tiny cat builds Det N ×1/3
⇒ the tiny cat builds a N ×1/2
⇒ the tiny cat builds a robot ×1/3
More runs:the sleepy cat builds a green robotthe cat that sees a robot sleepsthe cat builds a robotthe student sleeps the tiny robot
WHAT IT STORES
A handful of rewrite rules, written by a person
SAMPLING FROM THE JOINT
p̂(sentence) = Π p̂(rule | symbol)
INPUTthe leftmost symbol still to rewrite, e.g. NP
OUTPUTone of its rules, each equally likely: NP → Det Adj N with ⅓
p̂(this run) = 1/7776
WHERE YOU SAW IT TODAY
Round 1’s rule is a tiny grammar. It knows structure, not meaning: “the student sleeps the tiny robot”
A few rules, endlessly many sentences.
GrammarL-systemCountingTextureEnergyVAEGANDiffusionLLM
Stop 2: an L-system
Rewrite rules that grow shapes instead of sentences
THE RULE
start: F
F → FF−[−F+F+F]
    +[+F−F−F]
F: draw forward
+ −: turn 22.5°
[ ]: branch, then come back
SYMBOLS AFTER EACH REWRITE
1 → 20 → 172 → 1,388 → 11,116
WHAT IT STORES
One rewrite rule and a turning angle, written by a person
SAMPLING FROM THE JOINT
p̂(plant) = Π p̂(rule | F) = 1
INPUTeach F in the string, rewritten four times over, then drawn with a pen
OUTPUTits replacement. One rule, so p̂ = 1: every run grows the same plant
Give F two rules at ½ each: every plant is a new sample.
WHERE YOU SAW IT TODAY
The same idea as the grammar: a short description, a huge result
Rules can grow shapes, not just sentences.
Lindenmayer, 1968. Plant: Prusinkiewicz & Lindenmayer, 1990, fig. 1.24f; drawn by lsystem.py.
GrammarCountingTextureEnergyLLM
Stop 2: counting
THE 16 ROUND 1 WORDS WE COUNT FROM
kaminosuteratomekenrisukayonkiranabutankesobihiyomemarutensakudobontakutezuranwakerigomisepunarodokemu
AFTER “k”, IN THE 16 ROUND 1 WORDS
e4 of 9
a2 of 9
u2 of 9
i1 of 9
NEW WORDS IT WROTE
sukamemakan ✓sutakura ✓makuton ✓so ✓pudokezudomerakameken ✓domi ✓te ✓tokami ✓hisudontaro ✓takemezuterakera ✓
Round 1’s rule: 95% of 2,000 samples belong; 2 copy an example.
WHAT IT STORES
How often each letter follows each other letter, counted from the 16 round 1 words
SAMPLING FROM THE JOINT
p̂(word) = Π p̂(next letter | previous letter)
INPUTthe previous letter, e.g. k
OUTPUTthe next letter: p̂(e | k) = 4/9, p̂(a | k) = 2/9, …, or “end of word”
p̂(sukamemakan) = 2.4 × 10⁻⁹
WHERE YOU SAW IT TODAY
Round 1 again. One letter of context can’t tell “n + vowel” from a bare n, so some words start with one
No one wrote the rules: they were counted from examples.
GrammarCountingTextureEnergyLLM
Stop 3: texture synthesis
The counting idea, for images
THE SAMPLE
GROWN FROM IT
Each new pixel: find the spots in the sample whose 9×9 surroundings best match what is already grown, copy one of their pixels.
WHAT IT STORES
Nothing but the small sample image itself
SAMPLING FROM THE JOINT
p̂(image) = Π p̂(pixel | neighbourhood)
INPUTthe 9×9 patch already grown around the next empty spot
OUTPUTits pixel, copied from a random one of the best-matching spots in the sample
The sample’s patches play the part of the counts.
WHERE YOU SAW IT TODAY
The letter-pair table again, but the context is a patch of pixels, not one letter
Context in, next pixel out: an n-gram for images.
GrammarCountingTextureEnergyLLM
Stop 4: an energy landscape
Hopfield networks and Boltzmann machines: the first neural generators
THREE STORED PATTERNS
START FROM A + WITH 30 PIXELS FLIPPED, THEN SETTLE
-55.7
energy after 30 flips
WHAT IT STORES
Weights between every pair of units, set by the Hebbian rule so the stored patterns sit in low-energy valleys
SAMPLING FROM THE JOINT
p̂(x) ∝ e−E(x), one unit at a time
INPUTevery other unit’s current value
OUTPUTthis unit, on or off: p̂(on | the rest) grows with its weighted input
Hopfield always takes the likelier value; a Boltzmann machine samples it.
A network can store a landscape
GrammarCountingTextureEnergyVAEGANDiffusionLLM
Stop 5: a variational autoencoder
Compress, rebuild, and fill in the gaps
RANDOM NUMBERS IN, DIGITS OUT
(0.1, -0.1) →
(0.6, 0.1) →
(-0.5, 0.4) →
(1.3, 0.9) →
(-0.7, -1.3) →
(-0.6, 0.0) →
Real output of a small VAE trained on 8×8 handwritten digits: two random numbers in, one digit out.
WHAT IT STORES
An encoder (example → a few numbers) and a decoder (numbers → example)
SAMPLING FROM THE JOINT
p̂(z, x) = p̂(z) · p̂(x | z)
INPUTa code z drawn from p̂(z) = N(0, I)
OUTPUTa digit x from the decoder’s p̂(x | z)
Two pieces, one step: no long chain.
WHERE YOU SAW IT TODAY
Round 1: you boiled 16 words down to “the pattern”, then wrote new words. Next: how it works
Generation as decoding random numbers.
Kingma & Welling, 2013. Digits: scikit-learn’s 8×8 digits; model: vae_fill.py.
GrammarCountingTextureEnergyGANDiffusionLLM
Stop 5: a generative adversarial network
A generator and a judge, trained against each other
random numbersGeneratornetworka fakereal imagesDiscriminatornetworkreal or fake?its verdict, and which way to change
Discussing it today in deep learning.
AI 4200
Two networks trained against each other: a generator that makes fakes, and a discriminator that learns to tell real from fake.
A generator that learns from a judge: more in Deep Learning.
Goodfellow et al., 2014, “Generative Adversarial Nets”.
GrammarCountingTextureEnergyGANDiffusionLLM
Stop 6: diffusion
Start from noise, clean up a little at a time
FROM NOISE TO A DIGIT, A LITTLE AT A TIME
Here the noise was added to a real 16×16 image and the frames are shown backwards. A trained model does the backward direction itself.
WHAT IT LEARNS
  • Take real images, add noise, train a network to predict the noise
  • Generating: start from pure noise and subtract a little predicted noise, hundreds of times
WHAT IT STORES
One network that, given a noisy image, predicts the noise in it
SAMPLING FROM THE JOINT
p̂(xT…x0) = p̂(xT) Π p̂(xt−1 | xt)
INPUTthe noisier image xt; xT is pure static
OUTPUTa slightly cleaner xt−1: predicted noise removed, a little fresh noise added
x0, the last output, is the image.
WHERE YOU SAW IT TODAY
Random numbers in, an image out, in many small steps; the VAE at the end does it in one
Start from noise, clean up a little at a time.
Sohl-Dickstein et al., 2015; Ho et al., 2020. Frames: diffusion_frames.py.
GrammarCountingTextureEnergyLLM
Stop 5: a large language model
Predict the next token, sample it, repeat
your prompt
generated, one token per click
The
·cat
·sat
·on
·the
·mat
·and
·fell
·asleep
every token so far goes in
LLMa probability for every token
p̂(next token | the cat sat on the)
mat0.31floor0.18sofa0.12bed0.08roof0.05table0.04
SAMPLE ONE
sample
append it
WHAT IT STORES
One huge network: a probability for every possible next token
WHERE YOU SAW IT TODAY
Round 1 at enormous scale, with the whole context; then tuned on people’s ratings, like round 2
SAMPLING FROM THE JOINT
p̂(text) = Π p̂(xt | x1 … xt−1)
Input every token so far · output the next token, sampled
A chatbot reply is a sample from p̂(reply | prompt)
Shannon’s counting idea, with the whole context and a network.
AUTOENCODER → VAE · 1 OF 17 · THE AUTOENCODER
An autoencoder, layer by layer
The network you build in ML: 64 pixels squeezed through 2 numbers, then back to 64
ENCODER: x → z
DECODER: z → p̂
THE TWO NUMBERS: THE CODE z
The last encoder layer is Linear 64 → 2, so every image becomes two numbers. Our held-out 3: z = (−10.57, 20.04). Nobody assigns them a meaning: training does.
THE LABELS
None. The target is the input itself: the network is scored on giving back x. No one labels a single digit.
THE SIZES
64 → 128 → 64 → 2 → 64 → 128 → 64
ReLU between layers; 33,474 parameters. Torch Canvas writes each Linear as its weight, out × in; e = examples in the batch.
Encoder → two numbers → decoder, trained to give back its own input.
Drawn in Torch Canvas (open “autoencoder_digits”). The code of our 3 is from vae_by_hand.py.
AUTOENCODER → VAE · 2 OF 17 · ITS LOSS
No labels: the target is the input
The decoder gives a probability for every pixel; the loss scores it against the real pixel
ORIGINAL x
ABC
REBUILT p̂
ABC
Each p̂ᵢ = σ(ŷᵢ): the sigmoid of the decoder’s logit for pixel i, read as “how sure it is that pixel i is inked”. The true xᵢ is the ink, 0 to 1.
THE LOSS: BINARY CROSS-ENTROPY, SUMMED OVER THE 64 PIXELS
L(x, p̂) = −Σi [ xi log p̂i + (1 − xi) log(1 − p̂i) ]
Only x appears: the image is its own label.
THREE OF THE 64 TERMS, BY HAND
pixelxᵢp̂ᵢits term=A inked1.000.95−log 0.950.05B blank0.000.40−log(1 − 0.40)0.51C grey0.560.57−[0.56 log 0.57 + 0.44 log 0.43]0.69
C is as good as it gets: for a grey pixel even p̂ = x leaves 0.69. B is a real mistake: ink where there is none.
ALL 64 TERMS
For our 3: L = 19.59. Averaged over the 360 held-out digits: 21.2. Gradient descent on L trains the encoder and the decoder together.
The label is the image itself: an autoencoder needs no labelled data.
Pixels and probabilities from vae_by_hand.py; held-out average from ae_training.json. In PyTorch: 64 × F.binary_cross_entropy_with_logits(ŷ, x).
AUTOENCODER → VAE 
An autoencoder compresses, then rebuilds
ORIGINAL
REBUILT FROM ITS CODE
code (7.1, 3.2)
code (-1.1, 11.6)
code (-10.6, 20.0)
code (9.0, -3.8)
code (19.1, 7.4)
code (2.9, 17.8)
The code is a tiny summary; the decoder turns any code back into an image.
AUTOENCODER → VAE 
A variational autoencoder: a bubble, not a point
SETS THE LOCATION
MAKES THE BUBBLE
ONE DIGIT’S BUBBLE
mean: where it sitsstd. dev.how widerandom point → decoder
The encoder makes a bubble; a random point in it goes to the decoder.
Drawn in Torch Canvas (open “vae_digits”). The bubble on the right is a sketch.
AUTOENCODER → VAE 
Two pulls on every bubble
KL DIVERGENCE: GRAVITY
06423
To the centre, width 1
CROSS-ENTROPY: KEEP APART
06423
Digits stay apart
Gravity packs the bubbles together; the rebuild keeps them separate enough to decode.
Sketch, not data. The real result of the balance is the map on the next slide.
AUTOENCODER → VAE 
Where the digits live
0123456789ABC
PUT A DIGIT THROUGH
its bubble
(blue ring on the map)
OR PICK A RANDOM POINT NEAR THE CENTRE (STARS)
A
B
C
New digits nobody drew
IT’S GENERATIVE
Random point in, new digit out: sampling
Random point in, new digit out: the VAE is a generative model.
AUTOENCODER → VAE 
What if we drop the cross-entropy?
?
BEFORE THE NEXT CLICK
Keep only the KL. Where do the bubbles go?
WHAT HAPPENS
All on one spot: mean 0, width 1

Can’t tell digits apart: no digits out
Without the cross-entropy, everything piles up and nothing can be told apart.
Start: the trained VAE. End: real run with the KL weighted 4× (β = 4), the same as no cross-entropy. 150 of the 1,437 training digits.
AUTOENCODER → VAE 
What if we drop the cross-entropy?
?
BEFORE THE NEXT CLICK
Keep only the KL. Where do the bubbles go?
WHAT HAPPENS
All on one spot: mean 0, width 1

Can’t tell digits apart: no digits out
Without the cross-entropy, everything piles up and nothing can be told apart.
Start: the trained VAE. End: real run with the KL weighted 4× (β = 4), the same as no cross-entropy. 150 of the 1,437 training digits.
AUTOENCODER → VAE 
What if we drop the cross-entropy?
?
BEFORE THE NEXT CLICK
Keep only the KL. Where do the bubbles go?
WHAT HAPPENS
All on one spot: mean 0, width 1

Can’t tell digits apart: no digits out
Without the cross-entropy, everything piles up and nothing can be told apart.
Start: the trained VAE. End: real run with the KL weighted 4× (β = 4), the same as no cross-entropy. 150 of the 1,437 training digits.
AUTOENCODER → VAE 
What if we drop the KL divergence?
?
BEFORE THE NEXT CLICK
Keep only the cross-entropy. Where do the bubbles go?
WHAT HAPPENS
Bubbles fly apart and shrink to dots

Only 3% of digits near the centre: random codes miss
Without the KL, the bubbles fly apart and random codes miss.
Start: the trained VAE. End: the same digits with only the cross-entropy (β = 0). Axes ±12, so many codes fly off the edge (out to 43).
AUTOENCODER → VAE 
What if we drop the KL divergence?
?
BEFORE THE NEXT CLICK
Keep only the cross-entropy. Where do the bubbles go?
WHAT HAPPENS
Bubbles fly apart and shrink to dots

Only 3% of digits near the centre: random codes miss
Without the KL, the bubbles fly apart and random codes miss.
Start: the trained VAE. End: the same digits with only the cross-entropy (β = 0). Axes ±12, so many codes fly off the edge (out to 43).
AUTOENCODER → VAE 
What if we drop the KL divergence?
?
BEFORE THE NEXT CLICK
Keep only the cross-entropy. Where do the bubbles go?
WHAT HAPPENS
Bubbles fly apart and shrink to dots

Only 3% of digits near the centre: random codes miss
Without the KL, the bubbles fly apart and random codes miss.
Start: the trained VAE. End: the same digits with only the cross-entropy (β = 0). Axes ±12, so many codes fly off the edge (out to 43).
AUTOENCODER → VAE 
What makes it generative
PLAIN AUTOENCODER: TEN RANDOM CODES, DECODED
Nonsense: a random code isn't likely to give a real digit
VAE: THE SAME TEN RANDOM CODES, DECODED
Something new: digits nobody drew
A GENERATIVE MODEL
Randomness in, a new sample out
Random input in, new and intelligible output out: that is a generative model.
AUTOENCODER → VAE 
Why is 3 next to 9, 5 and 8?
?
0123456789
BEFORE THE NEXT CLICK
The model never saw a label. Why would 3s land next to 9s, 5s and 8s?
DECODE THE CENTRE OF EACH CLUSTER
3
9 · 0.67
5 · 0.95
8 · 1.20
Shared strokes, shared codes
Near in code space = similar pixels
Close in code space means similar on the page.
Centre = median code of each digit’s training examples, decoded by the trained VAE; distances from the 3 (next: 2 at 1.43). vae_neighbours.py.
AUTOENCODER → VAE 
Does the space have meaning?
THE DIGIT UNDER THE ORANGE SQUARE
WHERE THAT CODE SITS
0123456789
Codes from −1.65 to 1.65 on both axes (standard-normal quantiles 0.05 to 0.95), 12 per side. vae_map.py.
Yes: nearby points decode to similar digits. The VAE learns a smooth map of the data.
After Kingma & Welling, 2013, fig. 4. Press Play to walk the edge of the map.
AUTOENCODER → VAE · 4 OF 17 · A QUESTION
Maybe random codes give new digits?
Talk it over with a neighbour before we move on
?
BEFORE THE NEXT CLICK
Cut off the encoder and keep the decoder, which turns two numbers into a digit. Pick two random numbers, decode them: do we get new digits?
ONLY THE DECODER
✂ encoder cut off herez ~ N(0, I)random
TEN RANDOM CODES, DECODED
The same smeared blob, ten times: not one real digit, and no variety.
WHAT WE DID
  • Kept the decoder exactly as trained: Linear 2 → 64, ReLU, Linear 64 → 128, ReLU, Linear 128 → 64, sigmoid
  • Fed it ten codes z drawn from N(0, I): mean 0, spread 1 in each of the two numbers
  • Every one came out as the same blob. Why? Look at where the encoder put the real codes
Cut off the encoder and feed in random codes: blobs, not digits.
Random codes: ten draws from a standard normal. Decoder from the trained autoencoder; ae_random.py. Drawing: Torch Canvas.
AUTOENCODER → VAE · 5 OF 17 · WHY IT FAILS
Where the autoencoder put its codes
The code z of every training digit, from the encoder; orange crosses: the ten random codes
0123456789
READ THE PLOT
  • Codes run from about −24 to 23 across and −8 to 32 up. Our 3 (blue ring) is at (−10.6, 20.0)
  • N(0, I) draws stay within about 2 of (0, 0): one small gap, between the patches
  • The loss only asks for a good rebuild, so nothing says where codes should go. They sprawl, and each digit gets its own patches
  • The gaps were never trained: the decoder just smears there
A decoder alone isn’t a generator: we don’t know where good codes are.
Training codes from the autoencoder’s encoder (1,437 digits); ae_random.py and vae_by_hand.py.
AUTOENCODER → VAE · 6 OF 17 · THE FIX
A VAE fills in the gaps
Same digits, same axes, same ten random codes (orange); rings at 1, 2 and 3 from the centre, where N(0, I) samples
AUTOENCODER: CODES SPRAWL
0123456789
VAE: CODES PULLED IN
THE SAME TEN RANDOM CODES, DECODED BY THE VAE
Pulled into one filled cloud, any random code decodes to a plausible digit: sample, then decode.
Codes: ae_random.py (autoencoder) and vae_fill.py (VAE means). The pull is the KL term, next slides. Same ten random draws as before.
AUTOENCODER → VAE · 7 OF 17 · THE NETWORK
What the VAE adds to the autoencoder
Torch Canvas drawing of the VAE’s middle; the encoder layers below and decoder layers above are the autoencoder’s
MAKES THE CLOUD
PULLS TO THE CENTRE
THE CLOUD
σ = e½ log σ², ε ~ N(0, I), z = μ + σ ⊙ ε: every pass picks a different code near μ. Our 3: σ = (0.36, 0.23).
THE PULL
KL = ½Σ(μ² + σ² − log σ² − 1): μ² pulls the centre to 0, the rest pushes σ toward 1. Added to the BCE.
CODE SPACE
N(0, I)our 3
Orange: our 3’s cloud, μ ± σ. Blue arrow: the KL pull on μ, −∂KL/∂μ = −μ. Dashed: N(0, I), where every cloud is pulled.
The noise makes a cloud around μ; the KL term pulls every cloud toward N(0, I).
Drawn in Torch Canvas (open “vae_digits”). 33,604 parameters, 130 more than the autoencoder: the second head. Grey: training μ’s from vae_fill.py.
AUTOENCODER → VAE · 8 OF 17 · THE ENCODER
The encoder outputs a distribution, q̂(z | x)
Not one code but a small cloud of codes: a mean μ and a spread σ for each of the two numbers
OUR HELD-OUT 3
The same 3 as on the autoencoder slides.
THE TWO HEADS OUTPUT
μ = (−1.415, −0.463)
log σ² = (−2.053, −2.981)
The autoencoder gave one point, (−10.57, 20.04).
THE SPREAD, BY HAND
σ = e½ · log σ²
σ₁ = e−1.026 = 0.358
σ₂ = e−1.490 = 0.225
Why log σ²? A Linear layer can output any number; e(·) turns it into a spread that is always positive.
q̂(z | x) = N(μ, σ²) IN CODE SPACE
0123456789
Orange: our 3, at 1σ and 2σ. Grey: every training digit’s μ.
q̂(z | x) is a cloud N(μ, σ²), not a point: any code inside it should rebuild the 3.
Numbers from vae_by_hand.py; the cloud from vae_fill.py. Two heads of Linear 64 → 2 (in the NumPy code, one Linear 64 → 4, split in half).
AUTOENCODER → VAE · 9 OF 17 · SAMPLING A CODE
Sampling a code: z = μ + σ ⊙ ε
The randomness is ε, drawn from N(0, I); μ moves it and σ stretches it
μ = (−1.415, −0.463)   σ = (0.358, 0.225)
DRAW
ε ~ N(0, I)
σ ⊙ ε
z = μ + σ ⊙ ε
DECODED
REBUILD LOSS (BCE)
1
(−0.802, −1.324)
(−0.287, −0.298)
(−1.702, −0.761)
17.99
2
(−0.248, 0.420)
(−0.089, 0.095)
(−1.504, −0.368)
18.21
3
(1.136, 0.110)
(0.407, 0.025)
(−1.008, −0.438)
19.15
DRAW 1, BY HAND
z₁ = −1.415 + 0.358 × (−0.802) = −1.702    z₂ = −0.463 + 0.225 × (−1.324) = −0.761
Same μ and σ, three draws of ε: three slightly different 3s.
ε from a fixed seed; z, decodings and losses from vae_by_hand.py. ⊙ multiplies entry by entry.
AUTOENCODER → VAE · 10 OF 17 · THE LOSS
The VAE loss: the autoencoder’s, plus one term
Rebuild the image from the sampled z, and keep the cloud close to N(0, I)
THE LOSS FOR ONE IMAGE
L = BCE(x, p̂) + KL( q̂(z | x) ‖ N(0, I) )
the autoencoder’s loss, now with p̂ decoded from a sampled z  ·  new: how far the cloud is from N(0, I)
REBUILD TERM (THE AUTOENCODER’S)
BCE = −Σi [xi log p̂i + (1 − xi) log(1 − p̂i)]
Same formula as before; p̂ is decoded from draw 1, z = (−1.70, −0.76).
BCE = 17.99
The autoencoder scored 19.59 on this 3. Rebuilding from a random code near μ is a harder job.
KL TERM, BY HAND
KL = ½ Σj (μj² + σj² − log σj² − 1)
j = 1: ½(2.002 + 0.128 + 2.053 − 1) = 1.591
j = 2: ½(0.214 + 0.051 + 2.981 − 1) = 1.123
KL = 2.715
  • μ² pulls the cloud’s centre to 0
  • σ² − log σ² − 1 is 0 only at σ = 1: it stops the cloud shrinking to a point
For our 3: L = 17.99 + 2.71 = 20.70. Held-out average: 21.1 + 3.2.
This L is the negative ELBO (evidence lower bound). Numbers: vae_by_hand.py and vae_training.json.
AUTOENCODER → VAE · 11 OF 17 · A QUESTION
What if we drop the KL term?
Talk it over with a neighbour before we move on
?
BEFORE THE NEXT CLICK
Train the same VAE with only the rebuild term: L = BCE. What happens to the codes, and to random samples?
SAME NETWORK, LOSS = BCE ONLY
no KLVAEKL (how far from N(0, I))1363.2Largest |μ| of a code432.8Average σ0.000.22Rebuild BCE20.921.1
σ shrinks to 0, so every cloud becomes a point, and the points sprawl. It is the plain autoencoder again.
8 RANDOM CODES z ~ N(0, I), DECODED
Among samples a digit classifier is sure about, only 2 kinds of digit (VAE: 10).
Without the pull, the codes sprawl and random codes miss again.
Both trained 600 epochs on the same 1,437 digits (vae_beta.py, β = 0 and β = 1). Held-out numbers; the classifier is explained on the next slide.
AUTOENCODER → VAE · 12 OF 17 · TWO PULLS
Two pulls on every bubble
Each digit’s code is a bubble: μ is where it sits, σ how wide it is. The KL is gravity; the rebuild keeps digits apart
REBUILD ONLY · β = 0
−30+300123456789random draws land here
No gravity: σ shrinks to 0, so bubbles become dots that sprawl out to |μ| = 43. Only 3% sit inside the circle.
GRAVITY WINS · β = 4
−3+3
Every bubble sits at μ = 0 with σ = 1 (KL = 0). The decoder can’t tell them apart: no digits come out.
BOTH · β = 1
−3+30123456789
Packed into a disc but still sorted by digit: 87% of codes sit inside the circle, where 86% of random draws land.
Gravity packs the bubbles, the rebuild keeps them apart, so random codes land on real digits.
Each bubble: one of the 1,437 training digits, at μ with radii σ. Dashed circle: radius 2. Numbers: vae_bubbles.py.
AUTOENCODER → VAE · 13 OF 17 · HOW HARD TO PULL
How hard to pull? The KL weight β
L = BCE + β · KL. Same network, same data; only β changes
β
BCE
KL
max |μ|
avg σ
8 RANDOM SAMPLES
SURE · KINDS
0
20.9
136
43.3
0.00
67% · 2
0.1
20.8
6.6
3.2
0.05
51% · 10
1 (VAE)
21.1
3.2
2.8
0.22
55% · 10
2
23.4
1.6
2.3
0.46
25% · 6
4
27.2
0.0
0.0
1.00
0% · 0
Too weak: codes sprawl, gaps. Too strong: every code is N(0, I) and the code says nothing.
1,000 random codes per row. Judge: a logistic-regression digit classifier (96.7% held-out accuracy); “sure” = it gives one digit ≥ 0.9, “kinds” = how many digits among those. vae_beta.py.
AUTOENCODER → VAE · 14 OF 17 · TRAINING THROUGH NOISE
Backprop through a random draw
Draw ε on the side, and z = μ + σ ⊙ ε is an ordinary function of μ and σ: gradients flow back to the encoder
FROM THE DECODER, BY BACKPROP
∂L/∂z
= (−0.041, −3.472)
For our 3, draw 1, ε = (−0.80, −1.32). Sampling z straight from N(μ, σ²) would stop here.
TO μ
∂L/∂μ = ∂L/∂z · 1 + μ
= (−0.041, −3.472)
 + (−1.415, −0.463)
= (−1.456, −3.935)
second part: from the KL term
TO σ
∂L/∂σ = ∂L/∂z ⊙ ε + (σ − 1/σ)
= (0.033, 4.598)
 + (−2.434, −4.215)
= (−2.400, 0.383)
ε scales σ’s share: ∂z/∂σ = ε
Move the randomness into an input, ε, and gradients reach μ and σ: the reparameterization trick.
Kingma & Welling, 2013. One image (n = 1); gradients from vae_by_hand.py. Drawing: Torch Canvas.
AUTOENCODER → VAE · 15 OF 17 · THE CODE
Autoencoder vs. VAE in PyTorch
The code Torch Canvas writes for the two drawings; the VAE’s new lines are highlighted
class Autoencoder(nn.Module):
def __init__(self):
self.enc1 = nn.Linear(64, 128)
self.enc2 = nn.Linear(128, 64)
self.enc3 = nn.Linear(64, 2)
self.dec1 = nn.Linear(2, 64)
self.dec2 = nn.Linear(64, 128)
self.dec3 = nn.Linear(128, 64)
def forward(self, x):
a = F.relu(self.enc1(x)) # (e, 128)
a = F.relu(self.enc2(a)) # (e, 64)
z = self.enc3(a) # (e, 2)
a = F.relu(self.dec1(z)) # (e, 64)
a = F.relu(self.dec2(a)) # (e, 128)
return self.dec3(a)
def loss(self, recon, x):
return 64 * F.binary_cross_entropy_with_logits(recon, x)
class VAE(nn.Module):
def __init__(self):
self.enc1 = nn.Linear(64, 128)
self.enc2 = nn.Linear(128, 64)
self.enc_mu = nn.Linear(64, 2)
self.enc_logvar = nn.Linear(64, 2)
self.dec1 = nn.Linear(2, 64)
self.dec2 = nn.Linear(64, 128)
self.dec3 = nn.Linear(128, 64)
def forward(self, x):
a = F.relu(self.enc1(x)) # (e, 128)
a = F.relu(self.enc2(a)) # (e, 64)
mu = self.enc_mu(a) # (e, 2)
logvar = self.enc_logvar(a) # (e, 2)
std = torch.exp(logvar * 0.5) # (e, 2)
z = mu + torch.randn_like(std) * std # (e, 2)
a = F.relu(self.dec1(z)) # (e, 64)
a = F.relu(self.dec2(a)) # (e, 128)
return self.dec3(a), mu, logvar
def loss(self, recon, mu, logvar, x):
return (64 * F.binary_cross_entropy_with_logits(recon, x)
+ kl_normal(mu, logvar))
Ten changed lines turn an autoencoder into a generator.
kl_normal(μ, log σ²) = −½ · mean over the batch of Σ(1 + log σ² − μ² − σ²). super().__init__() and predict() left out; x.float() shortened to x.
AUTOENCODER → VAE · 16 OF 17 · SIDE BY SIDE
Autoencoder vs. VAE
Same layers, same data, no labels in either; what changes is the code and the loss
AUTOENCODER
VAE
Encoder ends in
Linear 64 → 2: the code z
two heads, Linear 64 → 2: μ and log σ²
The code
z = encoder(x): one point
z = μ + σ ⊙ ε: a sample from q̂(z | x)
Labels
none: the target is x
none: the target is x
Loss
BCE(x, p̂)
BCE(x, p̂) + KL(q̂(z | x) ‖ N(0, I))
Held-out loss
21.2
21.1 + 3.2
Parameters
33,474
33,604
Codes land
anywhere, with gaps
in one filled cloud around 0
To generate
can’t: random codes miss
z ~ p̂(z) = N(0, I), then x ~ p̂(x | z)
A VAE is an autoencoder whose code is a distribution you can sample.
Held-out numbers: ae_training.json, vae_training.json (360 digits). Generating samples from the joint p̂(z, x) = p̂(z) · p̂(x | z).
Takeaways
  • Discriminative models tell apart; generative models make new: learn the distribution, then sample from it.
  • Producing is harder than recognizing, and a generator is graded on three things at once: valid, novel, diverse.
  • Every generator samples from a joint, one conditional at a time: p̂(next piece | what came before).
  • An autoencoder needs no labels (the target is the input), but its codes sprawl, so random codes miss.
  • A VAE gives each digit a bubble: the KL pulls the bubbles together, the cross-entropy keeps them apart, so a random point decodes to a new digit.

Generative AI: learn the distribution, then sample from it.