Cross-attention, foundation models and PEFT
How a decoder reads an encoder, why pretraining makes for a good model, and how to adapt 
AI 4200 · DEEP LEARNING
All slides, one per page
© 2026 Dr. Jesse Roberts
Recap: encoder and decoder
[MASK]EP+qkvblackEP+qkvcatEP+qkvsleepsEP+qkvENCODER (BERT) · ONE LAYER · NO MASK+MLP+h₁→ the+MLP+h₂+MLP+h₃+MLP+h₄<s>EP+qkv+MLP+→ elDECODER (GPT) · ONE LAYER · ONE WORD AT A TIMEelEP+qkv+MLP+→ gatogatoEP+qkv+MLP+→ negronegroEP+qkv+MLP+→ come(eats)
# encoder self-attention (BERT) # X: one row per word, t × d Q, K, V = X @ W_Q.T, X @ W_K.T, X @ W_V.T S = Q @ K.T / sqrt(d) # no mask: every word sees every word A = softmax(S, dim=-1) X = X + A @ V # residual X = X + mlp(X) # residual
# decoder self-attention (GPT) # X: one row per written word, t × d Q, K, V = X @ W_Q.T, X @ W_K.T, X @ W_V.T S = Q @ K.T / sqrt(d) mask = torch.triu(torch.ones(t, t), diagonal=1).bool() S = S.masked_fill(mask, -inf) # nothing to the right A = softmax(S, dim=-1) X = X + A @ V # residual X = X + mlp(X) # residual
[MASK]EP+qkvblackEP+qkvcatEP+qkvsleepsEP+qkvENCODER (BERT) · ONE LAYER · NO MASK+MLP+h₁→ the+MLP+h₂+MLP+h₃+MLP+h₄<s>EP+qkv+MLP+→ elDECODER (GPT) · ONE LAYER · ONE WORD AT A TIMEelEP+qkv+MLP+→ gatogatoEP+qkv+MLP+→ negronegroEP+qkv+MLP+→ come(eats)
Encoder: every word sees every word. Decoder: only the words before it.
One head, one layer. Orange ring: the query on this click. Line width ∝ gate, illustrative. WO and LayerNorm omitted.
Cross-attention
theEP+qkvblackEP+qkvcatEP+qkvsleepsEP+qkv+MLP+h₁kv+MLP+h₂kv+MLP+h₃kv+MLP+h₄kvkeys, values made from HENCODER · READS THE WHOLE SOURCE · OUTPUT H<s>EP+qkv+q+MLP+elEP+qkv+q+MLP+gatoEP+qkv+q+MLP+negroEP+qkv+q+MLP+DECODER · MASKED SELF-ATTENTION, THEN CROSS-ATTENTION→ el→ gato→ negro→ duermecome
EVERY WRITTEN WORD QUERIES THE SOURCE
the
black
cat
sleeps
<s> → el
0.55
0.15
0.15
0.15
el → gato
0.10
0.20
0.60
0.10
gato → negro
0.10
0.60
0.20
0.10
negro → duerme
0.05
0.10
0.20
0.65
Cross-attention: q from the decoder, k and v from the encoder. The only wire between them.
Cross-attention
The decoder has written “el”. Which source word does it need next?
?
ENCODER OUTPUT Hthe[1, 0, 0, 0]×gate ?black[0, 1, 0, 0]×gate ?cat[0, 0, 1, 0]×gate ?sleeps[0, 0, 0, 1]×gate ?softmaxΣout “el”DECODER STATE Zel[0, 1, 0]WQquery: from the decoderq [0, 0, 2, 0]WKk [1, 0, 0, 0]·s 0WKk [0, 1, 0, 0]·s 0WKk [0, 0, 1, 0]·s 2WKk [0, 0, 0, 1]·s 0a 0.096a 0.096a 0.711a 0.096[0.10, 0.10, 0.71, 0.10]mostly cat → gato
Queries from the decoder; keys and values from the encoder. Nothing else changes.
Illustrative features and weights. H rows are (article, adjective, noun, verb); Z rows are (start, article written, noun written). WK = WV = I. No ÷√d, as in last week's running example.
Every written word queries the source
One row of gates per decoder word, all computed at once: softmax(QKᵀ)
the
black
cat
sleeps
SOURCE: KEYS AND VALUES FROM THE ENCODER
QUERIES
WRITES
THE GATE MATRIX
3 × 4: written words × source words.
Self-attention's was square, t × t.
“gato” looks back at black: Spanish puts the adjective after the noun. The gates reorder the words. Nobody wrote that rule.
The encoder runs once. Its output H is reused for every new word the decoder writes.
One row per written word, one column per source word: a soft alignment, learned.
Same illustrative weights as the previous slide. Each row sums to 1.
Same attention structure with different inputs
The usual diagram again; only the wires at the bottom change
ZdecoderHencoderLinear WQLinear WKLinear WVQKVMatMul QKᵀSoftMaxMatMul ·V
Self-attention
Cross-attention
Queries from
X
Z, the decoder
Keys, values from
X
H, the encoder
Gate matrix
t × t
ttgt × tsrc
Mask
causal, in a decoder
none: all of H is visible
# self-attention: one input
Q, K, V = X @ W_Q.T, X @ W_K.T, X @ W_V.T
# cross-attention: two inputs
Q    = Z @ W_Q.T
K, V = H @ W_K.T, H @ W_V.T
# the rest is identical
out = softmax(Q @ K.T / sqrt(d)) @ V
Self-attention is cross-attention with both inputs the same.
Inside one decoder block with cross attention
ZLayerNormMasked self-attention+LayerNormCross-attention+LayerNormMLP+residualresidualresidualout · t × dEncoder × NH: keys and valuesq, k, v from Zq from Z
WHERE CROSS-ATTENTION SHOWS UP
  • Original transformer, T5: text → text
  • Whisper: audio encoder → text decoder
  • Stable Diffusion: the denoiser reads the prompt
No Cross attention
  • GPT-style, decoder-only: prompt and answer share one sequence
  • BERT-style, encoder-only: one input, nothing to read from
Foundation models
Three ingredients, all needed at once
1
Extreme data
  • Images: ~300 M (JFT-300M, the original ViT)
  • Text: billions of words (BERT); no labels needed, so nearly unlimited
2
Just enough flexibility
CNN
locality + weight sharing
little data: great
at scale: blinders
Transformer
attention: global, input-dependent mixing
weights shared across positions
sweet spot for extreme data
MLP
everything wired to everything
nothing shared
needs even more data and parameters
3
A task that forces the relationships
Hide a word, predict it
BERT: must read context
(images: masked patches)
Captioned pictures, hide words
must read the image (multi-modality)
The task is
the teacher
Extreme data × just-flexible-enough model × the right task: pretrain once, adapt everywhere.
Adapting a pretrained model: the rest of today, and Project 3.
From one task to many
Where NLP stood before large language models
?
ONE MODEL PER TASK
Sentiment model
its own data, its own weights
Translation model
its own data, its own weights
Question-answering model
its own data, its own weights
Each one starts from zero.
BEFORE THE NEXT CLICK
Train one model on all of the tasks at once. Why might it beat the separate models?
MULTI-TASK LEARNING: ONE SET OF WEIGHTS
  • Tasks share structure: grammar, meaning, facts about the world
  • A feature learned for one task is free for the others
  • Small tasks borrow from big ones
EARLY EVIDENCE
  • decaNLP (2018): ten NLP tasks, all phrased as question answering, one model
  • T5 (2019): every task is text in → text out
Shared weights turn many tasks into one better starting point for each of them.
decaNLP: McCann et al., 2018, “The Natural Language Decathlon”. T5: Raffel et al., 2019.
A language model is already multi-task
Finish each of these by predicting the next word
SUMMARIZATION
… the whole news article … TL;DR: ___
TRANSLATION
the cat sleeps = el gato duerme
the black cat = ___
QUESTION ANSWERING
Q: Who wrote Hamlet? A: ___
GPT-2 · 2019
“Language Models are Unsupervised Multitask Learners”
  • Trained on one task: predict the next word of web text
  • No task labels, no task-specific training
  • Zero-shot: best published result on 7 of 8 language-modelling benchmarks
  • Other tasks: it starts to do them, well short of supervised models
Each blank is a different task. All of them are “what comes next?”
Predict the next word of everything, and the tasks come along: the task is the teacher.
Radford et al., 2019 (GPT-2, 1.5 B parameters).
The benefit: excellent priors
Training on many tasks at once makes a model broadly capable across a domain
FROM SCRATCHsentimenttranslationQAsummariesyour taskrandom startFROM A FOUNDATION MODELsentimenttranslationQAsummariesyour taskpretrained start
  • A prior: what the model already believes before it sees your data
  • Pretraining on a whole domain puts the start near every task in it
  • Less data, fewer steps, and fewer weights need to move
A foundation model is a prior over a whole domain: every task in it starts close.
A sketch, not data. Compare the model-space map from the generalization lecture: pretraining moves where training starts.
Foundation models across domains
Pretrained once on enormous data, then adapted to many tasks
LANGUAGE
BERT (2018)
Encoder only
3.3 B words, masked words
classification, search
LANGUAGE
Llama 3 (2024)
Decoder only
over 15 T tokens, next word
chat, code, summaries
IMAGE + TEXT
CLIP (2021)
Two encoders: ViT + text
400 M image–caption pairs
zero-shot labels, image search
SPEECH
Whisper (2022)
Encoder–decoder, cross-attention
680,000 hours of audio
transcription, translation
VISION
DINOv2 (2023)
Encoder only (ViT)
142 M images, no labels
features for any image task
SEGMENTATION
SAM (2023)
ViT encoder + small decoder
1.1 B masks on 11 M images
outline anything, from a click
PROTEINS
ESM-2 (2022)
Encoder only (BERT-style)
protein sequences, masked amino acids
structure, function, design
MEDICAL IMAGING
EVA-02 chest X-ray ViT
Encoder only (ViT)
520,000 chest X-rays
our elbow example
Same recipe everywhere: extreme data, a transformer, a task that forces the relationships.
Four ways to adapt a foundation model
Task: label movie reviews positive or negative with Llama 3 8B. They differ in how much of the model changes
1
In-context learning
Put labelled reviews in the prompt
WHAT CHANGES
only the prompt
WEIGHTS TRAINED
0
2
Head + frozen weights
Train only a new last layer
WHAT CHANGES
the head
WEIGHTS TRAINED
8,194
3
Full fine-tuning (SFT)
Train every weight on your labelled data
WHAT CHANGES
everything
WEIGHTS TRAINED
8.0 B
4
LoRA
Freeze the model, train small matrices beside it
WHAT CHANGES
small add-ons
WEIGHTS TRAINED
3.4 M
From here on: one adaptation method at a time.
ADAPTATION 1 OF 4 · IN-CONTEXT LEARNING
Meta-learning: learning to learn
GPT-3 (2020), 175 B parameters: “Language Models are Few-Shot Learners”
?
Meta-learning: train on many tasks so that a new task needs only a few examples.
OUTER LOOP · PRETRAINING
Gradient descent on huge amounts of text
Slow · changes the weights
INNER LOOP · THE PROMPT
A few examples in the prompt, one forward pass
Fast · no weight changes
= in-context learning
TRANSLATION
cat → gato
black → negro
sleeps → ___
TRANSLATION
cat → gato
black → negro
sleeps → ___
duerme
WORDS THAT DON'T EXIST · GPT-3 PAPER
whatpu = a small, furry animal from Tanzania
→ “We saw these very cute whatpus.”
farduddle = to jump up and down really fast
→ ___
LABELS THAT MEAN NOTHING · foo / bar
“A wonderful, moving film.”
foo
“Dull plot, wooden acting.”
bar
“I loved every minute.”
foo
“Two hours I'll never get back.”
___
LABELS THAT MEAN NOTHING · foo / bar
“A wonderful, moving film.”
foo
“Dull plot, wooden acting.”
bar
“I loved every minute.”
foo
“Two hours I'll never get back.”
___
bar
FLIPPED LABELS
“A wonderful, moving film.”
negative
“Dull plot, wooden acting.”
positive
“I loved every minute.”
negative
“Two hours I'll never get back.”
___
FLIPPED LABELS
“A wonderful, moving film.”
negative
“Dull plot, wooden acting.”
positive
“I loved every minute.”
negative
“Two hours I'll never get back.”
___
Small models: “negative” · follow their prior
Large models: “positive” · follow the examples
Trained on enough tasks, the model learns to learn a new one from a handful of examples.
Made-up words: Brown et al., 2020 (GPT-3), after Carey & Bartlett, 1978. foo / bar and flipped labels: Wei et al., 2023, “Larger language models do in-context learning differently”
ADAPTATION 1 OF 4 · IN-CONTEXT LEARNING
In-context learning, measured
GPT-3 175 B answers trivia questions. Its weights never change
A FEW-SHOT PROMPT · ILLUSTRATIVE
Q: Capital of Australia? A: Canberra Q: The Red Planet? A: Mars Q: Who painted the Mona Lisa? A: ___
zero-shot · no examples
one-shot · 1 example
few-shot · dozens, as many as fit
in its 2,048-token window
TRIVIAQA · ACCURACY, NO DOCUMENTS (CLOSED BOOK)
T5-11B
🔥 fine-tuned
50.1%
T5-11B + SSM
🔥 fine-tuned + extra pretraining
60.5%
GPT-3 zero-shot
❄️ prompt only
64.3%
GPT-3 one-shot
❄️ prompt only
68.0%
GPT-3 few-shot
❄️ prompt only
71.2%
A few examples in the prompt beat models fine-tuned on the task.
Brown et al., 2020 (GPT-3), Table 3.3. T5 rows: Roberts et al., 2020; SSM = salient span masking, extra pretraining that masks names and dates. 
ADAPTATION 2 OF 4 · HEAD + FROZEN WEIGHTS
A new classifying head
The pretrained ViT answers a 1,000-way question. MNIST asks a 10-way one
?
image224 × 224Patches+ positionEncoderBlock × 12attention + MLPPoolone vector · 768Linear 768 → 1000ImageNet classesŷLinear 768 → 1000ImageNet classesLinear 768 → 10digits · new, random startŷ
BEFORE THE NEXT CLICK
ImageNet has 1,000 classes: dogs, cars, mushrooms. MNIST has ten digits. Which part of the network has to change?
  • Body → 768 features: the prior
  • Old head: 1,000 ImageNet classes
  • New head: 7,690 weights (768 × 10 + 10)
  • Starts random: must be trained
GUESS · ❄️ BODY, 🔥 HEAD · MNIST TEST ACCURACY
5 per digit
50 images
? %
67.0%
50 per digit
500 images
? %
89.5%
500 per digit
5,000 images
? %
95.2%
Swap the head for the new task. Keep the body for its prior.
❄️ blue: pretrained, frozen. 🔥 orange: new, trained. ViT-B/16 (google/vit-base-patch16-224). Results: our molab runs, mean of 3 seeds, 10,000 test images; chance is 10%.
ADAPTATIONS 2 AND 3 OF 4 · HEAD + FROZEN WEIGHTS vs FULL FINE-TUNING (SFT)
Frozen weights
Used in every forward pass, never updated. ViT-B on MNIST, adapted two ways
image❄️Patches❄️Block 1❄️Block 2⋯❄️Block 12🔥Headŷ, lossgradient image🔥Patches🔥Block 1🔥Block 2⋯🔥Block 12🔥Headŷ, lossgradient
Head + frozen weights
❄️ body, 🔥 head
gradient stops at the head
Full fine-tuning
🔥 everything
gradient + 2 Adam numbers per weight
❄️ ONE FROZEN LAYER, Ŷ = XWᵀ, IN BACKPROP
✓
∂L/∂X = ∂L/∂Ŷ · W · still computed
✗
∂L/∂W = (∂L/∂Ŷ)ᵀ X · skipped
for p in model.vit.parameters(): p.requires_grad = False # ❄️ body opt = torch.optim.AdamW( model.classifier.parameters()) # 🔥 head
PER TASK · ViT-B
HEAD + FROZEN
FULL
Trained weights
7,690
85.8 M
Their gradients
0.03 MB
343 MB
Adam state (2 per weight)
0.06 MB
686 MB
Backward through the body
no
yes
Peak GPU memory, measured
0.4 GB
2.6 GB
Freezing saves the gradients and optimizer state; stopping the backward pass early saves the activations.
❄️ blue: frozen. 🔥 orange: trained. 4 bytes per number. Peak memory: our molab run, batch 10; most of it is activations kept for the backward pass.
Priors reach further than you expect
The obvious example is an LLM. Two that are less obvious:
?
VISION · IMAGENET → MNIST
ViT-B/16, pretrained on ImageNet photos
→ handwritten digits
  • Chance is 10%. With 5 images per digit, your guess?
60%80%100%550500training images per digit67%78%full fine-tuningnew head, body frozen
MEDICAL · CHEST → ELBOW
EVA-02 ViT, pretrained on 520,000 chest X-rays
→ normal or abnormal elbow
  • A small set of elbow X-rays (MURA)
  • 3 epochs, about 13 minutes on a free T4 GPU
≈ 80%validation accuracy
Strokes, edges and shapes transfer: the prior belongs to the domain, not to one task.
MNIST: our runs, mean of 3 seeds, epoch chosen on a validation set, test accuracy on 10,000 digits. Elbow: my NAIRR workshop notebook. No from-scratch model was trained for comparison.
Three ways to adapt the generalist
The choice you make in Project 3
?
Full fine-tuning
train every weight
  • Fits best with plenty of data
  • A full copy of the model per task
  • GPT-3: 350 GB for each copy
Linear probe
freeze everything, train a new last layer
  • Cheap and safe
  • The features cannot change
  • Stuck if the task needs new ones
PEFT
freeze the model, train a small add-on
  • a few small matrices per layer
  • Features can change, a little
  • Well under 1% of the weights
BEFORE THE NEXT CLICK
The ViT has 86 M weights and we gave it 50 images. Do all 86 M need to move? How many numbers does a task really need?
Far fewer than you would guess. The measurement is called the intrinsic dimension of the task.
PEFT: parameter-efficient fine-tuning. Keep the prior, train a small add-on.
350 GB: Hu et al., 2021 (the LoRA paper), for a fine-tuned GPT-3 175 B checkpoint.
ADAPTATION 4 OF 4 · WHY LoRA WORKS
Intrinsic dimension
How many weights have to change to solve a new task?
?
RoBERTa: over 10⁸ weights
each dot = 10,000 weights
GUESS
New task: are two sentences paraphrases?
How many numbers must change to get 90% of full fine-tuning?
≈ 200numbers
A fiftieth of the circled dot
THE IDEA
  • Keep every pretrained weight where it is
  • Let them move together along only 200 directions
  • Train just those 200 numbers
  • Pretraining makes this number small
  • Bigger pretrained models: smaller still
A good prior means a new task needs only a few directions. LoRA trains just those.
Aghajanyan et al., 2020: RoBERTa on MRPC, about 200 trained numbers reach 90% of full fine-tuning (the intrinsic dimension). Method from Li et al., 2018.
ADAPTATION 4 OF 4 · LoRA
LoRA, starting at rank 1
Freeze W0. Build the change ΔW from one column and one row
bn × 1×a1 × f=ΔW = b an × f
🔥 Train only b and a
Every row of ΔW is a, rescaled
One direction: rank 1
ŷ = x W0ᵀ + x ΔWᵀ
x·❄️ W₀ᵀ+x·🔥 ΔWᵀ=ŷ
WEIGHTS TRAINED · ONE 768 × 768 MATRIX
full ΔW · n · f
589,824
rank 1 · n + f
1,536
Rank 1: W0 can move in one direction, for n + f weights.
Hu et al., 2021, “LoRA: Low-Rank Adaptation of Large Language Models”. Heatmaps: random values, 8 × 8 for display; blue positive, orange negative.
ADAPTATION 4 OF 4 · LoRA
LoRA: a low-rank update
A few directions per weight matrix, chosen by training
Xe × f· W₀ᵀfrozen · n × f+Ŷe × n· AᵀHe × r· Bᵀ× α/rtrained · A is r × f, B is n × r
Intrinsic dimension: a few random directions for the whole model.
LoRA: a few learned directions for each matrix we choose.
THE LAYER
W = W0 + (α/r) B A
Ŷ = X W0ᵀ + (α/r) X Aᵀ Bᵀ
WEIGHTS TO TRAIN, PER MATRIX
full update ΔW
n · f
LoRA, rank r
r · (n + f)
r is small: 1 to 64, against n and f in the hundreds or thousands.
Freeze W0. Learn the change as two thin matrices, B and A.
Hu et al., 2021, “LoRA: Low-Rank Adaptation of Large Language Models”. Blue: frozen, pretrained. Orange: trained.
ADAPTATION 4 OF 4 · LoRA
LoRA by hand
One 4 × 4 layer, rank 1, α = 2, one input x
1 · BUILD ΔW = (α/r) b a, ONE ROW AT A TIME
×2
α/r
🔥 a 
1
0
1
1
🔥 b
0.5
0
−1
0
row 1 = 2 · b1 · a = 2 · 0.5 · [1, 0, 1, 1] = [1, 0, 1, 1]
row 2 = 2 · b2 · a = 2 · 0 · [1, 0, 1, 1] = [0, 0, 0, 0]
row 3 = 2 · b3 · a = 2 · −1 · [1, 0, 1, 1] = [−2, 0, −2, −2]
row 4 = 2 · b4 · a = 2 · 0 · [1, 0, 1, 1] = [0, 0, 0, 0]
ΔW is rebuilt from 8 numbers
Every row is a multiple of a: rank 1
2 · USE IT
x
1
2
0
1
·
❄️ W₀ᵀ
1
0
2
1
0
1
−1
1
−1
1
0
0
2
0
1
−1
+
x
1
2
0
1
·
🔥 ΔWᵀ
1
0
−2
0
0
0
0
0
1
0
−2
0
1
0
−2
0
=
x W₀ᵀ
3
2
1
2
+
x ΔWᵀ
2
0
−4
0
=
ŷ
5
2
−3
2
16 weights ❄️ frozen
8 weights 🔥 trained
After training, add ΔW into W0: one matrix again, no extra cost at inference.
b “after training” is illustrative; every other number follows from it. Blue cells positive, orange negative. In code b is B (n × r) and a is A (r × f).
ADAPTATION 4 OF 4 · LoRA
What rank changes
Rank r: B gets r columns, A gets r rows
Bn × 2×A2 × f=ΔW = B An × f · rank 2
Bn × 3×A3 × f=ΔW = B An × f · rank 3
r = 2: B has 2 columns, A has 2 rows
r = 3: B has 3 columns, A has 3 rows
One matrix multiply
r = the inner dimension
W = W0 + (α/r) B A
ŷ = x W0ᵀ + (α/r) x (B A)ᵀ
n × r · r × f
Larger r: ΔW can take more shapes
WEIGHTS TRAINED · ONE 768 × 768 MATRIX
RANK r
r · (n + f)
OF FULL
1
1,536
0.3%
8
12,288
2.1%
64
98,304
16.7%
full ΔW
589,824
100%
Rank is the dial: more shapes for ΔW, for r · (n + f) weights.
Heatmaps: random B and A, 8 × 8 for display; blue positive, orange negative. Counts for n = f = 768 (one ViT-B attention matrix).
ADAPTATION 4 OF 4 · LoRA
Initialization: B = 0, A random
Multiply by zeros first, then let the gradient fill B in
x❄️ · W₀ᵀ· Aᵀrandom· Bᵀ= 0+ŷ
B
A
ΔW = B A
W = W₀ + ΔW
Start
loss 1.333
×=→
B = 0, so B A = 0 for any A
W = W0 exactly: the prior is untouched
Step 1
loss 1.308
×=→
A ≠ 0, so ∂L/∂B ≠ 0: B moves first
ΔW is still tiny
Step 2
loss 1.174
×=→
ΔW grows a little, in the shape the task needs
A starts to move too, more slowly
Step 3
loss 0.571
×=→
W has moved smoothly away from W0
no random noise was ever added
B = 0 keeps the prior intact; A ≠ 0 keeps the gradient alive.
Toy 8 × 8 layer, r = 2, three steps of gradient descent toward a rank-1 change. One colour scale per column across all steps; blue positive, orange negative.
ADAPTATION 4 OF 4 · LoRA
Two matrices: keep the variance
Talk it over with a neighbour before we move on
?
THE RULE · INDEPENDENT, MEAN 0
var(x w) = var(x) · var(w)
var(sum of k such terms) = k · var(x) · var(w)
INTO A · f TERMS
hj = Σi=1..f xi Aji
var(h) = f · var(x) · var(A)
Want var(h) = var(x)
var(A) = ?
var(A) = 1/f std 1/√f
OUT OF B · r TERMS
uk = Σj=1..r hj Bkj
var(u) = r · var(h) · var(B)
Want var(u) = var(h)
var(B) = ?
var(B) = 1/r std 1/√r
Output variance = input variance when var(A) = 1/f and var(B) = 1/r.
Same fix as LeCun init for any layer. B starts at 0 (previous slide), so LoRA puts the correction in a scale on B A: α/r in the original paper, α/√r in rsLoRA (Kalajdzievski, 2023), which keeps the variance steady as r grows.
Variance through the LoRA path
Two sums, two corrections
?
INTO A: h = x Aᵀ SUMS f TERMS
LoRA draws A from U(−1/√f, 1/√f), so var(h) = var(x)/3 at any width.
WIDTH f
A ~ N(0, 1)
LoRA INIT
16
17
0.36
256
262
0.33
768
732
0.33
4,096
4,087
0.33
OUT OF B: THE UPDATE SUMS r TERMS
Size of scale · h Bᵀ, relative to r = 1. What happens as r grows?
RANK r
NO SCALE
÷ r
÷ √r
1
1×
1.00×
1.00×
4
2×
0.51×
1.01×
16
4×
0.25×
0.99×
64
8×
0.13×
1.00×
256
16×
0.063×
1.00×
Every sum needs its variance correction: 1/√f going into A, α/r coming out of B.
Simulated, n = f = 768, 2,000 random inputs. A random B (std 0.02) stands in for a trained one. LoRA code: Kaiming uniform, a = √5. rsLoRA: Kalajdzievski, 2023.
LoRA on ViT-B: the count
The MNIST run: r = 8, α = 16 on query and value, plus the new head
ONE ENCODERBLOCK, × 12, THEN THE HEAD · ❄️ FROZEN, 🔥 TRAINED
Linear q + LoRA
Linear k
Linear v + LoRA
Linear out
MLP
new head
One 768 × 768 Linear
589,824
Its LoRA pair, r = 8: 8 × (768 + 768) · 2.1% of the matrix
12,288
q and v in 12 blocks: 24 pairs
294,912
New head: 768 × 10 + 10
7,690
Trained, out of 86.1 M · 0.35% of the model
302,602
THE SAME TRICK ON GPT-3, 175 B
  • Checkpoint per task: 350 GB → 35 MB, about 10⁴× smaller
  • Training memory: 1.2 TB → 350 GB
  • Merged after training: no added inference latency
  • One shared base, many small adapters: swap the task by swapping a file
The base model stays shared; each task is a small file.
ViT counts: weights of the LoRA pairs, weights and biases of the head; 86,101,258 in all. GPT-3: Hu et al., 2021; the 35 MB is for r = 4 on query and value.
Adaptation methods accuracy
The same pretrained ViT on MNIST: head + frozen weights, LoRA + head, full fine-tuning
60%70%80%90%100%550500training images per digit6795.2% head + frozen weights7298.7% LoRA + head7899.1% full fine-tuning
BEFORE THE NEXT CLICK
Full fine-tuning wins at every size, and costs the most. When is LoRA the right call?
ADAPTATION 4 OF 4 · CHOOSING A METHOD
Pick capacity to match the data
The underfitting–overfitting story again
more capacity
Head + frozen weights
7,690 trained
LoRA + head
0.3 M trained
Full fine-tuning
86 M trained
  • Little data, task near the prior → LoRA: fewer parameters, less to overfit
  • Lots of data, or a task far from the prior → full fine-tuning has the room
  • Regularize the big model (early stopping) and it overfits less
OUR MNIST RUN
Full fine-tuning still led at 5 per digit: digits are far from ImageNet, and early stopping curbed overfitting.
Fewer trained weights, less to overfit: with little data, reach for LoRA first.
ADAPTATION 4 OF 4 · LoRA
Elbow X-rays from a chest X-ray prior
Same recipe as MNIST, on a medical task with little labelled data. Guess the accuracy
THE ADAPTATION
❄️ EVA-02 ViT
86 M weights
pretrained on 520,000+ chest X-rays
→
🔥 LoRA + head
elbow X-rays (MURA)
normal or abnormal
WHY IT WORKS
  • Chest and elbow X-rays share edges, bone, contrast
  • The prior already reads radiographs
  • LoRA only steers it to a new question
THE RUN
Weights trained
0.4%
Epochs
3
Time
≈ 13 min
GPU
free T4
Validation accuracy
? %≈ 80%
A good prior turns a small clinic dataset into a useful model in minutes.
From my NAIRR workshop (Knoxville): EVA-02 chest X-ray ViT adapted to MURA elbow X-rays with LoRA in a Colab notebook.
Takeaways
  • Cross-attention: queries from the decoder, keys and values from the encoder
  • Decoder block = encoder block + causal mask + cross-attention
  • Multi-task → meta-learning: one model, many tasks, then it learns to learn
  • A foundation model is an excellent prior over a whole domain
  • A good prior → low intrinsic dimension: a few directions adapt it
  • LoRA: W = W0 + (α/r) B A · W0 frozen · B = 0 at the start · merge after
  • Right tool: little data → LoRA; millions of examples → full training
Project 3: choose your generalist, then choose how to adapt it.