ADAPTATION 4 OF 4 · LoRA
LoRA by hand
One 4 × 4 layer, rank 1, α = 2, one input x
1 · BUILD ΔW = (α/r) b a, ONE ROW AT A TIME
×2
α/r
🔥 a
🔥 b
row 1 = 2 · b1 · a = 2 · 0.5 · [1, 0, 1, 1] = [1, 0, 1, 1]
row 2 = 2 · b2 · a = 2 · 0 · [1, 0, 1, 1] = [0, 0, 0, 0]
row 3 = 2 · b3 · a = 2 · −1 · [1, 0, 1, 1] = [−2, 0, −2, −2]
row 4 = 2 · b4 · a = 2 · 0 · [1, 0, 1, 1] = [0, 0, 0, 0]
ΔW is rebuilt from 8 numbers
Every row is a multiple of a: rank 1
=
x W₀ᵀ
+
x ΔWᵀ
=
ŷ
16 weights ❄️ frozen
8 weights 🔥 trained
After training, add ΔW into W0: one matrix again, no extra cost at inference.
b “after training” is illustrative; every other number follows from it. Blue cells positive, orange negative. In code b is B (n × r) and a is A (r × f).
© 2026 Dr. Jesse Roberts