CSC · TENNESSEE TECH
How does generative AI really work?
What AI is, how large language models write one token at a time, and Tennessee Tech’s new B.S. in Artificial Intelligence
Dr. Jesse Roberts
Department of Computer Science
PROMPT
…that cat jumped on the​
WHAT A MODEL THINKS COMES NEXT
"·alien"0.18
"·ship"0.12
"·table"0.08
"·roof"0.06
… and 50,253 more tokens
All slides, one per page
© 2026 Dr. Jesse Roberts
Today
01
What is AI?
Seventy years of ideas, and how AI, machine learning and data science fit together.
02
How does an LLM write?
Vectors, tokens, attention and next-token prediction. Then you get to be the language model.
03
Where do you fit?
Tennessee Tech’s B.S. in Artificial Intelligence: what you’ll take, and where it leads.
Who am I?
Dr. Jesse Roberts, Assistant Professor of Computer Science, Tennessee Tech
MY PATH WASN’T A STRAIGHT LINE
2010
Roane State Community College
2014
B.S. in EE, Tennessee Tech
2014–2021
7 years of software + hardware development
2017
M.S. in EE, Tennessee Tech
2024
Ph.D. Computer Science (AI), Vanderbilt
Now
Teaching and researching AI at Tech
What I work on
LAIR: Language and AI Research group
Deep learning fundamentals: what networks can compute, and why they train
Large language models: how they reason and handle uncertainty
Cherokee (Tsalagi) language tech, built with and for the community
Emerging: spiking neural networks, tornado detection
Torch Canvas: draw a network, get PyTorch code (right)
Built in Torch Canvasjessetnroberts.com/tools/torch-canvas
e × 784e × 256e × 256e × 256e × 512e × 512e × 512e × 512e × 512e × 256e × 256e × 256e × 256e × 256e × 512e × 512e × 512e × 512e × 512e × 256e × 256e × 256e × 256e × 256e × 512e × 512e × 512e × 512e × 512e × 256e × 256e × 256e × 256e × 256e × 256e × 256e × 256e × 256e × 256e × 256e × 10e × 10x, e × 784Linear256 × 784RMSNormLinear512 × 256Linear512 × 256σ⊙⊙Linear256 × 512+RMSNormLinear512 × 256Linear512 × 256σ⊙⊙Linear256 × 512+RMSNormLinear512 × 256Linear512 × 256σ⊙⊙Linear256 × 512+RMSNormLinear256 × 256Linear256 × 256σ⊙⊙Linear10 × 256SoftmaxP̂ e × 10
Community college, engineering, software + hardware, then AI: there is more than one road into this field.
What is AI?
?
Talk it over with a neighbour before we move on
When you hear “AI,” what do you picture?
What is AI?
?
Talk it over with a neighbour before we move on
When you hear “AI,” what do you picture?
ROUND TWO: AI OR NOT?
A calculator? A chess engine? Autocorrect? Your phone’s face unlock?
What is AI?
?
Talk it over with a neighbour before we move on
When you hear “AI,” what do you picture?
Getting computers to do things we would call intelligent if a person did them: search, recognize, decide, predict, create.
Adapted from Marvin Minsky (1968): “the science of making machines do things that would require intelligence if done by men”
What is AI?
?
Talk it over with a neighbour before we move on
When you hear “AI,” what do you picture?
Getting computers to do things we would call intelligent if a person did them: search, recognize, decide, predict, create.
Adapted from Marvin Minsky (1968): “the science of making machines do things that would require intelligence if done by men”
EVERY AI METHOD LEANS ON A BLEND OF THREE TRUNKS
Informed search
Explore many possibilities, guided by a sense of which ones look promising.
Ex: chess engines, GPS routes
What is AI?
?
Talk it over with a neighbour before we move on
When you hear “AI,” what do you picture?
Getting computers to do things we would call intelligent if a person did them: search, recognize, decide, predict, create.
Adapted from Marvin Minsky (1968): “the science of making machines do things that would require intelligence if done by men”
EVERY AI METHOD LEANS ON A BLEND OF THREE TRUNKS
Informed search
Explore many possibilities, guided by a sense of which ones look promising.
Ex: chess engines, GPS routes
Mathematics
Probability, linear algebra and calculus: describe data, measure error, improve a guess.
Ex: spam filters, A/B tests
What is AI?
?
Talk it over with a neighbour before we move on
When you hear “AI,” what do you picture?
Getting computers to do things we would call intelligent if a person did them: search, recognize, decide, predict, create.
Adapted from Marvin Minsky (1968): “the science of making machines do things that would require intelligence if done by men”
EVERY AI METHOD LEANS ON A BLEND OF THREE TRUNKS
Informed search
Explore many possibilities, guided by a sense of which ones look promising.
Ex: chess engines, GPS routes
Mathematics
Probability, linear algebra and calculus: describe data, measure error, improve a guess.
Ex: spam filters, A/B tests
Computing
Algorithms, systems and hardware fast enough to run the math on billions of examples.
Ex: GPUs, Oak Ridge’s Frontier
AI isn’t one invention. It is a toolbox grown from search, math and computing.
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
2026 you are here
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
2026 you are here
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
1956 Dartmouth summer workshop:
“artificial intelligence” is named
2026 you are here
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
1956 Dartmouth summer workshop:
“artificial intelligence” is named
1958 Rosenblatt’s perceptron learns from examples
2026 you are here
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
1956 Dartmouth summer workshop:
“artificial intelligence” is named
1958 Rosenblatt’s perceptron learns from examples
1969 ARPANET, the internet’s ancestor
2026 you are here
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
1956 Dartmouth summer workshop:
“artificial intelligence” is named
1958 Rosenblatt’s perceptron learns from examples
1969 ARPANET, the internet’s ancestor
2026 you are here
70 years The name came from John McCarthy’s 1955 proposal for the 1956 workshop. AI is older than the internet.
ChatGPT feels brand new, but it sits on seven decades of ideas. 
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
1956 Dartmouth summer workshop:
“artificial intelligence” is named
1958 Rosenblatt’s perceptron learns from examples
1969 ARPANET, the internet’s ancestor
2026 you are here
When will AI be as capable as a human? 20__ ?
ChatGPT feels brand new, but it sits on seven decades of ideas. 
How old is AI?
?
Talk it over with a neighbour before we move on
Guess the year the term “artificial intelligence” was coined. 19__ ? 20__ ?
1940
2030
1950 Alan Turing asks
“Can machines think?”
1956 Dartmouth summer workshop:
“artificial intelligence” is named
1958 Rosenblatt’s perceptron learns from examples
1969 ARPANET, the internet’s ancestor
2026 you are here
When will AI be as capable as a human? 20__ ?
1958 prediction: a perceptron “that will be able to read and write” would be finished “in about a year.” Later ones would “recognize people and call out their names.” (the Navy, in the New York Times)
Predictions are hard. 
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
INFORMED SEARCH
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
INFORMED SEARCH
MATHEMATICS
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
INFORMED SEARCH
MATHEMATICS
COMPUTING
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
1990s
Kernel methods
Ex: handwritten digit recognition
Probabilistic models
Ex: spam filters, speech recognition
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
1990s
Kernel methods
Ex: handwritten digit recognition
Probabilistic models
Ex: spam filters, speech recognition
2000s
Tree ensembles
Ex: credit-card fraud alerts, loan risk
Retrieval & indexing
Ex: web search, reverse image search
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
1990s
Kernel methods
Ex: handwritten digit recognition
Probabilistic models
Ex: spam filters, speech recognition
2000s
Tree ensembles
Ex: credit-card fraud alerts, loan risk
Retrieval & indexing
Ex: web search, reverse image search
2010s
Deep learning
Ex: face unlock, photo search, speech-to-text
Data & HPC systems
Ex: Oak Ridge’s Frontier
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
1990s
Kernel methods
Ex: handwritten digit recognition
Probabilistic models
Ex: spam filters, speech recognition
2000s
Tree ensembles
Ex: credit-card fraud alerts, loan risk
Retrieval & indexing
Ex: web search, reverse image search
2010s
Deep learning
Ex: face unlock, photo search, speech-to-text
Data & HPC systems
Ex: Oak Ridge’s Frontier
2015+
Deep reinforcement learning
Ex: AlphaGo, robots learning to walk
Spiking neural networks
Ex: Intel Loihi, low-power sensors
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
1990s
Kernel methods
Ex: handwritten digit recognition
Probabilistic models
Ex: spam filters, speech recognition
2000s
Tree ensembles
Ex: credit-card fraud alerts, loan risk
Retrieval & indexing
Ex: web search, reverse image search
2010s
Deep learning
Ex: face unlock, photo search, speech-to-text
Data & HPC systems
Ex: Oak Ridge’s Frontier
2015+
Deep reinforcement learning
Ex: AlphaGo, robots learning to walk
Spiking neural networks
Ex: Intel Loihi, low-power sensors
2020s
Large language & vision-language-action models · transformers · instruction tuning · learning from human feedback
Ex: Claude, ChatGPT, robots that follow spoken instructions
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
AI grows from three intertwined trunks
Left to right: the blend of search, math and computing a family leans on. Bottom to top: roughly when it was the field’s focus.
next
What comes next? The next branch hasn’t been named yet.
1950s & earlier
Heuristic search
Ex: chess engines, GPS routes
Statistical testing
Ex: comparing app designs
The perceptron
Ex: Mark I image recognizer
1960s
MDPs & dynamic programming
Ex: inventory planning
Optimization over candidates
Ex: delivery routing, training neural nets
1970s
Logic & expert systems
Ex: chip verification, class scheduling
Clustering
Ex: customer segments (no labels needed)
1980s
Classic reinforcement learning
Ex: TD-Gammon backgammon player
Multilayer perceptrons
Ex: ZIP-code and check readers
1990s
Kernel methods
Ex: handwritten digit recognition
Probabilistic models
Ex: spam filters, speech recognition
2000s
Tree ensembles
Ex: credit-card fraud alerts, loan risk
Retrieval & indexing
Ex: web search, reverse image search
2010s
Deep learning
Ex: face unlock, photo search, speech-to-text
Data & HPC systems
Ex: Oak Ridge’s Frontier
2015+
Deep reinforcement learning
Ex: AlphaGo, robots learning to walk
Spiking neural networks
Ex: Intel Loihi, low-power sensors
2020s
Large language & vision-language-action models · transformers · instruction tuning · learning from human feedback
Ex: Claude, ChatGPT, robots that follow spoken instructions
INFORMED SEARCH
MATHEMATICS
COMPUTING
Lineages:● neural networks■ sequential decisions◆ statistical learning▲ search & reasoningShadow = especially influential
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
ROUND TWO
Is all AI machine learning?
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
Machine learning is the automatic optimization of a model on a dataset.
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
Machine learning is the automatic optimization of a model on a dataset.
model
a structure that is adjustable (with params, connections, etc)
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
Machine learning is the automatic optimization of a model on a dataset.
model
a structure that is adjustable (with params, connections, etc)
dataset
examples of what it should do, like labelled photos
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
Machine learning is the automatic optimization of a model on a dataset.
model
a structure that is adjustable (with params, connections, etc)
dataset
examples of what it should do, like labelled photos
optimization
the computer changes the model to shrink mistakes on the data
No one writes “if pointy ears then cat.” The rules come out of the data.
What is machine learning?
?
Talk it over with a neighbour before we move on
How is machine learning different from ordinary programming?
Machine learning is the automatic optimization of a model on a dataset.
model
a structure that is adjustable (with params, connections, etc)
dataset
examples of what it should do, like labelled photos
optimization
the computer changes the model to shrink mistakes on the data
No one writes “if pointy ears then cat.” The rules come out of the data.
Artificial intelligence Machine learning Deep learning LLMs
Not all AI learns (a chess search can be pure rules), but nearly all modern AI is machine learning.
What is data science?
?
Talk it over with a neighbour before we move on
Is data science the same thing as AI?
What does a data scientist actually do all day?
Data science turns data into answers people can act on.
Collectsensors, surveys, logs, the web
Cleanfix gaps and errors (most of the job)
Exploreplots and statistics: what is going on?
Modeloften with machine learning
Communicateso someone can make a decision
Computing Math &statistics Domain knowledge Machinelearning Datascience
after Drew Conway’s data science Venn diagram (2010)
Data science is about answering questions with data and models.
Often those models are made through machine learning.
One family, many methods
AI, machine learning and data science overlap, and each draws on a big toolbox
AI Datascience Machine learning Deep learning
Search & reasoning
heuristic search · logic · planning · expert systems
Statistical learning
regression · decision trees · kernels · clustering · Bayesian models
Sequential decisions
dynamic programming · reinforcement learning
Neural networks
perceptron · multilayer perceptrons · deep learning
All of these are still in use. But one branch is behind almost every AI headline today.
Deep learning is behind today’s advances
Neural networks with many layers, trained on lots of data with lots of compute
2012
AlexNet
A deep network crushes the ImageNet image-recognition contest, trained on GPUs.
2016
AlphaGo
Deep networks plus search beat world champion Lee Sedol at Go.
2017
The transformer
“Attention Is All You Need” introduces the architecture behind every chatbot.
2020
AlphaFold 2
Predicts a protein’s 3D shape from its sequence, a 50-year-old biology problem.
2022
ChatGPT
A large language model goes public: a million users in five days.
2024
Two Nobel Prizes
Physics for the foundations of neural networks; chemistry for AlphaFold.
Different problems, one engine: deep neural networks. So how do they see the work?
What a neural network looks like today
The original transformer from “Attention Is All You Need” (2017), built for translation
e × 64e × 64 × 512e × 64 × 512e × 48e × 48 × 512e × 48 × 512e × 64 × 512e × 64 × 512e × 48 × 512e × 64 × 512e × 48 × 512e × 48 × 512e × 48 × 37000e × 48 × 37000e × 4848 × 512tiedtiedsrc, e × 64tgt, e × 48Embedding37000 × 512init: N σ=0.04419417×√d× constantsin/cos positions64 × 512Dropout 0.1Embedding37000 × 512tied to embedding×√d× constantsin/cos positions48 × 512Dropout 0.1EncoderLayer × 6512 → 512heads 8 · feed-forward width 2048LayerNormDecoderLayer × 6512, 512 → 512heads 8 · feed-forward width 2048LayerNormLinear37000 × 512tied to embeddinglogits e × 48 × 37000Cross-entropynext token · predict: softmax++
Not a web of little circles
A modern network is a pipeline of blocks, each one an operation: look up an embedding, add positions, attend, normalize, score.
Two stacks of 6 layers
The encoder reads the source sentence; the decoder writes the translation one token at a time, out of a vocabulary of 37,000 tokens.
Look at the arrows
Every one is labelled with a shape, like e × 64 × 512. What is flowing along them?
GPT, Claude and Gemini all descend from this design
Playing with an early LLM
GPT-2 (2019), an ancestor of today’s chatbots, writing one token at a time
Scan with your phone camera, or type the link.
Type a prompt and watch GPT-2 keep writing.
Try the start of a sentence you know. Does it finish the way you would?
Run the same prompt twice: why does the story change? Hold that thought.
Playing with an early LLM
GPT-2 (2019), an ancestor of today’s chatbots, writing one token at a time
Scan with your phone camera, or type the link.
Type a prompt and watch GPT-2 keep writing.
Try the start of a sentence you know. Does it finish the way you would?
Run the same prompt twice: why does the story change? Hold that thought.
?But how does it work?
What is a “language model”?
?
Talk it over with a neighbour before we move on
The “LM” in LLM. What does a language model actually do?
You have used several today. Your phone keyboard? A search box? A chatbot?
What is a “language model”?
?
Talk it over with a neighbour before we move on
The “LM” in LLM. What does a language model actually do?
You have used several today. Your phone keyboard? A search box? A chatbot?
ANSWER A system that predicts the next word given the context.
What is a “language model”?
?
Talk it over with a neighbour before we move on
The “LM” in LLM. What does a language model actually do?
You have used several today. Your phone keyboard? A search box? A chatbot?
ANSWER A system that predicts the next word given the context.
Phone keyboard
see you [tomorrow]
Search box
how to boil [an egg]
Chatbot
…cat jumped on the [alien]
From 1948 tables of counts to today’s transformers, every language model does this one job.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
4-gram
Once upon a
4-gram
Easier said than
5-gram
A penny for your
5-gram
The calm before the
Every chatbot today still does this one job: predict the next piece of text.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
Hermione
4-gram
Once upon a
4-gram
Easier said than
5-gram
A penny for your
5-gram
The calm before the
Every chatbot today still does this one job: predict the next piece of text.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
Hermione
4-gram
Once upon a
time
4-gram
Easier said than
5-gram
A penny for your
5-gram
The calm before the
Every chatbot today still does this one job: predict the next piece of text.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
Hermione
4-gram
Once upon a
time
4-gram
Easier said than
done
5-gram
A penny for your
5-gram
The calm before the
Every chatbot today still does this one job: predict the next piece of text.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
Hermione
4-gram
Once upon a
time
4-gram
Easier said than
done
5-gram
A penny for your
thoughts
5-gram
The calm before the
Every chatbot today still does this one job: predict the next piece of text.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
Hermione
4-gram
Once upon a
time
4-gram
Easier said than
done
5-gram
A penny for your
thoughts
5-gram
The calm before the
storm
Every chatbot today still does this one job: predict the next piece of text.
You are a language model
?
Shout out the missing word. This is Claude Shannon’s 1948 game.
N-GRAM
4-gram
Harry, Ron, and
Hermione
4-gram
Once upon a
time
4-gram
Easier said than
done
5-gram
A penny for your
thoughts
5-gram
The calm before the
storm
Count every N-gram in a big pile of text, then estimate
Pr( next word | the N−1 words before it )
and you have a language model: something that scores how likely each next word is.
Every chatbot today still does this one job: predict the next piece of text.
Looking up the next word
An N-gram model is a giant table of counts. Here N = 4, so the context is 3 words.
Contextonce upon a ​What comes next, and how likely is it?
Looking up the next word
An N-gram model is a giant table of counts. Here N = 4, so the context is 3 words.
Contextonce upon a ​What comes next, and how likely is it?
4-GRAM TABLE (counted from training text)
on the other hand
48,201
once in a while
12,430
once upon a dream
28
once upon a hill
61
once upon a midnight
143
once upon a time
9,812
one of the most
51,377
…millions more rows. Counts are illustrative.
Looking up the next word
An N-gram model is a giant table of counts. Here N = 4, so the context is 3 words.
Contextonce upon a ​What comes next, and how likely is it?
4-GRAM TABLE (counted from training text)
on the other hand
48,201
once in a while
12,430
once upon a dream
28
once upon a hill
61
once upon a midnight
143
once upon a time
9,812
one of the most
51,377
…millions more rows. Counts are illustrative.
MATCHES
9,812 + 143 + 61 + 28 = 10,044
Looking up the next word
An N-gram model is a giant table of counts. Here N = 4, so the context is 3 words.
Contextonce upon a ​What comes next, and how likely is it?
4-GRAM TABLE (counted from training text)
on the other hand
48,201
once in a while
12,430
once upon a dream
28
once upon a hill
61
once upon a midnight
143
once upon a time
9,812
one of the most
51,377
…millions more rows. Counts are illustrative.
MATCHES
9,812 + 143 + 61 + 28 = 10,044
P(word | once upon a) = count ÷ 10,044
time
0.977
midnight
0.014
hill
0.006
dream
0.003
Looking up the next word
An N-gram model is a giant table of counts. Here N = 4, so the context is 3 words.
Contextonce upon a ​What comes next, and how likely is it?
4-GRAM TABLE (counted from training text)
on the other hand
48,201
once in a while
12,430
once upon a dream
28
once upon a hill
61
once upon a midnight
143
once upon a time
9,812
one of the most
51,377
…millions more rows. Counts are illustrative.
MATCHES
9,812 + 143 + 61 + 28 = 10,044
P(word | once upon a) = count ÷ 10,044
time
0.977
midnight
0.014
hill
0.006
dream
0.003
Look up, count, divide: that is a whole N-gram language model.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
CONTEXT 1
Tom loved playing around the fence. Actually, one time, …
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
CONTEXT 1
Tom loved playing around the fence. Actually, one time, …
→ fence
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
CONTEXT 1
Tom loved playing around the fence. Actually, one time, …
→ fence
CONTEXT 2
Tom watched me catapult the toy onto the couch so …
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
CONTEXT 1
Tom loved playing around the fence. Actually, one time, …
→ fence
CONTEXT 2
Tom watched me catapult the toy onto the couch so …
→ couch
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
CONTEXT 1
Tom loved playing around the fence. Actually, one time, …
→ fence
CONTEXT 2
Tom watched me catapult the toy onto the couch so …
→ couch
CONTEXT 3
The spaceship landed with a thud. The green men came down the chute and before I knew it …
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Context changes the answer
?
The last five words are identical every time. Finish the sentence.
…that cat jumped on the ​
CONTEXT 1
Tom loved playing around the fence. Actually, one time, …
→ fence
CONTEXT 2
Tom watched me catapult the toy onto the couch so …
→ couch
CONTEXT 3
The spaceship landed with a thud. The green men came down the chute and before I knew it …
→ alien?
An N-gram only sees the last few words. The clue can be a whole paragraph back.
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
that cat jumped on the ​
210
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
that cat jumped on the ​
210
one time, that cat jumped on the ​
3
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
that cat jumped on the ​
210
one time, that cat jumped on the ​
3
Actually, one time, that cat jumped on the ​
0 never seen
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
that cat jumped on the ​
210
one time, that cat jumped on the ​
3
Actually, one time, that cat jumped on the ​
0 never seen
Tom loved playing around the fence. Actually, one time, that cat jumped on the ​
0 never seen
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
that cat jumped on the ​
210
one time, that cat jumped on the ​
3
Actually, one time, that cat jumped on the ​
0 never seen
Tom loved playing around the fence. Actually, one time, that cat jumped on the ​
0 never seen
20-word sequences from a 50,000-word vocabulary: 50,000²⁰ ≈ 10⁹⁴Atoms in the observable universe: about 10⁸⁰
Longer context, fewer matches
Look up the same blank with more and more of the context
CONTEXT
TIMES SEEN IN THE TABLE (log scale, illustrative)
the ​
1,200,000,000
on the ​
38,000,000
jumped on the ​
41,000
cat jumped on the ​
1,900
that cat jumped on the ​
210
one time, that cat jumped on the ​
3
Actually, one time, that cat jumped on the ​
0 never seen
Tom loved playing around the fence. Actually, one time, that cat jumped on the ​
0 never seen
20-word sequences from a 50,000-word vocabulary: 50,000²⁰ ≈ 10⁹⁴Atoms in the observable universe: about 10⁸⁰
With enough context, every sentence is unique: zero matches, nothing to divide. We need a model that generalizes.
Not every word matters equally
A lookup table treats every word in the context the same. Should it?
The spaceship landed with a thud. The green men came down the chute and before I knew it that cat jumped on the ​
Attention (2017) is the heart of the transformer, the network inside GPT and every modern chatbot.
Not every word matters equally
A lookup table treats every word in the context the same. Should it?
The spaceship landed with a thud. The green men came down the chute and before I knew it that cat jumped on the ​
Hide most of the words. You can still guess the ending, because a few words carry the meaning.
Attention (2017) is the heart of the transformer, the network inside GPT and every modern chatbot.
Not every word matters equally
A lookup table treats every word in the context the same. Should it?
The spaceship landed with a thud. The green men came down the chute and before I knew it that cat jumped on the ​
Hide most of the words. You can still guess the ending, because a few words carry the meaning.
You know what deserves your attention. A transformer learns the same skill: for every word, how much should each other word count? That is attention.
Attention (2017) is the heart of the transformer, the network inside GPT and every modern chatbot.
So, what is a large language model?
Our map for the next few slides. Values ride the arrows; boxes do the work.
Tokenize
step 1
Embed
step 2: look up a row
Attention → MLP × 12
step 3: combine, then untangle
· vocabulary table
step 4: score every token
softmax
scores → probabilities
Choose one
step 5: sample
text
token ids
768 numbers
per token
final vector
50,257 scores
probabilities
append it, repeat
Numbers are for GPT-2 small (2019): a 50,257-token vocabulary, 768 numbers per token, 12 blocks. Today’s chatbots have the same parts, just far bigger.
An LLM is one big loop: predict a distribution, pick a token, append it, go again.
Step 1: text becomes tokens
A token is a common chunk of text: a word or a piece of one. GPT-2 knows 50,257.
The464
·spaceship40,663
·landed11,406
·with351
·a257
·th294
ud463
.13
·The383
·green4,077
·men1,450
·came1,625
·down866
·the262
·shoot2,686
·and290
·before878
·I314
·knew2,993
·it340
·that326
·cat3,797
·jumped11,687
·on319
·the262
· = the token begins with a space. 25 tokens in all.
Step 1: text becomes tokens
A token is a common chunk of text: a word or a piece of one. GPT-2 knows 50,257.
The464
·spaceship40,663
·landed11,406
·with351
·a257
·th294
ud463
.13
·The383
·green4,077
·men1,450
·came1,625
·down866
·the262
·shoot2,686
·and290
·before878
·I314
·knew2,993
·it340
·that326
·cat3,797
·jumped11,687
·on319
·the262
· = the token begins with a space. 25 tokens in all.
Step 1: text becomes tokens
A token is a common chunk of text: a word or a piece of one. GPT-2 knows 50,257.
The464
·spaceship40,663
·landed11,406
·with351
·a257
·th294
ud463
.13
·The383
·green4,077
·men1,450
·came1,625
·down866
·the262
·shoot2,686
·and290
·before878
·I314
·knew2,993
·it340
·that326
·cat3,797
·jumped11,687
·on319
·the262
· = the token begins with a space. 25 tokens in all.
The model never sees letters, only a list of token IDs.
Step 1: text becomes tokens
A token is a common chunk of text: a word or a piece of one. GPT-2 knows 50,257.
The464
·spaceship40,663
·landed11,406
·with351
·a257
·th294
ud463
.13
·The383
·green4,077
·men1,450
·came1,625
·down866
·the262
·shoot2,686
·and290
·before878
·I314
·knew2,993
·it340
·that326
·cat3,797
·jumped11,687
·on319
·the262
· = the token begins with a space. 25 tokens in all.
“thud” is two tokens
Rare words get split into pieces the model has seen often: ·th + ud.
“The” is not “·The”
The first word is token 464; after a space it is token 383. To the model, every token is just an ID.
The model never sees letters, only a list of token IDs.
Step 1: text becomes tokens
A token is a common chunk of text: a word or a piece of one. GPT-2 knows 50,257.
The464
·spaceship40,663
·landed11,406
·with351
·a257
·th294
ud463
.13
·The383
·green4,077
·men1,450
·came1,625
·down866
·the262
·shoot2,686
·and290
·before878
·I314
·knew2,993
·it340
·that326
·cat3,797
·jumped11,687
·on319
·the262
· = the token begins with a space. 25 tokens in all.
“thud” is two tokens
Rare words get split into pieces the model has seen often: ·th + ud.
“The” is not “·The”
The first word is token 464; after a space it is token 383. To the model, every token is just an ID.
Each ID is an address: 464 looks up row 464, that token’s own vector, called its embedding.
Deep learning runs on vectors
Before a network can do anything, the world has to become lists of numbers.
An image
[0, 0, …, 0, 1, 1, 1, 1, …]
one number per pixel: 64 here,
about 36 million for a phone photo
A sound
[0.55, 0.55, −0.47, −0.53, …]
the wave’s height, measured
16,000 times a second for speech
A word
cat
[?, ?, ?, ?, …]
no obvious numbers at all.
We have to invent them.
Back to the transformer: every arrow carries vectors (e × 64 × 512 = 512 numbers for each of 64 tokens in each of e sentences), and every block transforms them.
vector in → block → vector → block → … → vector out
Images and sounds come with numbers. For words, we need a way to turn meaning into a vector. So what is a vector, exactly?
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
is also a point, or an arrow from the origin: 3 steps right, 2 steps up.
(3, 2) x y
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
is also a point, or an arrow from the origin: 3 steps right, 2 steps up.
(3, 2) x y
More numbers, more dimensions
[3, 2]a point on a page
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
is also a point, or an arrow from the origin: 3 steps right, 2 steps up.
(3, 2) x y
More numbers, more dimensions
[3, 2]a point on a page
[3, 2, 4]a point in a room
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
is also a point, or an arrow from the origin: 3 steps right, 2 steps up.
(3, 2) x y
More numbers, more dimensions
[3, 2]a point on a page
[3, 2, 4]a point in a room
[0.58, 0.50, −0.16, …]5 dimensions
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
is also a point, or an arrow from the origin: 3 steps right, 2 steps up.
(3, 2) x y
More numbers, more dimensions
[3, 2]a point on a page
[3, 2, 4]a point in a room
[0.58, 0.50, −0.16, …]5 dimensions
768 numbers768 dimensions
We can’t picture 768 dimensions, but the math (distances, directions) works exactly the same.
What is a vector?
?
Talk it over with a neighbour before we move on
What is a vector? Where have you seen one before?
Physics class? A video game? A GPS?
A list of numbers
[3, 2]
is also a point, or an arrow from the origin: 3 steps right, 2 steps up.
(3, 2) x y
More numbers, more dimensions
[3, 2]a point on a page
[3, 2, 4]a point in a room
[0.58, 0.50, −0.16, …]5 dimensions
768 numbers768 dimensions
We can’t picture 768 dimensions, but the math (distances, directions) works exactly the same.
Each number is a position along one axis: a vector is a point in space.
What is an embedding?
Neural networks only see numbers. Ever taken a personality test? It turns you into numbers.
TOM’S PERSONALITY TEST
Openness to experience79 / 100
Agreeableness75 / 100
Conscientiousness42 / 100
Negative emotionality50 / 100
Extraversion59 / 100
A list of questions → a score on each of five axes
rescale to −1 … 1
(score − 50) / 50
TOM AS A VECTOR
0.58
0.50
−0.16
0.00
0.18
Blue: above average · orange: below · gray: average
An embedding is a vector used to characterize something. This one characterizes Tom.
Who is most like Tom?
?
Talk it over with a neighbour before we move on
Tom
Openness0.58
Agreeableness0.50
Conscientiousness−0.16
Neg. emotionality0.00
Extraversion0.18
Marvin
Openness−0.40
Agreeableness0.80
Conscientiousness−0.32
Neg. emotionality−0.10
Extraversion0.51
Janet
Openness0.31
Agreeableness0.60
Conscientiousness0.34
Neg. emotionality−0.30
Extraversion−0.36
Is Marvin or Janet more similar to Tom? How could you measure “similar” with just these numbers?
People as points in space
Each number in the embedding is one axis. Add axes and the picture changes.
−1
1
Openness →
Tom
Marvin
Janet
1 dimension: openness only
A number line. Janet sits right next to Tom; Marvin is far away.
Similar things should be close together in the space. That is the whole point of an embedding.
People as points in space
Each number in the embedding is one axis. Add axes and the picture changes.
−1
1
Openness →
↑ Agreeableness
−1
Tom
Marvin
Janet
1 dimension: openness only
A number line. Janet sits right next to Tom; Marvin is far away.
2 dimensions: + agreeableness
A plane. Everyone moves up, but Janet is still closest to Tom.
Similar things should be close together in the space. That is the whole point of an embedding.
People as points in space
Each number in the embedding is one axis. Add axes and the picture changes.
−1
1
Openness →
↑ Agreeableness
−1
Tom
Marvin
Janet
1 dimension: openness only
A number line. Janet sits right next to Tom; Marvin is far away.
2 dimensions: + agreeableness
A plane. Everyone moves up, but Janet is still closest to Tom.
5 dimensions: all of Tom
Impossible to draw, but the math works the same in 5, 768 or 12,288 dimensions.
Similar things should be close together in the space. That is the whole point of an embedding.
Distance in any number of dimensions
Pythagoras from middle school, one more square per dimension
Tom vs Janet · 1 dimension openness → Janet 0.31 Tom 0.58 d = 0.27 d = √( 0.27² ) = 0.27
1Dd = √( Δx² )Δx = x₂ − x₁
dimensions
Tom → Marvin
Tom → Janet
1
0.98
0.27
Distance in any number of dimensions
Pythagoras from middle school, one more square per dimension
Tom vs Janet · 2 dimensions openness → ↑ agreeableness Janet Tom 0.27 0.10 d d = √( 0.27² + 0.10² ) = 0.29
1Dd = √( Δx² )Δx = x₂ − x₁
2Dd = √( Δx² + Δy² )
dimensions
Tom → Marvin
Tom → Janet
1
0.98
0.27
2
1.03
0.29
Distance in any number of dimensions
Pythagoras from middle school, one more square per dimension
Tom vs Janet · 3 dimensions across: openness · depth: conscientiousness · up: agreeableness Janet Tom 0.27 0.50 0.10 d d = √( 0.27² + 0.10² + 0.50² ) = 0.58
1Dd = √( Δx² )Δx = x₂ − x₁
2Dd = √( Δx² + Δy² )
3Dd = √( Δx² + Δy² + Δz² )
dimensions
Tom → Marvin
Tom → Janet
1
0.98
0.27
2
1.03
0.29
3
1.04
0.58
Distance in any number of dimensions
Pythagoras from middle school, one more square per dimension
Tom vs Janet · all 5 traitsgap between Tom and Janet, one per axisOpenness 0.27 Agreeableness 0.10 Conscientiousness 0.50 Neg. emotionality 0.30 Extraversion 0.54 d = √( 0.27² + 0.10² + 0.50² + 0.30² + 0.54² ) = 0.85 Can’t draw it, but the arithmetic is the same.
1Dd = √( Δx² )Δx = x₂ − x₁
2Dd = √( Δx² + Δy² )
3Dd = √( Δx² + Δy² + Δz² )
any nd = √( Δ₁² + Δ₂² + … + Δₙ² )
dimensions
Tom → Marvin
Tom → Janet
1
0.98
0.27
2
1.03
0.29
3
1.04
0.58
5
1.09
0.85
Distance in any number of dimensions
Pythagoras from middle school, one more square per dimension
Tom vs Janet · all 5 traitsgap between Tom and Janet, one per axisOpenness 0.27 Agreeableness 0.10 Conscientiousness 0.50 Neg. emotionality 0.30 Extraversion 0.54 d = √( 0.27² + 0.10² + 0.50² + 0.30² + 0.54² ) = 0.85 Can’t draw it, but the arithmetic is the same.
1Dd = √( Δx² )Δx = x₂ − x₁
2Dd = √( Δx² + Δy² )
3Dd = √( Δx² + Δy² + Δz² )
any nd = √( Δ₁² + Δ₂² + … + Δₙ² )
dimensions
Tom → Marvin
Tom → Janet
1
0.98
0.27
2
1.03
0.29
3
1.04
0.58
5
1.09
0.85
With all five traits, Janet is still closer to Tom, but by a lot less than one trait suggested.
Words need numbers too
Why can’t we just number the words in the dictionary?
Alphabetical numbers
The numbers say “cat” is next to “catapult” and far from “kitten.”
cat catapult kitten 1,021 1,022 3,977 close in number, unrelated in meaning
Meaning as location
Similar meanings get similar vectors (sketch in 2D).
cat kitten dog puppy catapult cannon trebuchet
“You shall know a word by the company it keeps.”J. R. Firth, 1957
Word2vec (2013) learned each word’s location from the words around it, across billions of sentences.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king
king − man + woman ≈ ?
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king queen
king − man + woman ≈ queen
Start at king, take the man→woman step.
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king queen
king − man + woman ≈ queen
Start at king, take the man→woman step.
capital of France Paris Germany
Paris − France + Germany ≈ ?
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king queen
king − man + woman ≈ queen
Start at king, take the man→woman step.
capital of France Paris Germany Berlin
Paris − France + Germany ≈ Berlin
The same “capital of” step works for every country.
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king queen
king − man + woman ≈ queen
Start at king, take the man→woman step.
capital of France Paris Germany Berlin
Paris − France + Germany ≈ Berlin
The same “capital of” step works for every country.
+ woman − man man woman doctor
doctor − man + woman ≈ ?
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king queen
king − man + woman ≈ queen
Start at king, take the man→woman step.
capital of France Paris Germany Berlin
Paris − France + Germany ≈ Berlin
The same “capital of” step works for every country.
+ woman − man man woman doctor nurse
doctor − man + woman ≈ nurse
Learned from human text, stereotypes included.
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Semantic math
Directions in the space carry meaning, so you can do arithmetic with words
+ woman − man man woman king queen
king − man + woman ≈ queen
Start at king, take the man→woman step.
capital of France Paris Germany Berlin
Paris − France + Germany ≈ Berlin
The same “capital of” step works for every country.
+ woman − man man woman doctor nurse
doctor − man + woman ≈ nurse
Learned from human text, stereotypes included.
Sketches in 2D. Real word embeddings have hundreds of dimensions.
Nobody programmed these directions. They emerged from reading text, biases included.
Step 2: tokens become vectors
The embedding table has one learned row of 768 numbers for every token in the vocabulary
rowdim 1dim 2dim 3dim 766dim 767dim 768
0−0.1101−0.03930.0331…−0.13640.01510.0453
10.0403−0.04860.0462…0.08610.00250.0432
2−0.12750.04790.1841…0.0899−0.1297−0.0879
⋮⋮⋮⋮⋮⋮⋮
464−0.0686−0.02030.0645…0.0639−0.01000.0031
⋮⋮⋮⋮⋮⋮⋮
50,254−0.0445−0.05480.0123…0.10440.0978−0.0695
50,2550.18600.01670.0461…−0.09630.0785−0.0225
50,2560.0514−0.02770.0499…0.00700.15520.1207
embedding table: 50,257 rows × 768 columns (GPT-2 small)
“The” = 464
go to row 464
“THE” AS A VECTOR
−0.0686
−0.0203
0.0645
⋮ 762 more
0.0639
−0.0100
0.0031
Do this for all 25 tokens: a 25 × 768 grid of numbers goes into the network.
Like Tom’s personality scores, but 768 traits per token, and the model learns them all.
Step 3: attention
To predict the next word, how much should the last token “the” listen to each earlier token?
The
·spaceship
·landed
·with
·a
·th
ud
.
·The
·green
·men
·came
·down
·the
·chute
·and
·before
·I
·knew
·it
·that
·cat
·jumped
·on
·thequery
spaceship
landed
green
men
cat
20 others
Query of “the”: what am I about to name?
Key of “spaceship”: I’m a vehicle, something you land.
Value: what a token passes along if it is picked.
WQ, WK, WV are learned. Dividing by 8 (√64) keeps scores from blowing up. Scores here are illustrative.
Step 3: attention
To predict the next word, how much should the last token “the” listen to each earlier token?
The
·spaceship
·landed
·with
·a
·th
ud
.
·The
·green
·men
·came
·down
·the
·chute
·and
·before
·I
·knew
·it
·that
·cat
·jumped
·on
·thequery
spaceship
landed
green
men
cat
20 others
match score q·k
40
33
34
37
31
≈18
Query of “the”: what am I about to name?
Key of “spaceship”: I’m a vehicle, something you land.
Value: what a token passes along if it is picked.
WQ, WK, WV are learned. Dividing by 8 (√64) keeps scores from blowing up. Scores here are illustrative.
Step 3: attention
To predict the next word, how much should the last token “the” listen to each earlier token?
The
·spaceship
·landed
·with
·a
·th
ud
.
·The
·green
·men
·came
·down
·the
·chute
·and
·before
·I
·knew
·it
·that
·cat
·jumped
·on
·thequery
spaceship
landed
green
men
cat
20 others
match score q·k
40
33
34
37
31
≈18
÷ 8, then softmax
0.20
0.08
0.10
0.14
0.07
0.41
attention weights: each between 0 and 1, all 25 adding up to 1
Query of “the”: what am I about to name?
Key of “spaceship”: I’m a vehicle, something you land.
Value: what a token passes along if it is picked.
WQ, WK, WV are learned. Dividing by 8 (√64) keeps scores from blowing up. Scores here are illustrative.
Step 3: attention
To predict the next word, how much should the last token “the” listen to each earlier token?
The0.01
·spaceship0.20
·landed0.09
·with0.01
·a0.01
·th0.03
ud0.03
.0.01
·The0.01
·green0.10
·men0.14
·came0.01
·down0.01
·the0.01
·chute0.07
·and0.01
·before0.01
·I0.01
·knew0.01
·it0.01
·that0.01
·cat0.07
·jumped0.05
·on0.01
·thequery
spaceship
landed
green
men
cat
20 others
match score q·k
40
33
34
37
31
≈18
÷ 8, then softmax
0.20
0.08
0.10
0.14
0.07
0.41
attention weights: each between 0 and 1, all 25 adding up to 1
Query of “the”: what am I about to name?
Key of “spaceship”: I’m a vehicle, something you land.
Value: what a token passes along if it is picked.
WQ, WK, WV are learned. Dividing by 8 (√64) keeps scores from blowing up. Scores here are illustrative.
Step 3: attention
To predict the next word, how much should the last token “the” listen to each earlier token?
The0.01
·spaceship0.20
·landed0.09
·with0.01
·a0.01
·th0.03
ud0.03
.0.01
·The0.01
·green0.10
·men0.14
·came0.01
·down0.01
·the0.01
·chute0.07
·and0.01
·before0.01
·I0.01
·knew0.01
·it0.01
·that0.01
·cat0.07
·jumped0.05
·on0.01
·thequery
spaceship
landed
green
men
cat
20 others
match score q·k
40
33
34
37
31
≈18
÷ 8, then softmax
0.20
0.08
0.10
0.14
0.07
0.41
attention weights: each between 0 and 1, all 25 adding up to 1
new “the”
0.20·v(spaceship) + 0.14·v(men)
+ 0.10·v(green) + … (all 25)
Query of “the”: what am I about to name?
Key of “spaceship”: I’m a vehicle, something you land.
Value: what a token passes along if it is picked.
WQ, WK, WV are learned. Dividing by 8 (√64) keeps scores from blowing up. Scores here are illustrative.
“the” now carries a bit of spaceship, men and green, so the next word leans toward alien.
Combine, untangle, repeat
Attention combines what the words say; an MLP then untangles each word’s vector
Attention
combine words by importance
MLP
untangle each word’s vector
× 12
25 × 768 numbers
final vector of the
last token “the”
on to step 4
Inside the MLP: a classic neural network
768 in3,072 hidden768 out
124 million
learned numbers in GPT-2 small
About ⅔ of each block’s numbers sit in the MLP
much of what the model “knows”
The same small network runs on every token’s vector, in every one of the 12 blocks.
Stack the block 12 times (96 in GPT-3) and the last token’s vector points toward what comes next.
Step 4: a probability for every token
“…that cat jumped on the ___”: the model checks every token it knows and gives each one a chance
will 
top 
of 
through 
be 
can 
again 
roof 
from 
time 
already 
all 
by 
his 
man 
being 
both 
1997 
ship 
world 
right 
state 
good 
over 
a 
these 
counter 
ramp 
in 
what 
would 
it 
banana 
alien 
been 
how 
then 
them 
might 
may 
their 
great 
house 
last 
under 
because 
my 
about 
number 
that 
end 
. 
should 
only 
well 
have 
into 
window 
years 
this 
off 
ly 
if 
, 
most 
large 
Paris 
after 
he 
here 
table 
some 
or 
made 
long 
two 
was 
chute 
: 
her 
high 
year 
couch 
still 
same 
me 
so 
not 
also 
with 
much 
more 
just 
than 
floor 
many 
there 
door 
those 
now 
other 
which 
we 
down 
!! 
up 
part 
few 
## 
on 
one 
has 
must 
no 
ground 
before 
any 
could 
best 
around 
came 
; 
ed 
you 
even 
old 
were 
who 
very 
own 
at 
without 
fence 
ing 
new 
are 
the 
I 
out 
and 
life 
for 
home 
when 
is 
day 
spaceship 
an 
er 
? 
every 
place 
small 
Tuesday 
they 
to 
another 
car 
where 
city 
cat 
she 
green 
people 
our 
back 
hood 
but 
while 
creature 
however 
like 
dog 
never 
robot 
bed 
quantum 
first 
as 
do 
… 50,077 more
Step 4: a probability for every token
“…that cat jumped on the ___”: the model checks every token it knows and gives each one a chance
will≈ 0
top0.003
of≈ 0
through≈ 0
be≈ 0
can0.0003
again≈ 0
roof0.06
from≈ 0
time≈ 0
already≈ 0
all≈ 0
by0.0003
his≈ 0
man0.04
being≈ 0
both≈ 0
1997≈ 0
ship0.12
world0.0004
right≈ 0
state≈ 0
good≈ 0
over≈ 0
a≈ 0
these≈ 0
counter0.008
ramp0.05
in≈ 0
what≈ 0
would≈ 0
it0.0003
banana≈ 0
alien0.18
been≈ 0
how0.0002
then≈ 0
them≈ 0
might≈ 0
may≈ 0
their0.0001
great≈ 0
house0.0002
last≈ 0
under≈ 0
because≈ 0
my≈ 0
about≈ 0
number≈ 0
that≈ 0
end≈ 0
.≈ 0
should≈ 0
only≈ 0
well≈ 0
have≈ 0
into≈ 0
window0.002
years≈ 0
this0.0005
off≈ 0
ly≈ 0
if≈ 0
,≈ 0
most≈ 0
large≈ 0
Paris≈ 0
after≈ 0
he0.0004
here≈ 0
table0.08
some≈ 0
or≈ 0
made≈ 0
long≈ 0
two0.0004
was≈ 0
chute0.03
:≈ 0
her0.0002
high≈ 0
year≈ 0
couch0.03
still0.0001
same≈ 0
me≈ 0
so≈ 0
not≈ 0
also0.0002
with≈ 0
much≈ 0
more≈ 0
just0.0003
than≈ 0
floor0.04
many≈ 0
there≈ 0
door0.003
those≈ 0
now≈ 0
other0.003
which0.0005
we≈ 0
down≈ 0
!!≈ 0
up≈ 0
part≈ 0
few≈ 0
##≈ 0
on0.0003
one≈ 0
has≈ 0
must≈ 0
no≈ 0
ground0.01
before≈ 0
any≈ 0
could≈ 0
best≈ 0
around0.0004
came≈ 0
;0.0002
ed≈ 0
you≈ 0
even≈ 0
old≈ 0
were≈ 0
who≈ 0
very0.0001
own≈ 0
at≈ 0
without0.0002
fence0.005
ing≈ 0
new0.0001
are≈ 0
the0.0001
I0.0002
out≈ 0
and≈ 0
life≈ 0
for0.0002
home≈ 0
when≈ 0
is≈ 0
day≈ 0
spaceship0.01
an≈ 0
er≈ 0
?≈ 0
every≈ 0
place≈ 0
small≈ 0
Tuesday≈ 0
they≈ 0
to0.0002
another≈ 0
car0.006
where≈ 0
city≈ 0
cat0.002
she≈ 0
green0.004
people≈ 0
our≈ 0
back0.002
hood0.02
but0.0002
while≈ 0
creature0.007
however≈ 0
like0.0004
dog0.002
never≈ 0
robot0.004
bed0.009
quantum≈ 0
first≈ 0
as≈ 0
do≈ 0
… 50,077 more
Step 4: a probability for every token
“…that cat jumped on the ___”: the model checks every token it knows and gives each one a chance
will≈ 0
top0.003
of≈ 0
through≈ 0
be≈ 0
can0.0003
again≈ 0
roof0.06
from≈ 0
time≈ 0
already≈ 0
all≈ 0
by0.0003
his≈ 0
man0.04
being≈ 0
both≈ 0
1997≈ 0
ship0.12
world0.0004
right≈ 0
state≈ 0
good≈ 0
over≈ 0
a≈ 0
these≈ 0
counter0.008
ramp0.05
in≈ 0
what≈ 0
would≈ 0
it0.0003
banana≈ 0
alien0.18
been≈ 0
how0.0002
then≈ 0
them≈ 0
might≈ 0
may≈ 0
their0.0001
great≈ 0
house0.0002
last≈ 0
under≈ 0
because≈ 0
my≈ 0
about≈ 0
number≈ 0
that≈ 0
end≈ 0
.≈ 0
should≈ 0
only≈ 0
well≈ 0
have≈ 0
into≈ 0
window0.002
years≈ 0
this0.0005
off≈ 0
ly≈ 0
if≈ 0
,≈ 0
most≈ 0
large≈ 0
Paris≈ 0
after≈ 0
he0.0004
here≈ 0
table0.08
some≈ 0
or≈ 0
made≈ 0
long≈ 0
two0.0004
was≈ 0
chute0.03
:≈ 0
her0.0002
high≈ 0
year≈ 0
couch0.03
still0.0001
same≈ 0
me≈ 0
so≈ 0
not≈ 0
also0.0002
with≈ 0
much≈ 0
more≈ 0
just0.0003
than≈ 0
floor0.04
many≈ 0
there≈ 0
door0.003
those≈ 0
now≈ 0
other0.003
which0.0005
we≈ 0
down≈ 0
!!≈ 0
up≈ 0
part≈ 0
few≈ 0
##≈ 0
on0.0003
one≈ 0
has≈ 0
must≈ 0
no≈ 0
ground0.01
before≈ 0
any≈ 0
could≈ 0
best≈ 0
around0.0004
came≈ 0
;0.0002
ed≈ 0
you≈ 0
even≈ 0
old≈ 0
were≈ 0
who≈ 0
very0.0001
own≈ 0
at≈ 0
without0.0002
fence0.005
ing≈ 0
new0.0001
are≈ 0
the0.0001
I0.0002
out≈ 0
and≈ 0
life≈ 0
for0.0002
home≈ 0
when≈ 0
is≈ 0
day≈ 0
spaceship0.01
an≈ 0
er≈ 0
?≈ 0
every≈ 0
place≈ 0
small≈ 0
Tuesday≈ 0
they≈ 0
to0.0002
another≈ 0
car0.006
where≈ 0
city≈ 0
cat0.002
she≈ 0
green0.004
people≈ 0
our≈ 0
back0.002
hood0.02
but0.0002
while≈ 0
creature0.007
however≈ 0
like0.0004
dog0.002
never≈ 0
robot0.004
bed0.009
quantum≈ 0
first≈ 0
as≈ 0
do≈ 0
… 50,077 more
A few tokens take most of the chance: alien 0.18, ship 0.12, table 0.08Tens of thousands get almost nothing (≈ 0)Add up all 50,257: exactly 1
The output is not a word. It is a probability for every one of the 50,257 tokens.
Telling apart vs. making new
Same cats (● orange), same dogs (■ blue), two very different questions
Discriminative AI
Learns the boundary between classes, then labels new data by which side it falls on.
Generative AI
Learns where each class lives (its distribution), then samples brand-new data from it.
? → “cat” boundary ☆ new samples
Discriminative asks “which one is this?”; generative asks “what else could exist?” An LLM is generative: it learned a distribution over text.
LLMs learn a distribution
Probabilities for the token after our spaceship story, largest first
prompt = "…that cat jumped on the"
alien0.18ship0.12table0.08roof0.06ramp0.05floor0.04man0.04chute0.03couch0.03hood0.02 … 50,247 more
Top 10 tokens hold about 65% of the probability.
Same prompt, same bars. The network itself is deterministic.
Illustrative values with the shape a real model gives. Like the generative model on the cats-and-dogs slide, it learned a distribution, here over words.
The model does not pick a word. It scores every possible next token; something else has to choose.
Which token do we pick?
?
Talk it over with a neighbour before we move on
…that cat jumped on the___
0.18alien
0.12ship
0.08table
0.06roof
0.05ramp
0.04floor
0.04man
0.03chute
0.03couch
0.02hood
The model gives us 50,257 probabilities. How should we choose the next token?
Always the most likely one?
Any token, at random?
Something in between?
What would go wrong with each?
Three ways to choose
Same prompt, same probabilities. Only the choosing rule changes. (Example outputs are illustrative.)
…that cat jumped on the___
only “alien”
0.18alien
0.12ship
0.08table
0.06roof
0.05ramp
0.04floor
0.04man
0.03chute
0.03couch
0.02hood
1 OF 3 · GREEDY
Always the most likely
Pick the tallest bar every time.
…that cat jumped on the alien. The alien jumped on the cat. The alien jumped on the cat. The alien jumped on the cat.
Repetitive: it loops.
Chatbots sample from the top of the distribution: likely enough to make sense, random enough to be new.
Three ways to choose
Same prompt, same probabilities. Only the choosing rule changes. (Example outputs are illustrative.)
…that cat jumped on the___
all 50,257
0.18alien
0.12ship
0.08table
0.06roof
0.05ramp
0.04floor
0.04man
0.03chute
0.03couch
0.02hood
2 OF 3 · RANDOM
Any token, by its odds
Roll the dice over all 50,257 tokens. Rare, weird tokens sometimes win.
…that cat jumped on the hood ordinance, whereupon Tuesday’s marmalade committee reorganized the plumbing.
…that cat jumped on the sandwich 1997 orbital spreadsheets, said nobody of the guild.
Unhinged: it wanders off.
Chatbots sample from the top of the distribution: likely enough to make sense, random enough to be new.
Three ways to choose
Same prompt, same probabilities. Only the choosing rule changes. (Example outputs are illustrative.)
…that cat jumped on the___
top k only
0.18alien
0.12ship
0.08table
0.06roof
0.05ramp
0.04floor
0.04man
0.03chute
0.03couch
0.02hood
3 OF 3 · TOP-K
Random, but only the top k
Keep the k most likely tokens, then roll the dice among them.
…that cat jumped on the alien. It hissed, and the little green man ran straight back up the chute.
…that cat jumped on the ship’s ramp and stared down the green men as if it owned the place.
New and reasonable.
Chatbots sample from the top of the distribution: likely enough to make sense, random enough to be new.
Predict, choose, append, repeat
Each new token goes back in as input. The whole sentence is fed through again every time.
every token so far goes in
your prompt (the spaceship story, last tokens shown)
generated, one token per click
…
·that
·cat
·jumped
·on
·the
·alien
.
·It
·hissed
GPT-2predict a distribution, then choose
Click by click, the chosen token slides out of the model and joins the input. Then the model runs again on the longer text.
This is why a chatbot’s answer streams in word by word: it is literally being written one token at a time.
Language models are autoregressive: their own output becomes their next input.
Activity: you are the LLM
In groups. First turn: whoever’s birthday is soonest after today. Last: whoever’s was yesterday.
1Predict. Write 3 possible next tokens (a word or punctuation) for the text so far.
2Sample. Search “random number generator” and pick a number from 1 to 3.
3Append. Add that token to the phrase and pass it to the next person.
4Stop at 25 new tokens or 7 minutes, whichever comes first.
STARTING PROMPT
The spaceship landed with a thud.
7
minutes
25
tokens
Best and strangest stories get read aloud.
Three candidates and a dice roll: you just ran a simple top-k sampler, with k = 3.
It only took 76 years
Remember the spaceship? Here is a modern model filling in the same blank.
1948 Shannon’s N-grams
word2vec 2013 · attention 2017 · GPT-2 2019 · GPT-4o 2024
almost all of it in the last 11 years
J
Complete this phrase with a single word.

The spaceship landed with a thud. The green men came down the shoot and before I knew it that cat jumped on the
AI
alien.
Completed with GPT-4o
From fill-in-the-blank to one word that uses a whole paragraph of context.
How did chatbots get so good?
Same next-token idea as GPT-2. Three things changed.
1
Volume
The internet holds tons of text, and many books are in the public domain. Predicting the next token on trillions of tokens teaches grammar, facts and style.
2
Quality
Carefully written text, like published books, teaches far more per token than random web pages. Much of it is copyrighted.
3
Feedback
Reinforcement learning: people (and other models) rate answers, and the model shifts its distribution toward helpful, honest replies.
Point 2 raises a hard question: is it legal to train on copyrighted books?
Is training on copyrighted books legal?
Two U.S. rulings from the Northern District of California, days apart in June 2025
Bartz v. Anthropic
Fair use: training Claude on books the company bought and scanned was “transformative” (Judge William Alsup).
Not fair use: keeping pirated copies of books. That part led to a $1.5 billion settlement, about $3,000 per book, approved in July 2026.
Kadrey v. Meta
Fair use, this time: training Llama on the authors’ books was found transformative (Judge Vince Chhabria).
But: the authors lost because they did not show specific evidence of harm to the market for their books. The judge said other cases could come out differently.
So far: the material must be legally acquired, and the model must not reproduce the original.
The law here is still being written, by courts and by the people who build these systems.
Why study AI now?
Lots of people can use a chatbot. Far fewer can build, test and fix one.
$140,300
median pay, computer and information research scientists (May 2025)
+22%
projected job growth, 2025–2035: “much faster than average,” with AI a key driver
Source: U.S. Bureau of Labor Statistics, Occupational Outlook Handbook, Computer and Information Research Scientists
AI is also reshaping manufacturing, energy, healthcare, defense and research, so the demand is not only at tech companies.
The jobs go to people who understand how it works under the hood.
The B.S. in Artificial Intelligence
Department of Computer Science, Tennessee Tech
1stbachelor’s in AI in Tennessee
Fall 2026launched; state-approved Oct 2025
≈ 30students already in the major
CS rootsgrew from the Data Science & AI concentration
What you get from the degree
1
A common base with computer science
CSC 1300, 1310, 2310, 2400, 2700 
2
Data science, in theory and in practice
Intro to DS & AI · Fundamentals of DS · Advanced DS & Apps · Statistics · Databases
3
Model development, in theory and in practice
Intro to AI & ML · Machine Learning · Deep Learning · Human-centered AI
4
Room for a minor: AI doesn’t exist in a vacuum
14 free elective hours for supporting tech or an application area
5
Computing, HPC and deploying AI
Tech’s HPC cluster in Deep Learning and other courses · Parallel Programming · DevOps
6
Built for transfer students
Finish in 60 hours after the CS Tennessee Transfer Pathway A.S.
What you’ll leave with
Programming and software development
Building powerful, useful AI models
Analyzing large, complex data to answer important questions
Breadth in an area of your choosing
Building and deploying large AI applications
A strong foundation for research at the frontier of AI
Everything in today’s talk, from embeddings to sampling, is what this degree teaches you to build.
Four years at a glance
Degree map · Catalog year 2026–2027 · BS, Artificial Intelligence · 120 credit hours
FIRST YEAR
FallCr.
CSC 1020Connections to Computing¹1
CSC 1300Intro to Programming4
MATH 1910Calculus I4
ENGL 1010Writing Composition I3
HIST 2010Early US History3
Total credit hours15
SpringCr.
CSC 1310Data Structures & Alg.4
MATH 2010Intro to Linear Algebra3
ENGL 1020Writing Composition II3
HIST 2020Modern US History3
Soc./Behav. Sci. Elective3
Total credit hours16
AI and data science courses free electives: 14 hoursMath: Calculus I, Linear Algebra, Statistics (no Calculus II)
¹ Not required for transfer students with more than 12 hours (they take a 1-hour free elective instead). ² Natural science: BIOL 1113 + 1123 or 2310; CHEM 1110 + 1120; GEOL 1040 + 1045; or PHYS 2010/2110 + 2020/2120. ³ 2000-level or higher.
Four years at a glance
Degree map · Catalog year 2026–2027 · BS, Artificial Intelligence · 120 credit hours
FIRST YEAR
FallCr.
CSC 1020Connections to Computing¹1
CSC 1300Intro to Programming4
MATH 1910Calculus I4
ENGL 1010Writing Composition I3
HIST 2010Early US History3
Total credit hours15
SpringCr.
CSC 1310Data Structures & Alg.4
MATH 2010Intro to Linear Algebra3
ENGL 1020Writing Composition II3
HIST 2020Modern US History3
Soc./Behav. Sci. Elective3
Total credit hours16
SOPHOMORE YEAR
FallCr.
CSC 2310Object-Oriented Prog.4
CSC 2510Intro to DevOps with Unix3
CSC 2700Discrete Structures3
Humanities/Fine Arts Elect.3
COMM 2025Comm. (or PC 2500)3
Total credit hours16
SpringCr.
CSC 2220Intro to DS and AI3
CSC 2400Design of Algorithms3
Natural Science²4
MATH 3070Statistics (or MATH 3470)3
Elective3
Total credit hours16
AI and data science courses free electives: 14 hoursMath: Calculus I, Linear Algebra, Statistics (no Calculus II)
¹ Not required for transfer students with more than 12 hours (they take a 1-hour free elective instead). ² Natural science: BIOL 1113 + 1123 or 2310; CHEM 1110 + 1120; GEOL 1040 + 1045; or PHYS 2010/2110 + 2020/2120. ³ 2000-level or higher.
Four years at a glance
Degree map · Catalog year 2026–2027 · BS, Artificial Intelligence · 120 credit hours
FIRST YEAR
FallCr.
CSC 1020Connections to Computing¹1
CSC 1300Intro to Programming4
MATH 1910Calculus I4
ENGL 1010Writing Composition I3
HIST 2010Early US History3
Total credit hours15
SpringCr.
CSC 1310Data Structures & Alg.4
MATH 2010Intro to Linear Algebra3
ENGL 1020Writing Composition II3
HIST 2020Modern US History3
Soc./Behav. Sci. Elective3
Total credit hours16
SOPHOMORE YEAR
FallCr.
CSC 2310Object-Oriented Prog.4
CSC 2510Intro to DevOps with Unix3
CSC 2700Discrete Structures3
Humanities/Fine Arts Elect.3
COMM 2025Comm. (or PC 2500)3
Total credit hours16
SpringCr.
CSC 2220Intro to DS and AI3
CSC 2400Design of Algorithms3
Natural Science²4
MATH 3070Statistics (or MATH 3470)3
Elective3
Total credit hours16
JUNIOR YEAR
FallCr.
AI 3000Intro to AI & ML3
CSC 3040Prof., Comm. & Research3
CSC 3220Fund. of Data Science3
CSC 3300Database Mgmt. Systems3
Elective3
Total credit hours15
SpringCr.
AI 3100Human-centered AI3
AI 3200Machine Learning3
CSC 4760Parallel Programming3
Natural Science²4
Humanities/Fine Arts Elect.3
Total credit hours16
AI and data science courses free electives: 14 hoursMath: Calculus I, Linear Algebra, Statistics (no Calculus II)
¹ Not required for transfer students with more than 12 hours (they take a 1-hour free elective instead). ² Natural science: BIOL 1113 + 1123 or 2310; CHEM 1110 + 1120; GEOL 1040 + 1045; or PHYS 2010/2110 + 2020/2120. ³ 2000-level or higher.
Four years at a glance
Degree map · Catalog year 2026–2027 · BS, Artificial Intelligence · 120 credit hours
FIRST YEAR
FallCr.
CSC 1020Connections to Computing¹1
CSC 1300Intro to Programming4
MATH 1910Calculus I4
ENGL 1010Writing Composition I3
HIST 2010Early US History3
Total credit hours15
SpringCr.
CSC 1310Data Structures & Alg.4
MATH 2010Intro to Linear Algebra3
ENGL 1020Writing Composition II3
HIST 2020Modern US History3
Soc./Behav. Sci. Elective3
Total credit hours16
SOPHOMORE YEAR
FallCr.
CSC 2310Object-Oriented Prog.4
CSC 2510Intro to DevOps with Unix3
CSC 2700Discrete Structures3
Humanities/Fine Arts Elect.3
COMM 2025Comm. (or PC 2500)3
Total credit hours16
SpringCr.
CSC 2220Intro to DS and AI3
CSC 2400Design of Algorithms3
Natural Science²4
MATH 3070Statistics (or MATH 3470)3
Elective3
Total credit hours16
JUNIOR YEAR
FallCr.
AI 3000Intro to AI & ML3
CSC 3040Prof., Comm. & Research3
CSC 3220Fund. of Data Science3
CSC 3300Database Mgmt. Systems3
Elective3
Total credit hours15
SpringCr.
AI 3100Human-centered AI3
AI 3200Machine Learning3
CSC 4760Parallel Programming3
Natural Science²4
Humanities/Fine Arts Elect.3
Total credit hours16
SENIOR YEAR
FallCr.
AI 4200Deep Learning3
CSC 4610Software Engineering I3
Elective2
Soc./Behav. Sci. Elective3
Elective³3
Total credit hours14
SpringCr.
CSC 4200Computer Networks3
CSC 4620Software Engineering II3
CSC 4260Adv. Data Science & Apps3
Upper Division Elective³3
Total credit hours12
AI and data science courses free electives: 14 hoursMath: Calculus I, Linear Algebra, Statistics (no Calculus II)
¹ Not required for transfer students with more than 12 hours (they take a 1-hour free elective instead). ² Natural science: BIOL 1113 + 1123 or 2310; CHEM 1110 + 1120; GEOL 1040 + 1045; or PHYS 2010/2110 + 2020/2120. ³ 2000-level or higher.
AI courses start in year 2, and 14 elective hours leave room to learn a field AI can serve.
Your community college courses, on the map
Computer Science A.S. (Tennessee Transfer Pathway) → BS, Artificial Intelligence · 2026–2027 map
FIRST YEAR
FallCr.
CSC 1020Connections to Computing¹1
CSC 1300Intro to Programming4
MATH 1910Calculus I4
ENGL 1010Writing Composition I3
HIST 2010Early US History3
Total credit hours15
SpringCr.
CSC 1310Data Structures & Alg.4
MATH 2010Intro to Linear Algebra3
ENGL 1020Writing Composition II3
HIST 2020Modern US History3
Soc./Behav. Sci. Elective3
Total credit hours16
SOPHOMORE YEAR
FallCr.
CSC 2310Object-Oriented Prog.4
CSC 2510Intro to DevOps with Unix3
CSC 2700Discrete Structures3
Humanities/Fine Arts Elect.3
COMM 2025Comm. (or PC 2500)3
Total credit hours16
SpringCr.
CSC 2220Intro to DS and AI3
CSC 2400Design of Algorithms3
Natural Science²4
MATH 3070Statistics (or MATH 3470)3
Elective3
Total credit hours16
JUNIOR YEAR
FallCr.
AI 3000Intro to AI & ML3
CSC 3040Prof., Comm. & Research3
CSC 3220Fund. of Data Science3
CSC 3300Database Mgmt. Systems3
Elective3
Total credit hours15
SpringCr.
AI 3100Human-centered AI3
AI 3200Machine Learning3
CSC 4760Parallel Programming3
Natural Science²4
Humanities/Fine Arts Elect.3
Total credit hours16
SENIOR YEAR
FallCr.
AI 4200Deep Learning3
CSC 4610Software Engineering I3
Elective2
Soc./Behav. Sci. Elective3
Elective³3
Total credit hours14
SpringCr.
CSC 4200Computer Networks3
CSC 4620Software Engineering II3
CSC 4260Adv. Data Science & Apps3
Upper Division Elective³3
Total credit hours12
Course numbers follow the pathway’s common numbering; names vary by college. Tech transfers take Linear Algebra as their third pathway math. ¹ CSC 1020 is waived for transfers with 12+ hours (Calculus II’s 4th hour covers the 1-hour elective). ² Lab science sequence. ³ 2000-level or higher.
Your community college courses, on the map
Computer Science A.S. (Tennessee Transfer Pathway) → BS, Artificial Intelligence · 2026–2027 map
FIRST YEAR
FallCr.
Waived for transfers¹1
CISP 1010Computer Science I4
MATH 1910Calculus I4
ENGL 1010English Comp. I3
HISTHistory3
Total credit hours15
SpringCr.
CISP 1020Computer Science II4
MATH 2010Linear Algebra3
ENGL 1020English Comp. II3
HISTHistory3
Social/Behav. Science3
Total credit hours16
SOPHOMORE YEAR
FallCr.
CSC 2310Object-Oriented Prog.4
CSC 2510Intro to DevOps with Unix3
CSC 2700Discrete Structures3
Humanities/Fine Arts3
COMM 2025Communication3
Total credit hours16
SpringCr.
CSC 2220Intro to DS and AI3
CSC 2400Design of Algorithms3
Lab science I4
MATH 3070Statistics (or MATH 3470)3
MATH 1920Calculus II → elective3
Total credit hours16
JUNIOR YEAR
FallCr.
AI 3000Intro to AI & ML3
CSC 3040Prof., Comm. & Research3
CSC 3220Fund. of Data Science3
CSC 3300Database Mgmt. Systems3
CISP 2410Assembly & Comp. Org.3
Total credit hours15
SpringCr.
AI 3100Human-centered AI3
AI 3200Machine Learning3
CSC 4760Parallel Programming3
Lab science II4
Humanities/Fine Arts3
Total credit hours16
SENIOR YEAR
FallCr.
AI 4200Deep Learning3
CSC 4610Software Engineering I3
Elective2
Social/Behav. Science3
Literature3
Total credit hours14
SpringCr.
CSC 4200Computer Networks3
CSC 4620Software Engineering II3
CSC 4260Adv. Data Science & Apps3
Upper Division Elective³3
Total credit hours12
taken at community college: 60 hours Calculus II isn’t required for the AI degree, so it comes in as an electiveeverything else: the 60 hours you take at Tech
Every pathway course has a home on the map: half the degree is done before you arrive.
Course numbers follow the pathway’s common numbering; names vary by college. Tech transfers take Linear Algebra as their third pathway math. ¹ CSC 1020 is waived for transfers with 12+ hours (Calculus II’s 4th hour covers the 1-hour elective). ² Lab science sequence. ³ 2000-level or higher.
Four semesters at Tech
A sample schedule for the 60 hours left after the CS transfer pathway, in degree-map order
YEAR 1 AT TECH
FallCr.
CSC 2310Object-Oriented Programming4
CSC 2510Intro to DevOps with Unix3
CSC 2700Discrete Structures3
CSC 2220Intro to Data Science and AI3
MATH 3070Statistics (or MATH 3470)3
Total credit hours16
SpringCr.
CSC 2400Design of Algorithms3
AI 3000Intro to AI & Machine Learning3
CSC 3220Fundamentals of Data Science3
CSC 3300Database Management Systems3
CSC 3040Prof., Communication & Research3
Total credit hours15
YEAR 2 AT TECH
FallCr.
AI 3100Human-centered AI3
AI 3200Machine Learning3
CSC 4760Parallel Programming3
CSC 4610Software Engineering I3
Elective2
Total credit hours14
SpringCr.
AI 4200Deep Learning3
CSC 4260Advanced Data Science & Apps3
CSC 4620Software Engineering II3
CSC 4200Computer Networks3
Upper-division elective: your application area3
Total credit hours15
AI and data science your application-area electiveAbout 15 hours a semester, 60 in all. Sample only: course timing and prerequisites are confirmed with your advisor.
Arrive with the pathway done, and four semesters later you have a B.S. in AI.
Fast track to a master’s
Start the M.S. in Computer Science while you finish your B.S. in AI
YEARS 1–2Community college · CS transfer pathway
YEARS 3–4Tennessee Tech · B.S. in AI
YEAR 5M.S. in CS
Up to 6 graduate hours
count toward both degrees
WHO QUALIFIES
Junior or senior standing
CSC 2400 (Algorithms) done
3.25 overall GPA
3.5 GPA in CSC courses
WHAT YOU GET
Graduate courses in your junior or senior year that count toward the B.S. and the M.S.
Both degrees in about five years: B.S. in the spring, M.S. by the end of the following summer.
THE FINE PRINT
Earn a B or better in each graduate course.
Courses must be taken at Tech and approved as substitutions in the B.S.
Talk to the CS department early: meeting the minimums doesn’t guarantee a spot.
B.S. and M.S. in three years at Tech
After the transfer pathway: grad courses and research in your senior year, a thesis M.S. in year three
YEAR 1 AT TECHB.S.
Fall
B.S. courses15
Spring
B.S. courses15
Apply to fast track
YEAR 2 AT TECHB.S. + start the M.S.
Fall
B.S. courses12
Fast-track grad course3
Grad course, undergrad rate3
Research starts
Spring
B.S. courses12
Fast-track grad course3
Grad course, undergrad rate3
Research continues
Graduate: B.S. in AI
YEAR 3 AT TECHM.S. in CS
Fall
Grad courses6
Thesis3
Spring
Grad courses6
Thesis3
Defend thesis: M.S.
The 30-hour M.S. (thesis option): where each hour comes from
6 fast track: count twice
6 taken as an undergrad
12 grad courses
6 thesis
Year 3 is light: 9 hours a semester. Fast track needs junior standing, CSC 2400, a 3.25 GPA (3.5 in CSC) and a B or better in each grad course. Sample plan: confirm with the CS graduate coordinator.
Community college to a master’s in five years: start research early and use the open slots in your senior schedule.
Where it leads
The degree is new, but the path isn’t
Roles AI majors grow into, and what they pay
Machine learning engineer
$135,980BLS: software developers
Data scientist
$120,230BLS: data scientists
AI research scientist
$140,300BLS: computer & info. research scientists
AI infrastructure / MLOps
$134,050BLS: computer network architects
Median annual pay, May 2025, nearest U.S. Bureau of Labor Statistics occupation (no AI titles yet).
Glassdoor, “machine learning engineer” (Oct 2026): typical range $133K–$209K
OUR TRACK RECORD
Tech students who took the AI path before the degree existed now work as AI/ML professionals in defense and other organizations.
ACROSS INDUSTRIES
defensemanufacturingenergyhealthcareresearch labsgraduate school
You would be among the first graduates with “Artificial Intelligence” on the diploma.
Ask the people already doing it: talk to our AI majors and their advisors.
How to join
From the transfer pathway to a B.S. in AI
At community college now?
Follow the Computer Science A.S. Tennessee Transfer Pathway for CS.
Getting ready to transfer?
Apply to Tech as a transfer student and contact the College of Engineering Student Success Center to map your credits onto the AI degree.
Questions?
Dr. Jesse Roberts · jtroberts@tntech.edu
Department of Computer Science, Bruner Hall 238 · (931) 372-3691. Or catch me after the talk.
The AI major page
tntech.edu/majors/
artificial-intelligence.php
Talk to us before your last community college semester so every hour counts.
What comes next?
Back to the top of the tree
The next branch hasn’t been named yet.
2020s · Large language & vision-language-action models
2015+ · Deep reinforcement learning
2015+ · Spiking neural networks
2010s · Deep learning
Data & HPC systems
Maybe you name it. Questions?
Keep the handout: the whole tree, seventy years of AI on one page.