HON 4013 · AI, FINANCE, AND HUMAN FUTURES

Reinforcement learning, predictions about you, and the ads that pay for it

Dr. Jesse Roberts · Tennessee Technological University · Fall 2026

Learning from rewards

Goal: find the best move in every square, the one that leads to the most total reward

action

new state, reward

Agent: the learner · the black dot

State: where it is · its square

Action: what it does · up, right, down, left

Reward: the score it gets · −1 a move, +10, −10

Policy: its rule for acting · the arrows

Arrow = a possible move · number = its estimated total reward · thick = current best move

Learned: the best move in every square

4.3

−2.0

6.2

1.4

8.0

4.6

4.4

10

−9.8

6.0

1.0

−3.2

6.2

−10

2.7

9.4

1.6

−7.5

−2.3

−3.6

4.6

1.7

6.4

−0.6

−3.5

−1.4

−4.4

1.8

−2.5

−3.6

3.1

−1.2

0.0

3.4

−2.0

−4.1

−0.4

0

0.6

−2.2

1.8

−2.9

−3.0

−1.7

−3.3

0.4

−2.6

1×

Nobody tells the agent the route: it finds one from rewards alone.

Our Q-learning run: 100 episodes, learning rate 0.5, discount 0.9, random moves falling from 100% to 10% (seed 32). Every episode ends at the goal or the pit. The notebook reproduces every frame.

Explore or exploit?

?

Same three machines, same luck. The only difference is the strategy

71 wins total

won 71 of 198

win rate 36%

won 0 of 1

win rate 0%

won 0 of 1

win rate 0%

120 wins total

won 14 of 43

win rate 33%

won 1 of 7

win rate 14%

won 105 of 150

win rate 70%

200 pulls done

Multi-armed bandit: named for slot machines

Contextual bandit: the best arm depends on who pulls

A few pulls can’t tell you which machine is best, so a little exploring pays.

Our simulation: each machine’s run of wins and losses is fixed, so both players get the same luck (seed 13). Explorer: ε-greedy, ε = 0.1. Averages: 2,000 independent games of 200 pulls (seed 7).

Rewards that arrive later

Most moves pay nothing right away. Their value comes from the rewards that follow

Value along the route, walking back from the goal

blue outline = learned route · grey = off-route values

A good move now can pay off much later, so RL learns the value of a state, not just its reward.

Values from our Q-learning run (γ = 0.9, 100 episodes). The start’s −0.4 is the exact value of the 8-step route.

You’re the company

?

A free app full of posts. Advertisers pay. Where does the money come from?

Revenue =

×

×

×

Attention: what users give you Ads: what you sell

Meta Platforms Q3 2025 earnings call (Oct 29, 2025): family of apps ad revenue $50.1B; ad impressions +14%; average price per ad +10%.

What the business needs

Revenue = users × minutes × ads per minute × price per ad. What feeds each term?

×

×

×

Each need has to become a number before a machine can optimize it.

The feed is an RL agent

Same loop as the grid world. Click to swap in the feed

action

new state

reward

The feed learns whatever the reward measures, so choosing the reward is choosing the product.

The funnel: millions to ten

Each request, the feed narrows everything it could show down to what fits on your screen

X, 2023 code: ~1,500 candidates per request · heavy ranker ≈ 48M parameters

Cheap models filter, expensive models decide, so each request stays fast.

Counts per stage vary by platform and are typical orders of magnitude. X figures: Twitter engineering blog and the twitter/the-algorithm repositories, March 2023.

One net, many guesses

For each candidate post, one network predicts the chance of everything you might do with it

one post, one viewer (illustrative values)

Heads: Meta’s published Reels list · each trained on what viewers actually did

The model never decides; it predicts a probability for each thing you might do.

Prediction list: Meta Transparency Center, “Instagram Reels Chaining AI system” (predictions include watching under 3 s, watching more than 95% of viewers, liking, commenting, resharing, tapping the audio). Probabilities shown are illustrative.

The post scorecard

?

Many guesses become one number, the way insurers score drivers

Insurance scorecard

points for facts about you (illustrative)

Facts: checkable, disputable

X’s feed score, 2023

score = Σ weight × predicted probability

Predictions × chosen weights

Same shape as an insurance scorecard, but every line is a prediction. There are side effects...

X weights: twitter/the-algorithm-ml, projects/home/recap README (heavy ranker, as published April 2023); X says weights can change at any time. Insurance points are illustrative.

The ad side: an auction every request

Advertisers bid for your attention, and the same kind of prediction model decides who wins

Lowest bid wins the highest expected payout: you are predicted 3× likelier to click, 2× likelier to buy

Then: one slot, two contenders

vs

your next slot

who wins: the ad-load dial, next

The better the feed predicts you, the more each ad slot sells for: that is why your data is the asset.

Illustrative bids and probabilities. Real auctions add quality and relevance terms, but rank ads by expected value per impression in the same way.

The ad-load dial is a bandit

Same slot machine, new arms: 0, 1 or 2 ads in the next load, chosen per person, per moment

ShareChat A/B test · 80M users · 200M sessions: satisfaction ↑ and ad revenue ↑

How many ads you see is learned for you, from how people like you reacted.

Sagtani, Jhawar, Mehrotra and Jeunen, “Ad-load Balancing via Off-policy Learning in a Content Marketplace,” WSDM 2024 (ShareChat). The two users are illustrative.

Optimizing the whole session

Two rankers on the same toy user. P = provocative post

Once the reward spans the session, the feed optimizes the sequence, not just the next post.

Our toy feed, averages over 20,000 simulated sessions. Real systems: YouTube’s RL recommender (Chen et al., WSDM 2019); Meta credits recommendations with +5% Facebook time spent (Q3 2025 call).

Always learning

The model and the product both keep changing, all day

The model retrains as you scroll

The company tests itself

v1,049

20,000+

ByteDance Monolith: new model about every minute (1 s here)

Google, LinkedIn, Microsoft: 20,000+ tests a year each · blue = shipped

The model updates in minutes, the product by experiment: two learning loops, one target.

Liu et al., “Monolith: Real Time Recommendation System With Collisionless Embedding Table,” 2022 (ByteDance). Kohavi, Tang, Xu et al., “Online randomized controlled experiments at scale,” Trials, 2020.

One swipe, about 200 milliseconds

Everything so far runs between your swipe and the next post

200 ms

~10 posts to your screen

All of it finishes before your thumb does, billions of times a day.

Stage times are an illustrative budget, not measurements from one platform. 1,500 candidates: X’s 2023 open-source timeline code.

The loop closes

Each swipe feeds the next one

reward

learning

new state

action

blue labels = the RL loop from the start, with you as the world

Nobody coded the hook

Three ordinary choices add up to a slot machine

uncertain predictor

exploration

greedy ranking

7

7

7

X, 2023: every reply scores

+13.5

+13.5

Arousal wins the score

The hook falls out of the optimization; no one has to design it.

Feed ratings illustrative. Reply weight 13.5: twitter/the-algorithm-ml recap README (2023); the scoring has no tone or sentiment term. Variable-ratio rewards: a standard result in behavioural psychology.

No natural end

Every request is a fresh decision

One toy-feed session: the user’s hidden hook level (0–5) and what the feed showed · P = provocative post · ads land at the peaks

The action list, at every request

“stop” is not an action

Where it ends

Nothing in the action list says “stop”, so the policy never learns to.

Run the feed

?

Toy feed, simulated user, Q-learning. Reward: 0.1 per post, 1 per ad. Nothing else

What it learned to show, by hook level

Revenue per session while it learns

Our simulation (notebook): hook level 0–5; we assumed provocative posts raise it more often than calm ones (60% vs 40%) and lower it less (10% vs 20%), an ad lowers it by 2, and users leave more often at low levels. Best policy by value iteration; Q-learning seed 11.