Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Most people use LLM APIs for months without knowing what physically happens when they hit 'generate'. But every serving decision you'll make — batching, caching, GPU sizing, latency budgets — flows from one fact: generation is an autoregressive loop that runs the model once per output token. Understanding that the model produces text one token at a time, re-reading everything so far each step, is the mental model the rest of this course builds on. Without it, serving optimizations look like magic incantations; with it, they're obvious consequences.
The demo strips generation to its essence: a loop that feeds the current sequence to the model, gets a probability distribution over the next token, picks one, appends it, and repeats until a stop token. This is what an API call is doing behind the scenes.
# The autoregressive loop, conceptually (real serving does this on a GPU, batched):
def generate(model, tokenizer, prompt, max_new=50):
ids = tokenizer.encode(prompt)
for _ in range(max_new):
logits = model.forward(ids) # run the WHOLE model over the sequence
next_logits = logits[-1] # we only care about the last position
next_id = int(next_logits.argmax()) # greedy: pick the most likely token
ids.append(next_id) # append and loop again
if next_id == tokenizer.eos_id:
break
return tokenizer.decode(ids)
# Key insight: one model.forward() per OUTPUT token. A 500-token answer = 500 forward passes.python3 main.py