Stop treating the model as a black box: 100 challenges take you from a single gradient to a fine-tuned model you can query.
You can call an API and get text back, but you cannot say what happens between the prompt and the reply — or why the model stalls, rambles, or costs what it costs. This course closes that gap by making you build the thing. You start with scalars and hand-written gradients in Python, Go, and Rust, then an autograd engine, a byte-pair encoding (BPE) tokenizer, and a counting bigram model. From there you write scaled dot-product attention yourself, assemble a decoder block, and train a roughly 10-million-parameter generative pre-trained transformer (GPT) on TinyShakespeare to a measured validation loss. The last third is the part most courses skip: memory math, gradient accumulation, distributed data parallel training across graphics processing units (GPUs), ZeRO/FSDP sharding, FlashAttention, and key-value caching; then supervised fine-tuning, low-rank adaptation (LoRA), and direct preference optimization (DPO); then int4 quantization and a streaming chat endpoint with moderation and logs. Every module ships runnable code and a project. The capstone is a small but real large language model you trained, fine-tuned, and deployed.
Built by Lakshya Kumar
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
From raw text to integer IDs to a first working language model. Character-level and BPE tokenizers, a counting bigram, its neural equivalent, sampling, and perplexity.
A single layer cannot separate two crossed classes; you hand-derive every gradient in a network that can, with no framework hiding it.
Swap sparse one-hot vectors for learned embeddings, then tune temperature, top-k, and top-p until the same model stops sounding robotic.
Write query, key, and value projections yourself, mask the future, and see exactly why doubling the context quadruples the cost.
Residuals, normalization, and the feedforward network are what keep a deep stack trainable — build the block that holds them together.
Warmup, cosine decay, gradient clipping, and bf16 turn a run that diverges at step 200 into one that finishes and generates text.
Do the memory math, shard the optimizer state, cache keys and values, and land a run that used to crash on hardware you can rent.
A base model completes text; it does not answer. Fine-tune one until it does, and evaluate it without fooling yourself.
Quantize to int4, stream tokens as they generate, add moderation and logs — the gap between a checkpoint and something people use.
Complete all modules, then submit the required number of capstone projects. Each must earn a passing rating from an admin reviewer.
End-to-end: BPE tokenizer on a dataset you picked, a 10–50M-param GPT trained to a measurable val loss, instruction fine-tune with LoRA, DPO preference tune, int4 quantize, deploy behind a streaming chat API with moderation and structured logs. Ship as a repo with README, measurements, sample outputs, and 5 concrete failure modes you discovered.
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
I'm considering a "Build a Large Language Model From Scratch: Tensors to Chat API" course. It builds up from tensors/autograd, to neural nets, to tokenization, to attention, to a full Transformer, to training GPT on TinyShakespeare, to training-at-scale tricks (FlashAttention, ZeRO, mixed precision), to SFT + LoRA + DPO fine-tuning, to int4 quantization and deployment behind a streaming chat API. 100 challenges total. Python throughout, with Go + Rust + Node on the engineering/deployment modules via code tabs. Context about me: 1. My current role/focus: [e.g. "backend dev who's curious about ML", "data scientist who only ever calls model.fit", "undergrad who's already watched Karpathy's videos once"] 2. The deepest I've gone into ML so far: [e.g. "nothing", "sklearn + XGBoost", "trained a CNN in PyTorch once", "fine-tuned Llama with a HuggingFace trainer"] 3. What I'm hoping this course changes about me: [e.g. "I can read an LLM paper and implement it", "I can deploy my own fine-tuned model at work", "I can start my own AI startup"] Answer these: - For my background, which 2 modules will give me the highest leverage in the next 3 months, and why? - Name a concrete artifact I'd build during the course that I could actually use on my resume or at work. - Is 60 hours worth it for me, or should I do something shorter first (just Karpathy, a course on ML basics, etc.)? Give your honest pick. - What should I explicitly NOT expect — e.g. "you will not train a 70B model", "you will not beat GPT-4 at anything", "you will not learn RAG"?
Implement scaled dot-product attention, multi-head attention, and a full transformer block from scratch (no torch.nn.MultiheadAttention). Compare correctness against the reference impl on synthetic data. Profile memory and FLOPs at sequence lengths 128, 512, 2048.
Implement BPE tokenizer training and encoding from scratch. Train on 100MB of text; produce a 32k-vocab tokenizer. Encode/decode round-trip 10MB of text and compare compression ratio + speed against tiktoken. Document the byte-level fallback.
Take your trained base model and instruction-tune it on a small instruction dataset (Alpaca-style, 5k examples). Evaluate the instruct model on held-out instructions: response quality (manual scoring) + structured task success (JSON output, classification). Compare to base model.
Train 4 sizes of your model (1M, 10M, 100M, ~500M params) on the same data budget. Plot loss vs parameter count and vs FLOPs. Identify your local Chinchilla-optimal point. Compare findings to the canonical scaling-laws papers and explain deviations.
The book version of this course. Read a chapter when a module doesn't fully land.