LLM From Scratch

LLM From Scratch

Understand and build the main parts of an LLM stack, then learn the production tools used for real open models. Nineteen phases, in order, from PyTorch tensors to a served model you trained yourself.

19phases
227 to 360focused hours
$0 to $10compute per phase
7 weeksat 30 to 35 hours
Aug 2026last updated

How to use this. Do the phases in order. Each lists what to learn, what to build, which resources to pull from, and a concrete test for whether you are done. No resource is meant to be finished end to end; every one has a single job, listed in section 2. Search with Ctrl K, move between sections with the arrow keys, switch to full scroll in the sidebar when you want to search the page or print it. Progress is tracked per phase and saved in this browser only, so download a backup now and then. The dashboard tells you when your backup has gone stale.

Start at section 1
01

End goal#

By the end of this roadmap you should be able to:

  • work comfortably with PyTorch tensors, shapes, broadcasting, einsum, and einops
  • explain and implement reverse-mode autodiff and backpropagation
  • implement a BPE tokenizer
  • implement and train a decoder-only transformer from scratch
  • explain Q, K, V, causal masking, multi-head attention, residuals, normalization, and feed-forward layers
  • implement RoPE, RMSNorm, SwiGLU, GQA, weight tying, and KV caching
  • write a pretraining loop with validation, checkpointing, gradient accumulation, mixed precision, and learning-rate schedules
  • build a pretraining data pipeline with filtering, deduplication, tokenization, sharding, and sequence packing
  • run controlled training experiments on roughly 10M–100M parameter models
  • fine-tune a real open model with LoRA and QLoRA
  • perform supervised fine-tuning and DPO
  • evaluate models with held-out loss, public benchmarks, and a private task-specific eval set
  • quantize and run a model locally
  • serve a model through llama.cpp or vLLM
  • explain DDP, FSDP, ZeRO, tensor parallelism, pipeline parallelism, and the communication bottlenecks behind distributed training
  • run an end-to-end pipeline such as nanochat
  • read modern LLM architecture papers and understand their main design choices
02

Resource roles#

Do not complete every resource end to end. Use each one for a specific purpose.

Resource Use it for
Karpathy Zero to Hero Primary first-principles teaching
Sebastian Raschka, LLMs From Scratch Written reference and additional implementations
ARENA Tensor, optimization, backprop, and evaluation exercises
Stanford CS336 Advanced training, systems, data, scaling, and distributed training
AI Engineering From Scratch Supplementary exercises and breadth reference
PyTorch / Hugging Face docs Production APIs after the underlying mechanism is clear
nanochat End-to-end capstone
03

Exact order#

Five stages, twenty steps. Each one assumes the one before it.

Foundations

No GPU needed
  1. 01Environment and PyTorch setup
  2. 02Tensor fluency
  3. 03Backpropagation and neural-network fundamentals
  4. 04Small language models before transformers
  5. 05Tokenization

The model

Free GPU
  1. 06Vanilla GPT from scratch
  2. 07Modern decoder architecture
  3. 08KV cache and inference

Training at scale

Cheap GPU
  1. 09Training engineering
  2. 10Pretraining data pipeline
  3. 11Scaling laws and resource accounting
  4. 12Distributed training fundamentals

Real models

16 to 48 GB
  1. 13Fine-tune a real open model
  2. 14Chat templates and instruction data
  3. 15Evaluation
  4. 16DPO and preference optimization

Shipping

Optional GPU
  1. 17Quantization
  2. 18Serving
  3. 19nanochat capstone
  4. 20Optional specialization
Phase 0

Environment and workflow#

Time4–8 hours
GPUNone
Cost$0

Learn#

  • shell navigation
  • paths, pipes, redirection
  • environment variables
  • processes and exit codes
  • Git basics
  • Python virtual environments
  • uv
  • SSH
  • scp
  • rsync
  • tmux
  • VS Code Remote SSH
  • nvidia-smi
  • basic CUDA/PyTorch compatibility

Build#

Create:

llm-from-scratch/
├── README.md
├── pyproject.toml
├── src/
├── experiments/
├── notes/
└── tests/

Create src/gpu_check.py that prints:

  • Python version
  • PyTorch version
  • CUDA availability
  • GPU name
  • total GPU memory
  • allocated GPU memory

Resources#

Required#

Optional#

Use only the setup lessons you actually need.

Done when#

You can:

  • SSH into a remote GPU machine
  • clone the repo
  • install the environment
  • start tmux
  • run a Python process
  • disconnect and reconnect
  • confirm the process is still running
  • push changes to GitHub
Phase 1

Tensor fluency#

Time12–18 hours
GPUNone
Cost$0

Learn#

NumPy and PyTorch#

  • arrays and tensors
  • shape
  • rank
  • dtype
  • indexing
  • slicing
  • masking
  • broadcasting
  • reductions
  • matrix multiplication
  • batching
  • vectorization
  • views vs copies
  • device placement

PyTorch operations#

  • .reshape()
  • .view()
  • .transpose()
  • .permute()
  • .contiguous()
  • .to()
  • torch.matmul
  • torch.einsum
  • gather

einops#

  • rearrange
  • repeat
  • reduce

Shape notation#

B = batch size
T = sequence length
C = model dimension
H = number of heads
D = head dimension
V = vocabulary size

Track shapes such as:

(B, T, C)
→ (B, T, H, D)
→ (B, H, T, D)

Math required here#

  • vectors
  • matrices
  • tensors
  • dot product
  • matrix multiplication
  • transpose
  • norms
  • derivative
  • partial derivative
  • gradient
  • chain rule

Build#

Create exercises for:

  • broadcasting
  • masks
  • batching
  • matrix multiplication
  • reshape/transpose/permute
  • einsum
  • attention-style shapes

Then implement a small attention-score calculation using random tensors.

Resources#

Required#

Math support#

Supplementary#

Useful lessons:

  • Linear Algebra Intuition
  • Vectors, Matrices & Operations
  • Calculus for ML
  • Chain Rule & Automatic Differentiation
  • Information Theory
  • Tensor Operations
  • Numerical Stability
  • Norms & Distances

Skip for now:

  • PCA
  • t-SNE
  • UMAP
  • complex numbers
  • Fourier transform
  • graph theory
  • stochastic processes
  • convex optimization

Done when#

Given:

x.shape == (8, 128, 12, 64)

you can determine the shape after:

x.transpose(1, 2)

without running it.

Phase 2

Backpropagation and neural-network fundamentals#

Time15–22 hours
GPUNone
Cost$0

Learn#

  • weights and biases
  • linear layers
  • nonlinear activations
  • forward pass
  • computational graphs
  • loss functions
  • reverse-mode autodiff
  • gradient accumulation
  • SGD
  • momentum
  • Adam
  • AdamW
  • weight decay
  • train/validation split
  • overfitting
  • initialization
  • gradient clipping

Activations#

  • ReLU
  • GELU
  • SiLU

Losses#

  • MSE
  • cross entropy

Build#

A. Scalar autodiff engine#

Implement:

class Value:
    data
    grad
    _prev
    _op
    backward()

Support:

  • addition
  • multiplication
  • powers
  • ReLU or tanh
  • topological sorting
  • backward traversal

B. Neural-network classes#

Implement:

  • Neuron
  • Layer
  • MLP

Train on a small classification dataset.

C. PyTorch comparison#

Rebuild the same network in PyTorch and compare:

  • forward values
  • loss
  • gradients

Resources#

Required#

Supporting#

Supplementary#

Useful lessons:

  • Backpropagation from Scratch
  • Activation Functions
  • Loss Functions
  • Optimizers
  • Weight Initialization
  • Learning Rate Schedules
  • Debugging Neural Networks

Done when#

You can explain:

  • reverse-mode autodiff
  • gradient accumulation
  • SGD vs Adam
  • Adam vs AdamW
  • why nonlinear activations are required
Phase 3

Small language models before transformers#

Time12–18 hours
GPUCPU or free GPU
Cost$0

Learn#

A language model estimates:

P(next token | previous tokens)

Understand:

  • autoregressive modeling
  • context
  • logits
  • softmax
  • categorical sampling
  • negative log likelihood
  • cross entropy
  • perplexity
textraw stringtoken IDsintsembeddings(B, T, C)hidden(B, T, C)logits(B, T, V)softmaxprobssampleone ID
Every language model, from a bigram table to a 70B decoder, is this pipeline. Only the hidden step changes.

Build#

Implement and train:

  1. character bigram model
  2. count-based model
  3. neural bigram model
  4. MLP language model

Record train loss, validation loss, and generated samples.

Resources#

Required#

Supplementary#

Done when#

You can explain:

text
→ token IDs
→ embeddings
→ hidden representation
→ logits
→ softmax
→ next-token distribution
→ sampled token
Phase 4

Tokenization#

Time8–12 hours
GPUNone
Cost$0

Learn#

  • characters vs words vs subwords
  • BPE
  • vocabulary size
  • merge rules
  • byte-level tokenization
  • UTF-8
  • Unicode
  • byte fallback
  • special tokens
  • BOS
  • EOS
  • PAD
  • UNK
  • chat role tokens
  • multilingual tokenization

Understand:

larger vocabulary
→ larger embedding/output matrices
→ often fewer tokens per document

Build#

Implement:

train(text)
encode(text)
decode(ids)

Train on a few MB of text.

Test:

  • English
  • numbers
  • code
  • emoji
  • accented text
  • one non-Latin language
  • whitespace edge cases
  • special tokens

Train two vocabulary sizes and compare:

  • total tokens
  • compression ratio
  • average sequence length

Resources#

Required#

Supporting#

Supplementary#

Done when#

You can explain:

  • why hello and hello may tokenize differently
  • why languages have different token fertility
  • how vocabulary size affects compute and sequence length
Phase 5

Vanilla GPT from scratch#

Time20–30 hours
GPUFree GPU
Cost$0

Learn#

Embeddings#

  • token embeddings
  • positional embeddings

Attention#

  • Q
  • K
  • V
  • dot-product attention
  • scaling by sqrt(d_k)
  • causal masking
  • softmax
  • weighted value aggregation

Multi-head attention#

Track:

(B, T, C)
→ Q/K/V
→ (B, H, T, D)
→ attention
→ concatenate heads
→ (B, T, C)

Transformer block#

  • pre-norm
  • residual connections
  • attention
  • feed-forward network
  • final normalization
  • vocabulary projection
input(B, T, C) Q, K, V projections3 x (B, T, C) split heads(B, H, T, D) scores = Q · Kᵀ / √D (B, H, T, T) causal mask softmax over keys weighted sum of V merge heads(B, T, C) output proj(B, T, C) B batch, T sequence, C model dim, H heads, D head dim, C = H x D
Multi-head attention as shape bookkeeping. If you can recite this chain you can debug most attention bugs.
token embeddings transformer block, repeated N times norm causal self-attention + norm feed-forward, SwiGLU + the vertical line is the residual stream: never normalized, only added to final norm, then projection to (B, T, V) Weight tying makes that last projection reuse the token embedding matrix.
The decoder-only stack, drawn as a residual stream with two branches per block.

Build#

Implement:

TokenEmbedding
PositionalEmbedding
CausalSelfAttention
MultiHeadAttention
MLP
TransformerBlock
GPT
generate()

Use PyTorch but not Hugging Face model classes.

Train roughly:

10M–30M parameters

Start with Tiny Shakespeare or another small corpus.

Required tests#

  • causal mask test
  • tensor-shape assertions
  • overfit a tiny batch
  • checkpoint/save
  • checkpoint/load
  • resume training

Resources#

Required#

Papers and explanations#

Supplementary exercises#

Useful lessons:

  • Self-Attention from Scratch
  • Multi-Head Attention
  • Positional Encoding
  • GPT Causal Language Modeling
  • Build a Transformer from Scratch

Do not use as primary training reference#

AI Engineering From Scratch, Pre-Training a Mini GPT (124M) has an incomplete demonstration backward pass: the supplied training code does not update the attention Q/K/V/output projection matrices. Use it as an architecture walkthrough, not as the canonical GPT pretraining implementation.

Done when#

You can draw the decoder-only forward pass and state the major tensor shapes.

Phase 6

Modern decoder architecture#

Time18–28 hours
GPUFree or cheap GPU
Cost$0–$5

Learn and implement#

RMSNorm#

RoPE#

SwiGLU#

GQA#

Implement and compare:

  • MHA
  • MQA conceptually
  • GQA

Weight tying#

Share token embedding and output projection weights.

PyTorch SDPA#

Compare manual attention with SDPA.

FlashAttention#

Understand why memory I/O, not just FLOPs, dominates standard attention at scale.

Build#

Create src/modern_gpt/ with:

  • RMSNorm
  • RoPE
  • SwiGLU
  • GQA
  • weight tying
  • SDPA path

Compare against the vanilla GPT:

  • parameter count
  • training throughput
  • inference throughput
  • peak VRAM

Resources#

Required#

Reference implementations#

Supplementary#

Useful lessons:

  • Positional Encoding
  • KV Cache, Flash Attention & Inference Optimization
  • Attention Variants

Use its newer architecture walkthroughs after the core implementation works.

Done when#

You can read a Llama/Qwen-style config and explain:

  • hidden size
  • number of attention heads
  • KV heads
  • head dimension
  • intermediate size
  • layer count
  • RoPE settings
  • vocabulary size
Phase 7

KV cache and inference#

Time10–15 hours
GPUFree GPU
Cost$0

Learn#

Generation#

prompt
→ forward pass
→ next-token logits
→ sample
→ append token
→ repeat

Prefill#

Process the prompt in parallel.

Decode#

Generate subsequent tokens one at a time.

KV cache#

Understand:

  • what is stored
  • how memory grows with context length
  • how memory grows with batch size
  • why GQA reduces KV-cache memory

Sampling#

Implement:

  • greedy
  • temperature
  • top-k
  • top-p

Then learn:

  • min-p
  • repetition penalty
prefill: whole prompt in one forward pass tok 1 tok 2 tok 3 tok 4 modelparallel over T KV cache filled4 positions, TTFT ends here decode: one token at a time, reusing the cache last tokenT = 1 modelreads cached K, V sampletemp, top-k, top-p append tokencache grows by 1 Cache memory = 2 x layers x KV heads x head dim x context x batch x bytes. GQA cuts the KV-head term.
Prefill is compute bound, decode is memory bandwidth bound. Serving work is mostly about the second line.

Build#

Add KV caching to your model.

Benchmark:

context length
tokens/sec without cache
tokens/sec with cache
peak memory

Resources#

Required#

Your own GPT implementation

Supplementary#

Use these for conceptual examples of KV cache, continuous batching, prefix caching, and speculative decoding.

Done when#

You can explain:

  • TTFT
  • prefill
  • decode
  • KV-cache memory
  • context-length cost
  • GQA's effect on serving memory
Phase 8

Training engineering#

Time20–30 hours
GPUFree or cheap GPU
Cost$0–$10

Learn#

Training loop#

load batch
zero gradients
forward
loss
backward
optimizer step
scheduler step
logging
validation
checkpoint

Training features#

  • gradient accumulation
  • effective batch size
  • fp16
  • bf16
  • fp8 conceptually
  • warmup
  • cosine decay
  • linear decay
  • gradient clipping
  • checkpointing
  • resume
  • reproducibility
  • activation checkpointing

Metrics#

Track:

  • train loss
  • validation loss
  • tokens/sec
  • step time
  • peak VRAM
  • GPU utilization
load batchfrom shardsforwardbf16 autocastlosscross entropybackwardaccumulateclip + stepAdamWlogmetricsEvery N steps: validation loss, tokens/sec, peak VRAM, checkpoint. Without those four numbers a run is not an experiment.
One optimizer step. Gradient accumulation repeats forward and backward before the step, which is how you fake a large batch on small hardware.

Build#

Train several models, for example:

10M
30M
60M
100M

For every run record:

  • parameter count
  • training tokens
  • context length
  • batch size
  • optimizer
  • learning rate
  • hardware
  • runtime
  • cost
  • train loss
  • validation loss
  • throughput
  • peak memory

Controlled experiments#

Change one variable at a time:

  • learning rate
  • depth
  • width
  • context length
  • tokenizer vocabulary
  • LayerNorm vs RMSNorm
  • GELU vs SwiGLU
  • manual attention vs SDPA

Resources#

Required#

Experiment tracking#

Advanced#

Done when#

You can distinguish from logs and profiling data:

  • modeling failure
  • optimizer instability
  • data bottleneck
  • OOM
  • GPU underutilization
  • overfitting
Phase 9

Pretraining data pipeline#

Time12–20 hours
GPUMostly CPU
Cost$0

Learn#

  • data licensing
  • document filtering
  • exact deduplication
  • MinHash
  • near deduplication
  • language identification
  • quality filters
  • PII handling
  • tokenization at scale
  • sharding
  • memory mapping
  • dataloader workers
  • prefetching
  • shuffling
  • sequence packing
  • contamination
raw docslicensed sourcefilterquality, languagededupeexact, MinHashtokenizeuint16 idsshardmemory mappedpacked batchesno padding wasteProfile each stage separately. A GPU at 30 percent utilization is usually a dataloader problem, not a model problem.
The pretraining data path. Contamination checks belong at the filter stage, before anything is tokenized.

Build#

Start with:

50M–500M tokens

Pipeline:

raw documents
→ filter
→ deduplicate
→ tokenize
→ shard
→ dataloader
→ training batches

Resources#

Required#

Supplementary#

Useful for:

  • MinHash
  • LSH
  • simple filtering
  • packing
  • dataloader structure

Do not copy its ASCII-only cleaning rule into a general multilingual pipeline.

Done when#

You can identify whether slow training comes from:

  • disk
  • CPU preprocessing
  • tokenization
  • dataloader
  • host-to-device transfer
  • GPU compute
Phase 10

Scaling laws and resource accounting#

Time8–12 hours
GPUNone
Cost$0

Learn#

Relationships between:

  • parameter count
  • training tokens
  • training compute
  • loss

Use the common rough dense-transformer estimate:

training FLOPs ≈ 6 × parameters × tokens

Memory accounting#

Estimate:

  • weights
  • gradients
  • optimizer states
  • activations
  • temporary buffers
  • KV cache

Build#

Create:

estimate_training_flops()
estimate_training_time()
estimate_training_memory()
estimate_kv_cache()

Compare estimates with one actual run.

Resources#

Done when#

You can predict, before launching a run you have never done:

  • training FLOPs from parameter count and token count
  • wall-clock time on a named GPU
  • peak training memory, split into weights, gradients, optimizer states and activations
  • KV cache size at a given batch size and context length

Your estimates land within roughly a factor of two of one real run you then measure, and you can say what Chinchilla changed about the parameters-to-tokens tradeoff.

Phase 11

Distributed training fundamentals#

Time10–20 hours
GPUOptional multi-GPU
Cost$0–$10 for small experiments; more for real multi-GPU runs

Learn#

DDP#

One model replica per GPU, different batches, synchronized gradients.

FSDP#

Shard model state across GPUs.

ZeRO#

Understand stages 1, 2, and 3.

Tensor parallelism#

Split large tensor operations across GPUs.

Pipeline parallelism#

Split layers across GPUs.

NCCL#

Communication#

Understand:

  • PCIe
  • NVLink
  • inter-node networking
  • collective communication

Resources#

Required#

Supplementary#

Use it as an introduction; use CS336 and Ultra-Scale for the detailed treatment.

Done when#

You can explain what problem each solves:

  • DDP
  • FSDP
  • ZeRO
  • tensor parallelism
  • pipeline parallelism
Phase 12

Fine-tune a real open model#

Time15–25 hours
GPU16–48 GB depending on model
Cost$0–$10 for small models

Start with a current model in roughly the:

0.5B–3B parameter

range.

Learn#

  • base vs instruct
  • full fine-tuning
  • supervised fine-tuning
  • LoRA
  • QLoRA
  • rank
  • alpha
  • target modules
  • adapter merging
  • quantized frozen base weights

Build#

Create:

500–2,000 high-quality instruction examples

Train:

Experiment A#

Your own small pretrained model → SFT assistant

Experiment B#

Real open 0.5B–3B base model → LoRA/QLoRA using similar data

Compare:

  • task eval
  • training loss
  • training cost
  • inference behavior

Resources#

Papers#

Libraries#

Model hubs#

Supplementary#

Its LoRA implementation is useful for understanding low-rank adapters. Its NF4 example is a simplified simulation, so use bitsandbytes/QLoRA tooling for the real experiment.

Done when#

Two checkpoints exist: your own pretrained model taken through SFT, and a real 0.5B–3B base model taken through LoRA or QLoRA. You can:

  • state the rank, alpha and target modules you chose, and why those
  • say how many parameters you actually trained, as a fraction of the base model
  • merge an adapter back into the base weights and get the same outputs
  • account for the gap in cost and quality between the two runs
Phase 13

Chat templates and instruction data#

Time8–12 hours

Learn#

  • system/user/assistant roles
  • special chat tokens
  • EOS handling
  • response boundaries
  • assistant-only loss masking
  • dataset formatting
  • instruction diversity
  • duplicate detection
  • synthetic data
  • filtering
  • task balance

Build#

Train the same base model on:

Dataset A#

Small curated dataset.

Dataset B#

Larger noisier dataset.

Evaluate both on the same private eval set.

Resources#

Done when#

You can take one raw conversation and show, token by token, what the model actually trains on:

  • where each special token sits and what it delimits
  • which positions are masked out of the loss and which are not
  • what happens at inference if EOS is never emitted

Both datasets are trained and scored on the same private eval set, and you can say which won for a reason other than being bigger.

Phase 14

Evaluation#

Time15–22 hours
GPUFree or cheap GPU
Cost$0–$5

Learn#

Base-model metrics#

  • held-out loss
  • perplexity

Benchmarks#

Know what these test:

  • MMLU
  • ARC
  • HellaSwag
  • GSM8K
  • HumanEval
  • IFEval

Evaluation concepts#

  • zero-shot
  • few-shot
  • contamination
  • pairwise evaluation
  • LLM-as-judge
  • judge bias
  • task-specific evaluation

Build#

Create:

evals/private_eval.jsonl

with roughly 50–100 examples.

Each item should include:

  • prompt
  • expected information
  • evaluation criteria
  • explicit failure conditions

Compare:

  • base model
  • SFT model
  • DPO model later

Resources#

Done when#

Your private eval set exists, holds 50 or more items, and every item carries an explicit failure condition rather than a judgement call. You can:

  • score the same model twice and get the same number
  • say what each of MMLU, GSM8K, HumanEval and IFEval actually tests
  • explain how contamination would show up in your results
  • name a specific bias in your judge, and what you did about it
Phase 15

Preference optimization#

Time10–18 hours
GPUCheap GPU
Cost$2–$10

Learn#

Classic RLHF pipeline:

pretraining
→ SFT
→ preference data
→ reward model
→ RL optimization

Start practical work with DPO rather than implementing PPO first.

base modelnext-token onlySFTloss on the replypreference pairschosen, rejectedDPOno reward modelaligned modelevaluated, servedScore every stage on the same private eval set. Benchmarks tell you about the field, your eval set tells you about your model.
Post-training in the order you should build it. PPO and reward models are worth understanding, but DPO is where to start.

Build#

Preference example:

prompt
chosen
rejected

Train DPO and compare against the SFT checkpoint.

Resources#

Supplementary#

Done when#

A DPO checkpoint exists and is scored against its SFT parent on the same private eval set. You can:

  • explain why DPO needs no reward model, and what PPO uses one for
  • point at concrete behaviour that changed between the two checkpoints
  • show a case where DPO made the model worse, not better
Phase 16

Quantization#

Time10–15 hours
GPUOptional
Cost$0

Learn#

  • fp32
  • fp16
  • bf16
  • int8
  • int4
  • quantization error
  • GGUF
  • GPTQ
  • AWQ

Build#

Quantize one model at two levels.

Measure:

  • file size
  • RAM
  • VRAM
  • tokens/sec
  • TTFT
  • eval degradation

Resources#

Done when#

The same model exists at two quantization levels, and one table puts file size, memory, tokens per second, TTFT and private-eval score beside the unquantized baseline. You can:

  • say which level you would ship, and what you are trading away to do it
  • name a task where the degradation is visible, not just a number that moved
  • explain what GGUF, GPTQ and AWQ are each for
Phase 17

Serving#

Time10–15 hours
GPUOptional
Cost$0–$5

Learn#

  • static batching
  • continuous batching
  • PagedAttention
  • prefix caching
  • throughput
  • latency
  • TTFT
  • tokens/sec
  • speculative decoding
  • structured generation
  • constrained decoding

Build#

Serve a model through an OpenAI-compatible endpoint using:

  • llama.cpp server, or
  • vLLM

Benchmark:

  • one request
  • concurrent requests
  • short prompt
  • long prompt
  • different output lengths

Resources#

Supplementary#

Done when#

Your model answers through an OpenAI-compatible endpoint you started yourself, and you have benchmark numbers for a single request and for many concurrent ones. You can:

  • show throughput rising while per-request latency degrades, and explain that shape
  • explain what continuous batching does that static batching cannot
  • say what PagedAttention solves, in terms of the KV cache you built in Phase 7
Phase 18

nanochat capstone#

Time10–20 hours plus compute
GPUMulti-GPU if following the full reference run
Costtens of dollars depending on provider and current GPU pricing

Baseline run#

Record:

  • hardware
  • runtime
  • cost
  • model size
  • training tokens
  • loss curves
  • eval results

Modified run#

Change one variable:

  • depth
  • width
  • training tokens
  • tokenizer vocabulary
  • context length
  • learning rate
  • SFT mixture

Compare against baseline.

Done when#

You can map every major nanochat stage to an implementation or concept already covered earlier in the roadmap.

Phase 19

Stanford CS336#

Use CS336 after the transformer and training phases.

Focus on:

  • tokenizer training
  • transformer implementation
  • optimizer implementation
  • systems profiling
  • FlashAttention
  • Triton
  • distributed training
  • scaling
  • data
  • evaluation

Complete the assignments relevant to your target role rather than treating the course as mandatory end-to-end work.

24

Optional: LLM application engineering#

This is separate from model training.

Topics#

  • prompting
  • structured outputs
  • embeddings
  • context engineering
  • RAG
  • advanced RAG
  • function calling
  • tools
  • MCP
  • LLM application evaluation
  • caching
  • guardrails
  • production deployment
  • agents

Resource#

Useful lessons:

  • Prompt Engineering
  • Structured Outputs
  • Embeddings
  • Context Engineering
  • RAG
  • Advanced RAG
  • Fine-Tuning with LoRA & QLoRA
  • Function Calling
  • Evaluation & Testing
  • Caching & Cost Optimization
  • Guardrails
  • Production LLM Application
  • MCP
  • Prompt Caching

Tools and protocols#

Use for:

  • tool interfaces
  • function calling
  • tool schemas
  • MCP clients and servers
  • MCP transports
  • security
  • authorization
  • gateways and registries
25

Optional: Speech and voice AI#

Relevant lessons:

  • audio fundamentals
  • spectrograms
  • ASR
  • Whisper
  • speaker recognition
  • TTS
  • voice conversion
  • audio-language models
  • real-time audio
  • voice assistant pipeline
  • neural audio codecs
  • VAD
  • turn-taking
  • streaming speech-to-speech
  • audio evaluation
26

Optional: Multimodal models#

Topics include:

  • ViT
  • CLIP
  • BLIP-2
  • LLaVA
  • Qwen-VL
  • InternVL
  • document understanding
  • multimodal RAG
  • video-language models
  • audio-language models
  • multimodal agents
27

Optional: Triton and kernels#

Learn:

  • GPU execution model
  • memory hierarchy
  • tiling
  • kernel fusion
  • matrix multiplication
  • fused softmax
  • normalization kernels
  • attention kernels
28

Optional: Mixture of Experts#

Study:

  • experts
  • router
  • top-k routing
  • load balancing
  • auxiliary losses
  • expert parallelism
  • serving implications
29

Optional: Reasoning and RL#

Study:

  • PPO
  • reward models
  • verifiable rewards
  • GRPO
  • rejection sampling
  • reasoning data
  • distillation
  • test-time compute
30

GPU strategy#

Use the cheapest hardware that fits the experiment.

CPU#

Use CPU for:

  • shell
  • Python
  • NumPy
  • tensor exercises
  • micrograd
  • tokenizers
  • tiny language models
  • most data preprocessing

Free GPU options#

Kaggle#

Useful for:

  • PyTorch exercises
  • small GPT training
  • LoRA experiments
  • 10M–50M models

Free quotas vary.

Google Colab#

GPU type and runtime limits are not guaranteed.

Lightning AI#

Check the live free-credit allowance and GPU availability.

Paid GPU providers#

Compare live prices before every paid run.

31

GPU selection by task#

Task Hardware target
micrograd CPU
tensor exercises CPU
BPE tokenizer CPU
makemore CPU/free GPU
10M GPT free GPU
30M GPT free GPU
50M–100M experiments 16–24 GB GPU
LoRA 0.5B–3B 16–24 GB GPU
QLoRA 3B–8B 24 GB often sufficient depending on context and batch size
larger single-GPU experiments 48 GB GPU
full fine-tuning requiring more memory A100 80 GB-class
multi-GPU capstone H100-class if following nanochat reference hardware
33

Profiling slow training#

Possible bottlenecks:

  • disk
  • CPU preprocessing
  • tokenization
  • dataloader workers
  • host-to-device copy
  • GPU compute
  • GPU memory bandwidth
  • synchronization
  • network communication

Tools#

34

Required mathematics#

Linear algebra#

Required:

  • vectors
  • matrices
  • tensors
  • dot products
  • matrix multiplication
  • transpose
  • norms
  • projection intuition

Calculus#

Required:

  • derivative
  • partial derivative
  • gradient
  • chain rule

Probability#

Required:

  • random variables
  • categorical distributions
  • expectation
  • variance
  • sampling

Information theory#

Required:

  • entropy
  • cross entropy
  • KL divergence
  • log probability
  • perplexity
36

Literal resource queue#

If you want one exact queue, use this.

  8. Implement RMSNorm, RoPE, SwiGLU, GQA

  9. Read RoPE, GQA, FlashAttention

  10. Add KV cache and benchmark generation

  12. Run multiple 10M–100M training experiments

  14. Build a 50M–500M-token data pipeline

  15. Study scaling laws

  17. Fine-tune a current 0.5B–3B open base model with LoRA/QLoRA

  18. Build a private eval set

  19. Run DPO

  20. Quantize with llama.cpp

  21. Serve with vLLM or llama.cpp

  22. Run nanochat end to end

37

Skip until later#

Skip these during the core LLM path:

  • decision trees
  • random forests
  • SVMs
  • KNN
  • PCA/t-SNE/UMAP
  • most computer vision
  • GANs
  • diffusion models
  • full RL curriculum
  • advanced mechanistic interpretability
  • Triton
  • CUDA kernels
  • Kubernetes
  • multi-node orchestration
  • MoE implementation
  • multimodal training
38

Repository structure#

llm-from-scratch/
│
├── README.md
├── pyproject.toml
├── notes/
│   ├── tensors.md
│   ├── backprop.md
│   ├── tokenization.md
│   ├── attention.md
│   ├── training.md
│   ├── data.md
│   ├── finetuning.md
│   └── inference.md
│
├── src/
│   ├── autograd/
│   ├── tokenizer/
│   ├── makemore/
│   ├── vanilla_gpt/
│   ├── modern_gpt/
│   ├── training/
│   ├── data/
│   ├── inference/
│   └── evals/
│
├── experiments/
│   ├── 001_learning_rate/
│   ├── 002_depth_width/
│   ├── 003_context_length/
│   ├── 004_tokenizer_vocab/
│   ├── 005_sdpa/
│   ├── 006_lora/
│   ├── 007_dpo/
│   └── nanochat/
│
├── evals/
├── tests/
└── checkpoints/

Do not commit large checkpoints directly to Git.

39

Experiment template#

# Experiment

## Question
What am I testing?

## Hypothesis
What result do I expect?

## Baseline
What is the comparison?

## Change
What one variable changed?

## Configuration
Model:
Parameters:
Training tokens:
Batch:
Context:
Learning rate:
Optimizer:
Hardware:

## Results
Train loss:
Validation loss:
Throughput:
Peak VRAM:
Eval score:
Runtime:
Cost:

## Conclusion
What changed and why?
40

Completion checklist#

Foundations#

Language modeling#

Tokenization#

Transformer#

Modern architecture#

Training#

Data#

Real-model work#

Evaluation#

Post-training#

Inference#

Scaling and systems#

Capstone#

41

Intensive schedule#

Week 1#

  • environment
  • tensors
  • backprop
  • micrograd

Week 2#

  • makemore
  • BPE
  • vanilla GPT

Week 3#

  • RMSNorm
  • RoPE
  • SwiGLU
  • GQA
  • KV cache
  • inference benchmarking

Week 4#

  • training loop
  • mixed precision
  • checkpointing
  • controlled model-size experiments
  • data pipeline

Week 5#

  • scaling laws
  • distributed training concepts
  • real open model
  • LoRA/QLoRA
  • chat templates

Week 6#

  • evaluation
  • DPO
  • quantization
  • serving

Week 7#

  • nanochat
  • modified capstone run
  • benchmark comparison

This schedule assumes roughly 30–35 focused hours per week. At 15–20 hours per week, use roughly twice the calendar time.

42

Final project#

Data#

Document:

  • source
  • license
  • cleaning
  • deduplication
  • document count
  • token count

Tokenizer#

Document:

  • vocabulary size
  • compression ratio
  • special tokens

Base model#

Document:

  • architecture
  • parameter count
  • training tokens
  • optimizer
  • learning-rate schedule
  • hardware
  • runtime
  • cost
  • loss curves

Modern architecture#

Use:

  • RMSNorm
  • RoPE
  • SwiGLU
  • GQA

Post-training#

Run:

  • SFT
  • LoRA/QLoRA
  • DPO

Evaluation#

Include:

  • held-out loss
  • selected public benchmarks
  • private eval set

Inference#

Include:

  • KV cache
  • quantized model
  • local endpoint
  • TTFT
  • tokens/sec

Comparison#

Compare against a similarly sized public model and document:

  • where the public model performs better
  • where your model performs better
  • likely causes
  • limitations of the comparison