Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Full fine-tuning a 7B-parameter model requires storing and updating 7 billion gradient values — roughly 28 GB just for the optimizer state in Adam, before you count activations or model weights. LoRA (Low-Rank Adaptation) sidesteps this by observing that the weight update ΔW during fine-tuning tends to be low-rank: instead of updating the full W matrix, you learn two small matrices A and B such that ΔW ≈ BA. For a 4096×4096 weight matrix with rank 16, you update 2×(4096×16)=131,072 parameters instead of 16,777,216 — 128× fewer. This is why you can fine-tune a Llama-3 8B model on a single 24 GB consumer GPU: LoRA parameters are ≤1% of the original model.
LoRA replaces a full weight update ΔW ∈ ℝ^{d×d} with the product of two skinny matrices A ∈ ℝ^{r×d} and B ∈ ℝ^{d×r}, where r≪d. Because B is initialized to zero, the adapter contributes nothing at the start of training — the pretrained model is preserved exactly — then gradually learns the task delta. The code below makes the memory arithmetic concrete: a 4096×4096 layer at rank 16 trains 131 K parameters instead of 16.7 M.
rank=16 to rank=1. Recompute LoRA params and the ratio. At rank=1, what does ΔW look like geometrically? (It's an outer product of two vectors — the most compressed possible update.)rank=512 (equal to half the layer size). How do LoRA params compare to the full W? At what rank does LoRA stop saving memory?layer.B. Confirm layer(x) == F.linear(x, layer.W) before any training step. This zero-init ensures LoRA doesn't perturb the pretrained model at the start of training.from peft import LoraConfig, get_peft_model; config = LoraConfig(r=16, lora_alpha=32, target_modules=['q_proj','v_proj']); model = get_peft_model(model, config); model.print_trainable_parameters(). Compare the trainable % to your manual calculation.import torch
import torch.nn as nn
import torch.nn.functional as F
class LoRALinear(nn.Module):
"""A linear layer with a low-rank ΔW = B @ A injected alongside the frozen W."""
def __init__(self, in_features, out_features, rank=16, alpha=32):
super().__init__()
self.in_features = in_features
self.out_features = out_features
self.rank = rank
self.scale = alpha / rank # scaling factor
# Frozen pretrained weight
self.W = nn.Parameter(torch.randn(out_features, in_features) * 0.02)
self.W.requires_grad = False # freeze W
# Trainable LoRA matrices
self.A = nn.Parameter(torch.randn(rank, in_features) * 0.01) # small init
self.B = nn.Parameter(torch.zeros(out_features, rank)) # zero init → ΔW=0 at start
def forward(self, x):
base = F.linear(x, self.W) # frozen path
delta = F.linear(F.linear(x, self.A), self.B) * self.scale # ΔW path
return base + delta
layer = LoRALinear(4096, 4096, rank=16)
total_params = sum(p.numel() for p in layer.parameters())
lora_params = sum(p.numel() for p in [layer.A, layer.B])
frozen_params = layer.W.numel()
print(f"Total: {total_params:,}") # 16,910,336
print(f"Frozen: {frozen_params:,}") # 16,777,216
print(f"LoRA: {lora_params:,}") # 131,072
print(f"Ratio: {lora_params/frozen_params*100:.2f}%") # 0.78%
x = torch.randn(1, 512, 4096) # (batch, seq_len, d_model)
out = layer(x)
print(f"Output shape: {out.shape}") # (1, 512, 4096)python3 main.py