Project · Fine-tune and deploy a domain-adapted model

Hints

• For a 7B model on a 24GB GPU, use QLoRA (4-bit + LoRA). On an 80GB GPU, plain LoRA in bf16 is faster.
• Start with rank=16, alpha=32, target_modules=['q_proj','v_proj','k_proj','o_proj']. This is the standard starting point — adjust after seeing initial results.
• The most common mistake: computing loss on prompt tokens. Always use DataCollatorForCompletionOnlyLM and verify labels are -100 for prompt tokens.
• Use model.merge_and_unload() after training to merge LoRA weights into the base model before deploying to vLLM. vLLM doesn't natively support unmerged LoRA for all model types.

Expected output

$ python fine_tune.py
Loading model: mistralai/Mistral-7B-Instruct-v0.3 in 4-bit...
Trainable parameters: 20,185,088 (0.29% of total)
Training on 500 examples, 3 epochs...
Epoch 1/3: loss=1.24  lr=1.8e-4
Epoch 2/3: loss=0.89  lr=1.2e-4
Epoch 3/3: loss=0.71  lr=2.1e-5
Peak GPU memory: 14.2 GB
Saved checkpoint: ./checkpoints/mistral-7b-lora-customer-support

$ python eval.py
Evaluating 20 examples (base model vs fine-tuned)...
Fine-tuned wins: 14/20 (70%)
Ties:             3/20 (15%)
Base wins:        3/20 (15%)
Win+tie rate: 85%

Stretch goals

• Run DPO after SFT: collect 200 preference pairs from the SFT model (have an LLM generate both a good and bad response to each eval prompt, with scores), then train a DPO checkpoint. Does win rate improve further?
• Compare three rank settings (4, 16, 64) on the same dataset. Plot win rate vs trainable parameter count. Is there a point of diminishing returns?
• Set up an A/B test: 50 eval examples sent to both models, with a human judge (you) providing verdicts instead of an LLM judge. Compare human judgement to the LLM judge's results. What's the agreement rate?
• Add quantized inference: compare bf16 serving vs AWQ int4 serving on vLLM. Measure TTFT and tokens/sec for each. Is the accuracy tradeoff (measured by win rate) acceptable?