Project · Ship naive RAG end-to-end

Hints

• Don't reach for a managed vector DB. numpy or pgvector is enough at this scale.
• If your corpus is < 100 docs, you don't need RAG yet — paste the whole thing in the system prompt.
• Pick an embedding model and stick with it. Don't compare four — compare two if anything.
• Eval first. Build the eval set BEFORE the second iteration of retrieval, not after.

Expected output

# Naive RAG results
- Corpus: 412 docs (internal engineering wiki)
- Chunks: 1,847 (~500 tokens, 100 overlap)
- Embedding: text-embedding-3-small (1536 dim)
- Store: pgvector + HNSW

# Eval (30 cases)
- Retrieval@5: 73%
- Answer accuracy (LLM-judge): 67%
- Refusal rate: 12%

# Top failure modes
1. Chunk boundary splits answer (8/30 cases) -> Module 2 fixes
2. Query vocabulary mismatch (5/30) -> Module 5 query rewriting fixes
3. Multi-hop required (3/30) -> Module 6 fixes

Stretch goals

• Add per-tenant filtering: each chunk has a tenant_id metadata field; queries filter to one tenant.
• Compare two embedding models on the same eval set. Often the difference is < 5% — but you'll know.
• Build a tiny web UI (just a textbox + answer) so you can sit with non-engineers and watch them break it.
• Add a 'why this answer?' debug toggle: show the chunks that were injected, their similarities, and the chosen threshold.