Project · Submit: live hybrid RAG + eval results + writeup

Hints

• Don't over-engineer chunking. Recursive + markdown-aware handles most.
• Use Anthropic Citations API if you can.
• Set the eval CI gate before iterating. Otherwise you'll never know what helped.
• Pick a corpus you actually know. Eval cases are easier when you understand the content.

Expected output

# Hybrid RAG over Engineering Wiki

Corpus: 412 markdown docs, ~1900 chunks
Eval: 50 cases (15 user-Qs, 25 from team logs, 10 adversarial)

Final metrics:
  recall@5:        0.91  (target 0.85 ✓)
  accuracy:        0.86  (target 0.80 ✓)
  p99 latency:     1.7s  (target 2.0s ✓)
  avg cost/query:  $0.0072 (target $0.01 ✓)

Surprises:
- "version-number" queries (e.g. "v2.1") needed BM25; dense missed them.
- Citations API moved trust the most; users started clicking sources.
- Rerank lifted accuracy 14% over hybrid-only.

Stretch goals

• Add Anthropic's Contextual Retrieval (prepend a chunk-context blurb at index time). Often a +20% lift.
• Multi-tenant: add a tenant_id filter and verify isolation.
• Build a 'why this answer?' debug toggle that shows the chunks injected + scores.
• Ship to a teammate; capture their feedback for 1 week.