Why more context is not better context

Why this matters

It's tempting to dump everything into the window 'just in case' — the model is smart, let it sort it out. But irrelevant context actively hurts: it dilutes attention, raises the chance the model latches onto the wrong passage, increases latency and cost, and pushes important material toward positions the model attends to less. The skill is subtractive. A tight context of five relevant chunks beats a sprawling one of fifty, almost every time. This is the single most counterintuitive lesson for people coming from prompt engineering, where 'add more instructions' usually helps.

Demo

This is measurable, not philosophical. Take a question with one correct supporting passage, then answer it twice: once with just that passage, once with the passage buried among 40 distractors. Accuracy and latency both move in the wrong direction as you add noise.

Try it yourself

1. Run it. Both may answer correctly here, but note the latency gap — noise costs time and money even when accuracy survives.
2. Bury GOLD in the exact middle of 60 distractors and ask again. This is the 'lost in the middle' failure mode you'll study in Module 9.
3. Replace distractors with near-duplicates of GOLD that contradict it (e.g. '24 months'). Watch accuracy collapse — conflicting context is worse than irrelevant context.
4. Reduce to the single GOLD block and confirm the answer is both correct and fastest. Subtraction is the optimization.

References

• Liu et al. — Lost in the Middle (2023)
• Anthropic — Long context tips

Chat about this lesson