Start From the Question, Not the Tool

Why this matters

Most instrumentation is added tool-first: someone installs an agent, a dashboard appears, and nine months later an incident happens and the dashboard cannot answer the question being asked. The fix is to invert the order. Write down the five questions you were actually asked during your last three incidents — 'is it everyone or one customer?', 'did it start when we deployed?', 'is it us or the payment provider?' — and treat each one as a requirement. Every question implies a specific signal with specific attributes. A question you cannot answer from what you emit today is a gap; a signal that answers no question on the list is cost you are paying for nothing.

Demo

This is a worked example of the inversion. Keep it in the repo as telemetry-questions.md and revisit it after every incident, because incidents are the only honest source of new questions. Note how narrow each answer is: the point is not to emit more, it is to emit the one attribute that turns a shrug into a bisection.

Try it yourself

1. Open your last three incident write-ups and copy out every question someone asked in the channel. Do not paraphrase them into tidy requirements yet.
2. For each question, name the single attribute or signal that would have answered it in one query. If you need two joins, the answer is 'missing', not 'present'.
3. Mark each one present, partial, or missing against what your service emits today — check the actual exporter output, not the documentation.
4. List every metric or label you emit that answers nothing on your question list, and put a rough series count next to each.
5. Pick the single missing item that appeared in the most incidents. That is the first thing you instrument in module 2.

References

• OpenTelemetry — Observability primer
• Google SRE Workbook — Alerting on SLOs