Project · Answer one incident question end to end

Hints

Pick the smallest service you have, not the most important one. If you have no incident history, use the questions your on-call rotation asks in the first five minutes of any page. The unaccounted gap between a parent span's start and its first child is the fastest way to find where your code is doing invisible work.

Expected output

$ docker run --rm -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one
$ OTEL_SERVICE_NAME=checkout GIT_SHA=a3f91c2 ENV=local \
    opentelemetry-instrument python app.py

$ pytest tests/test_telemetry.py -q
..                                                              2 passed

trace 4bf92f3577b34da6a3ce929d0e0e4736  (jaeger UI)
  POST /checkout                    412ms  SERVER
    tenant.id = acme                       <-- the question, answered
    service.version = a3f91c2
    ├─ (338ms unaccounted)                 <-- gap list item 1
    ├─ SELECT cart_items               11ms CLIENT
    └─ POST /v1/charges                58ms CLIENT

gap-list.md
  1. 338ms in pricing.compute() — which of 3 branches ran is invisible
  2. cache lookup in cart.get() — hit or miss not recorded
  3. the 3 retry attempts against /v1/charges collapse into one span

Stretch goals

• Add the parent-child assertion to CI and break it on purpose in a pull request to see the failure message a colleague would get.
• Send the same traces to two different local backends via a Collector and compare what each UI surfaces.
• Measure the overhead: request latency and CPU with the SDK registered versus with a no-op provider.
• Write the query for each of your five questions and record which ones are still unanswerable.