Project · Write the spec for your agent's telemetry

project

hard

module project

Ship something real. Submit your work when you're done.

Brief

Before instrumenting anything, produce the design document: your failure taxonomy from real incidents, your five questions with what each requires, your chosen unit of work, your sampling strategy with arithmetic, and an honest inventory of what your current tooling already gives you. Then stand up the local stack and get one turn traced with the correct topology, proven by a test.

Deliverables

A `failure-taxonomy.md` with at least six failure modes from your own incidents, each with its telemetry signature and current detection status.
An `agent-telemetry-questions.md` with five real questions, what each requires, and present/partial/missing.
A one-paragraph decision on your unit of work, naming the questions it makes easy and the one it makes hard.
A sampling strategy with the arithmetic: your step-count and cost distributions, what uniform sampling would lose, and your stratification rules with weights.
An honest inventory of your current LLM tooling: what it gives you and what it does not, with one slow turn you could not fully account for.
A running local stack with one agent turn visible as a single trace, plus the span-topology test using a scripted model.

How we grade it

The failure taxonomy comes from your own incidents, not from the module's list — at least two entries must be specific to your product.
Every question in the questions file names concrete attributes, not categories like 'more context'.
The sampling arithmetic uses your measured distributions and shows what a uniform sample would capture from the tail.
The inventory includes one real slow turn with the unaccounted duration quantified.
The span-topology test fails if the turn root span is removed or the trace splits — demonstrate both.

Hints

Do the taxonomy from support tickets rather than from imagination; the ordering by real frequency is the whole value, and it usually surprises people. For the local stack, set OTEL_TRACES_SAMPLER=always_on immediately — a sampled-out trace during development wastes more time than any other single thing here. The scripted stub model is worth building properly now: every later module's tests depend on it.

Expected output

$ docker compose -f docker-compose.telemetry.yml up -d $ OTEL_SERVICE_NAME=research-agent OTEL_TRACES_SAMPLER=always_on \ OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:14317 python agent.py \ --q "what changed in the Q3 filing?" $ pytest tests/test_agent_spans.py -q ..... 5 passed trace 4bf92f3577b34da6a3ce929d0e0e4736 (jaeger, 1 root, 6 spans) invoke_agent research-assistant 1,840ms SERVER gen_ai.conversation.id = c_8813 gen_ai.conversation.turn_index = 0 ├─ chat gpt-4o 410ms CLIENT ├─ execute_tool search_docs 180ms INTERNAL ├─ chat gpt-4o 520ms CLIENT └─ chat gpt-4o 730ms CLIENT sampling-strategy.md measured: steps p50 3 p95 7 p99 14 cost p50 $0.012 p99 $0.31 uniform 1% of 10,000 turns keeps ~0 turns with >=14 steps stratified: user_feedback 100% · debug 100% · version_rollout 100% enterprise 100% · first_turn 100% · baseline 2% (w=50) estimated kept/day: 610 of 10,000 (6.1%), containing every flagged turn and every enterprise turn inventory.md slow turn 8ac9de11: vendor UI accounts for 1,900ms of 4,210ms unaccounted 2,310ms -> a SELECT and a vector search, invisible there

Stretch goals

Run the same prompt fifty times and plot the step-count and cost distributions rather than quoting percentiles.

Compute mean cost per turn from a stratified sample with and without weights, and quantify the error.

Add an integration test against a real model asserting only the attributes a real response provides.

Write the one-line standard for your unit of work and circulate it to whoever owns the second agent service.

Identify which of your failure modes no amount of telemetry will catch, and say what would.