Zero errors, normal latency, confidently wrong answer. The instrumentation that makes an LLM system debuggable rather than merely monitored.
Your agent returned 200 in 1.9 seconds and told the customer something that is not true. Nothing in your monitoring will ever tell you that, because every assumption ordinary tracing makes — same input, same call graph; a knowable request shape; bounded cost; success is a status code — is false for an agent. This course is the instrumentation that closes the gap, built on the OpenTelemetry GenAI semantic conventions rather than on any vendor's SDK, because the instrumentation is the asset that outlives the tool. You start by writing down the questions you get asked after a bad answer, then make each one answerable. The gen_ai.* vocabulary, including the request-versus-response model split that is the only way to catch a provider silently repointing an alias underneath you. One wrapper so no call site can forget an attribute, with token normalisation across providers and cached tokens priced separately, because pricing a cache hit at the full rate overstates cost by most of your cache hit rate. Then the shape of an agent run: step spans that make the loop measurable, the context-growth curve that predicts a runaway before the bill does, three kinds of retry that need three attributes, and the critical path through a parallel fan-out. Tool spans that record whether a tool returned nothing — one boolean, and the strongest predictor of a hallucinated answer you can get for free. Retrieval provenance that routes a RAG complaint to retrieval, the prompt, or the context budget in one query, and finds the stale documents your answers keep citing. Streaming instrumented properly, so the span outlives the first byte and time-to-first-token becomes the number your SLO is written on. A prompt-capture policy you could defend to an auditor: structure by default, redaction in the Collector, full content only for a narrow consented slice, and a deletion path that has actually been executed. Cost derived from tokens rather than duplicated into a counter that will disagree with itself, with exemplars turning a spend spike into two clicks. Handoff telemetry that catches the intent loss multi-agent systems fail on, and cycle detection that a depth limit misses. Then quality as a real SLI — sampled, weighted, judged by a pinned model with a calibration set, because the first thing to check when quality drops is whether your measuring instrument moved. It ends with the pipeline assembled and a dashboard on-call will actually read. Python throughout, verifiable on localhost with no vendor account.
Built by Lakshya Kumar
Paste this into any AI chat. Fill in the bracketed parts with your context — you'll get back a straight answer on whether this belongs on your plate.
We grant free access case-by-case — students, career-switchers, builders on a tight budget. Sign in to send us a note.
Sign in to applyFinished the tasks? Take the prompt to your AI and get tested on it. We copy the prompt and open the app — just paste it in.
One agreed vocabulary for models, tokens, tools and agents — experimental, moving, and still far better than inventing your own.
One wrapper around your provider client, emitting conventional spans and metrics that any OTLP backend understands.
Steps, retries, sub-agents and fan-out each need a shape. Get the tree right and the critical path reads itself.
Tools are where an agent touches the world. Instrument the argument, the result, the failure kind and the authorisation.
Which chunks came back, which survived reranking, and which actually reached the prompt. Provenance, per span.
The span must outlive the first byte. Instrument the stream itself and time-to-first-token becomes your real SLI.
You cannot debug what you cannot see, and you cannot store what you must not keep. Redact at the Collector, sample by tenant.
Derive cost from span attributes rather than maintaining a parallel counter, so the trace and the invoice cannot disagree.
Every handoff is a lossy summary. Instrument what was passed, what was dropped, and whether the intent survived.
Your error rate can be zero during a total quality outage. Sample a quality signal, then burn-rate alert on it.
Langfuse, Phoenix and Arize all speak OTLP. Fan out from one Collector, keep what each is good at, change your mind later.
I'm taking "See Inside Your Agents: Telemetry for AI Systems" — an agent instrumentation course in the Agentic and Applied AI track. Twelve modules: why LLM systems break normal tracing, the gen_ai semantic conventions, vendor-neutral instrumentation, span topology for agent loops, tool and MCP telemetry, RAG span provenance, streaming, prompt capture without leaking, cost as a derived metric, multi-agent handoffs, quality SLOs, and wiring up a backend. My context: 1. My stack and framework: [describe — LangChain / LlamaIndex / bare SDK / other] 2. What telemetry exists today: [nothing / a vendor LLM tool / partial OTel / a full APM] 3. Whether my agent uses tools, retrieval, sub-agents, streaming: [describe] 4. My monthly model spend, if I know it: [number] 5. The last wrong answer I could not explain: [describe] 6. Whether I can store prompts at all, and under what policy: [describe] Given that, answer: - Which module should I start with, given what already exists? Skipping ahead is fine if the earlier ground is covered. - Which parts of this course do NOT apply to my situation, and why? - For my last unexplained wrong answer, which specific attribute would have explained it? - What is the single cheapest change that would most improve my ability to diagnose the next one?
The specification pages matter more than the language guides once you are past setup.