Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
It is easy to treat AI observability as a separate discipline with its own tools, and that framing produces a parallel telemetry stack that does not talk to the rest of your system. Almost everything from ordinary instrumentation carries over unchanged. Your agent makes HTTP calls that need CLIENT spans; it runs behind a request that needs context propagation; it writes metrics that will explode your cardinality if you label them by user; it hands work to queues; it needs a Collector for redaction and sampling. The genuinely new material is narrow — the gen_ai.* conventions, span topology for loops, retrieval provenance, and quality as an SLI. Everything else you already know, or can learn from the engineering telemetry course.
The carry-over list with the specific gotcha each one produces in agent code. The async-context item is the one that bites hardest, because agent frameworks are heavily async and a lost context silently splits every trace.
# ── Carries over unchanged, with the agent-specific gotcha ────────
# 1. SPAN KINDS still decide your service map.
# gotcha: an LLM provider call is a CLIENT span. People mark it
# INTERNAL and then wonder why "is it us or OpenAI?" is
# unanswerable and the provider is absent from the service map.
with tracer.start_as_current_span("chat gpt-4o",
kind=trace.SpanKind.CLIENT) as s:
s.set_attribute("server.address", "api.openai.com")
# 2. CONTEXT PROPAGATION still works the same way.
# gotcha: agent frameworks are async and fan out heavily. A
# parallel tool-call gather that loses context produces one root
# span per tool and the loop becomes invisible.
import asyncio, contextvars
async def call_tools_in_parallel(calls):
# asyncio.gather copies context correctly. A thread pool does
# NOT — and several frameworks use one for sync tools.
return await asyncio.gather(*[execute_tool(c) for c in calls])
def call_sync_tools(calls):
ctx = contextvars.copy_context() # inside the span
with ThreadPoolExecutor(4) as pool:
return list(pool.map(lambda c: ctx.run(execute_tool, c), calls))
# 3. CARDINALITY rules are unchanged and MORE dangerous here.
# gotcha: every tempting agent label is unbounded.
# conversation.id unbounded -> span attribute only
# prompt text unbounded -> never a label, ever
# tool arguments unbounded -> span attribute
# model name BOUNDED -> a good label
# tool name BOUNDED -> a good label
# finish reason BOUNDED -> a good label
# error type bounded IF normalised (engineering
# telemetry course, module 6)
# An agent generates far more distinct strings than a CRUD
# service, so the discipline matters more, not less.
# 4. STATUS POLICY still applies, with one addition.
# gotcha: a content filter refusing to answer is NOT a server
# error. Nor is a model declining. Both are the system working.
# Record them as events with a bounded reason; leave status UNSET.
if finish_reason == "content_filter":
span.add_event("gen_ai.refusal", {"reason": "content_filter"})
# status stays UNSET
# 5. THE COLLECTOR is the same Collector, doing more work here:
# prompt redaction, tail sampling on outcome, routing by tenant.
# 6. QUEUES: an agent that hands work to a worker crosses the same
# boundary with the same rules — link, do not parent.
# 7. LOGS still need trace_id injection, and an agent's logs are
# unusually valuable because the framework logs its own decisions.
# 8. EXEMPLARS: a p99 turn-latency spike should link to a trace.
# Same mechanism, and it works here too.
# ── Genuinely new in this course ────────────────────────────────
# gen_ai.* semantic conventions module 2
# span topology as a design problem module 4
# tool and MCP call telemetry module 5
# retrieval provenance at span level module 6
# streaming instrumented in app code module 7
# prompt capture done safely module 8
# cost DERIVED from span attributes module 9
# multi-agent handoff telemetry module 10
# quality as an SLI, with burn rate module 11
#
# Nine things. If you have not done the ordinary telemetry work
# first, do that first — this course assumes it, and instrumenting
# an agent well on top of a broken telemetry foundation does not
# work.python3 main.py