Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Before instrumenting anything it is worth writing down what you are trying to catch, because the list is longer and stranger than for ordinary software. An agent can loop until its budget is gone, call a tool with malformed arguments, retrieve the wrong documents and answer confidently from them, retrieve the right documents and ignore them, hand off to a sub-agent that loses the original intent, be prompt-injected by its own retrieved content, degrade silently when a provider swaps a model version, or simply produce a fluent answer with no basis. Each has a different signature in telemetry, and knowing the signature is what turns instrumentation from decoration into detection.
The taxonomy with a telemetry signature for each. The column that matters is the last one: for most of these, the signal that would catch it is not something a normal APM emits, which is why this is a course rather than a configuration change.
# ── The taxonomy, with the signal that catches each ──────────────
# 1. RUNAWAY LOOP
# what: the model keeps calling tools and never concludes
# signature: step count in the tail; total tokens climbing while
# the answer never arrives; the same tool called with
# near-identical arguments repeatedly
# signal: gen_ai.agent.step_count histogram + a hard ceiling
# today: usually invisible until the bill arrives -> mod 4
# 2. MALFORMED TOOL CALL
# what: the model produces arguments that fail schema validation
# signature: an execute_tool span with an error and a validation
# error type, followed by a retry
# signal: tool error taxonomy as a bounded attribute -> mod 5
# today: often swallowed by a retry and never recorded
# 3. RETRIEVAL MISS, CONFIDENT ANSWER
# what: the wrong documents come back and the model answers anyway
# signature: low retrieval scores, then a long confident output
# with citations that do not support it
# signal: per-chunk scores on the retrieve span + which chunks
# actually reached the prompt -> mod 6
# today: invisible: the trace shows a successful retrieval
# 4. RETRIEVAL HIT, IGNORED CONTEXT
# what: the right documents come back and the model ignores them
# signature: high retrieval scores; output that contradicts or
# omits the retrieved content
# signal: a groundedness score on a sample of outputs -> mod 11
# today: invisible
# 5. HANDOFF INTENT LOSS
# what: a sub-agent receives a summarised task and solves the
# wrong problem
# signature: the handoff payload differs materially from the
# original request; the sub-agent's output does not
# address the user's question
# signal: handoff spans carrying the task delta -> mod 10
# today: invisible; the sub-agent's trace looks fine
# 6. INDIRECT PROMPT INJECTION
# what: retrieved content contains instructions the model follows
# signature: a tool call the user's request cannot explain; an
# output that references content nobody asked about
# signal: tool calls attributed to the turn that requested them,
# plus retrieved-content provenance -> mod 5 and 6
# today: invisible. Note: llm-ai-security covers the ATTACK;
# this course covers DETECTING it in telemetry.
# 7. SILENT MODEL DRIFT
# what: the provider changes what "gpt-4o" points at
# signature: gen_ai.response.model changes while
# gen_ai.request.model stays the same; output length,
# token ratio and step count distributions shift
# signal: RESPONSE model as a distinct attribute -> mod 2
# today: almost nobody records the response model, which is why
# this is discovered from user complaints
# 8. FLUENT AND BASELESS
# what: a confident answer with no support anywhere
# signature: none in conventional telemetry. Zero errors, normal
# latency, normal cost.
# signal: a sampled quality SLI, and a burn-rate alert -> mod 11
# today: found by users
# ── The uncomfortable summary ────────────────────────────────────
# Of eight failure modes, ordinary telemetry catches roughly one
# (2, sometimes). Three are visible only with span-level detail
# nobody emits, and four require evaluating the OUTPUT, which means
# telemetry alone is insufficient and you need a quality signal
# flowing back into the same pipeline.
#
# Write your own version of this list for your own agent, ordered
# by how often it has actually happened. That ordering, not this
# one, decides what you instrument first.python3 main.py