Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
The engineering telemetry course starts by writing down the questions asked during real incidents, and the same discipline applies here with a different question set. For an agent the questions are not 'which service was slow' but 'why did it say that', 'was it the retrieval or the model', 'did this get worse after we changed the prompt', 'which customers saw the bad version', and 'how much did that cost us'. Each requires something specific to have been recorded at the time, and several require recording things people are nervous about — prompts, retrieved content, model versions. Deciding in advance is how you avoid discovering the gap during the incident.
The five questions with what each requires. The last column is the honest one: most of these are missing in most systems, and the ordering of the rest of this course follows from the list.
# agent-telemetry-questions.md — before any SDK is configured
## Q1. "Why did it say that?"
# asked by: support, after a user complaint
# requires: the exact prompt sent (post-template, post-retrieval),
# the exact completion, the model VERSION that answered,
# and the tool results that were in context
# note: this is the one people flinch at, because it means
# storing prompts. Module 8 is about doing it safely
# rather than not doing it.
# today: MISSING / partial
## Q2. "Was it retrieval or the model?"
# asked by: whoever has to fix it
# requires: per-chunk retrieval scores, WHICH chunks made it into
# the prompt after reranking and truncation, and the
# answer's citations
# why it matters: these are two different teams' work and two
# completely different fixes
# today: MISSING — the retrieve span usually records a count
## Q3. "Did it get worse after the prompt change?"
# asked by: the person who changed the prompt
# requires: a prompt VERSION identifier on every span, plus a
# quality metric sliceable by it
# note: requires that prompts are versioned artefacts.
# agentic-applied-ai/13 module 9 covers the prompt
# platform; this course covers the telemetry side.
# today: MISSING in most systems
## Q4. "Which customers saw the bad version?"
# asked by: the account manager, urgently
# requires: tenant.id on the turn span, correlatable with the
# model version, prompt version and time window
# today: partial — tenant is often there, version is not
## Q5. "How much did that cost, and what is our margin per turn?"
# asked by: finance, eventually and then repeatedly
# requires: input and output tokens per call, model pricing at
# the time, and a per-turn rollup
# note: cost DASHBOARDS are covered elsewhere in the
# catalogue. What is missing is deriving it from span
# attributes so it agrees with the trace. -> module 9
# today: partial — a total exists; per-turn attribution does not
# ── What the answers require, deduplicated ──────────────────────
# gen_ai.request.model + gen_ai.response.model -> mod 2
# prompt and completion, redacted and sampled -> mod 8
# prompt.version, agent.version -> mod 2, 11
# per-chunk retrieval scores and post-rerank set -> mod 6
# tool call arguments and results -> mod 5
# input/output tokens per call -> mod 2, 9
# tenant.id on the turn -> mod 2
# a sampled quality score -> mod 11
#
# Eight things. Six of them are not in a default setup. That list,
# in that order, is the rest of this course.
# ── The discipline ──────────────────────────────────────────────
# Write YOUR version from YOUR complaints. If nobody has ever asked
# Q3 in your organisation, do not instrument for it yet. If someone
# asks a sixth question you cannot answer, that is the next module
# you actually need — and it may not be one of these.python3 main.py