Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
If you already use an LLM observability product you have prompt and completion capture, a trace view, token counts and probably a cost estimate — and it took an afternoon to set up. It is worth being clear about what that leaves out, because the gaps are the reason this course exists. Vendor SDKs generally do not join their traces to the rest of your distributed trace, so the database query and the upstream API call that share the request are in a different system. They rarely emit metrics you can put in an SLO. Their sampling and retention are theirs. And their instrumentation is not portable, so a vendor change is a re-instrumentation project.
An honest inventory. The recommendation at the end is not to rip anything out: it is to make the standard the substrate and let the vendor be a destination, which keeps what you have and closes the gaps.
# ── What a vendor LLM SDK typically gives you ────────────────────
# [x] prompt and completion capture, with a diffable UI
# [x] a per-call trace view with nesting
# [x] token counts and a cost estimate
# [x] latency per call
# [x] dataset and eval tooling attached to the same traces
# [x] a working setup in an afternoon
#
# This is real value. Do not throw it away.
# ── What it typically does NOT give you ─────────────────────────
# [ ] JOIN to the rest of your distributed trace
# Your agent called Postgres and an internal pricing API in
# the same request. Those spans are in your APM. The LLM
# calls are in the vendor. Nothing connects them, so "why
# was this turn slow" needs two tools and a timestamp guess.
#
# [ ] metrics you can put in an SLO
# Most give you dashboards, not a metrics endpoint your
# alerting stack can query. So your agent's error budget
# lives in a different system from every other service's.
#
# [ ] control over sampling and retention
# Their rules, their retention, their bill.
#
# [ ] portability
# The instrumentation is the expensive, long-lived asset
# (engineering telemetry course, module 1). A vendor SDK
# makes it disposable.
#
# [ ] redaction you can prove
# Prompt capture is on by default in several. If a card
# number is in a prompt, it is in their store. Proving to an
# auditor what is redacted, where, is much harder when the
# redaction is a checkbox in someone else's product.
# ── The gap that matters most, concretely ───────────────────────
# A slow turn. Where did the 4.2 seconds go?
#
# With a vendor SDK only:
# vendor UI: 3 LLM calls, 1.9s total.
# the other 2.3s: unaccounted. Not in the vendor's view at all.
#
# With OTel and one trace:
# invoke_agent 4,210ms
# ├─ SELECT user_preferences 890ms <- there it is
# ├─ chat gpt-4o 410ms
# ├─ execute_tool search_docs 1,180ms
# │ └─ POST /v1/embeddings 140ms
# │ └─ vector_search collection=docs 980ms <- and here
# ├─ chat gpt-4o 520ms
# └─ chat gpt-4o 980ms
#
# The two biggest contributors are a database query and a vector
# search — neither of which is an LLM call, and neither of which
# the LLM-specific tool can see.
# ── The recommendation: substrate, then destination ─────────────
# 1. Instrument with OpenTelemetry and gen_ai.* conventions. This
# is the part you own and the part that lasts.
# 2. Export OTLP to the Collector.
# 3. Fan out from there: to your existing trace backend AND to the
# LLM vendor, which accepts OTLP (module 12).
#
# You keep the vendor's UI and eval tooling, gain the join to the
# rest of your system, get metrics into your own alerting, and can
# change vendors with a YAML edit.
#
# If your vendor SDK is already deeply wired in, module 12 covers
# running both during a migration.python3 main.py