Capstok — learn by doing

Why this matters

The famous 'Hidden Technical Debt in Machine Learning Systems' paper made a claim that lands harder every year: only a tiny fraction of a real ML system is the ML code — the rest is plumbing, configuration, data dependencies, and glue that quietly accrues debt. Entanglement (changing anything changes everything, the CACE principle), undeclared consumers, data dependencies that no one owns, and pipeline jungles are the specific debts that make ML systems brittle. Recognizing these anti-patterns by name lets you avoid building them, and the whole point of MLOps tooling is to pay down or prevent exactly this debt. This is the conceptual backbone the rest of the course operationalizes.

Demo

The demo visualizes the 'CACE principle' (Changing Anything Changes Everything): a small tweak to one feature's preprocessing ripples through every downstream model, illustrating why ML systems resist isolated changes.

Try it yourself

List the downstream consumers of one feature or model output in your system — can you name all of them?
Find one 'undeclared consumer': something reading an output that its owner doesn't know about.
Run change_feature on a shared feature and note how many models need re-eval, not one.
Identify one 'pipeline jungle' or glue-code path in your stack that no single person fully understands.

Prompt your AI

Use these three in order. Each builds on the one before.

1. Basics & terminology

In one paragraph, explain what 'technical debt in ML systems' means and why it's worse than normal software debt.

2. Why it works (the mechanism)

Walk me through the specific ML debt anti-patterns — entanglement/CACE, undeclared consumers, data dependencies, glue code, pipeline jungles — and how each arises.

3. Advanced — application & what's next

Given my ML/LLM system, help me audit it for the hidden-technical-debt anti-patterns and prioritize which to pay down first.

References

Chat about this lesson

# CACE: Changing Anything Changes Everything.
# One feature's scaling change silently shifts every model that consumes it.
feature_consumers = {
    "user_age": ["churn_model", "ltv_model", "fraud_model"],
    "session_count": ["churn_model", "recommender"],
}
def change_feature(feature):
    affected = feature_consumers.get(feature, [])
    return f"Editing '{feature}' may invalidate: {affected} -> all need re-eval, not just one."
print(change_feature("user_age"))

# Undeclared consumers are worse: a model reads an output you didn't know was a dependency.
declared = {"recommender"}
actual_readers = {"recommender", "ads_ranker"}   # ads_ranker started reading silently
print("UNDECLARED consumer (will break on change):", actual_readers - declared)

Run: python3 main.py

Technical debt in ML systems