Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Here is a trap almost every team falls into once: your interaction logs only contain feedback on items your current system actually showed. If item X was never surfaced, you have zero signal on whether users would have loved it — not negative evidence, just missing evidence. Train a new model naively on this data and it learns to reproduce the exposure pattern of the old system, because that's the only distribution it ever saw examples from. This is selection bias, and it's the seed of the feedback loop problem you'll dig into properly in the final module. The practical takeaway now is: know which parts of your catalog are systematically under-exposed, and treat 'never shown' very differently from 'shown and ignored' when you build training data — otherwise your shiny new model just calcifies the old system's blind spots with better math.
The demo simulates a catalog where an old system only ever shows a fixed 20-item subset. A naive model trained only on the logged (shown, clicked) pairs is measured against the true underlying preferences over the full catalog, revealing how badly it misjudges the never-shown items.
logs, what fraction of the catalog has zero data points at all, and explain why 'zero data points' is a different case from 'observed and disliked.'import random
random.seed(5)
N_ITEMS = 100
TRUE_QUALITY = {i: random.random() for i in range(N_ITEMS)} # ground truth, unobservable
# Old system's exposure policy: only ever shows a fixed 20-item shortlist,
# regardless of true quality of the other 80 items.
EXPOSED = set(range(20))
def log_interaction(item):
if item not in EXPOSED:
return None # never shown -> no data point at all, not a negative
# Clicked with probability proportional to true quality
return random.random() < TRUE_QUALITY[item]
logs = [(i, log_interaction(i)) for i in range(N_ITEMS)]
observed = [(i, c) for i, c in logs if c is not None]
# A naive model that only sees observed rows "learns" the average click
# rate among *exposed* items and has literally no signal on the other 80.
observed_click_rate = sum(c for _, c in observed) / len(observed)
true_avg_quality_all = sum(TRUE_QUALITY.values()) / N_ITEMS
true_avg_quality_unexposed = sum(TRUE_QUALITY[i] for i in range(N_ITEMS) if i not in EXPOSED) / (N_ITEMS - len(EXPOSED))
print(f"items ever exposed: {len(EXPOSED)} / {N_ITEMS}")
print(f"observed click rate (exposed): {observed_click_rate:.3f}")
print(f"true avg quality, unexposed 80: {true_avg_quality_unexposed:.3f}")
print(f"true avg quality, whole catalog: {true_avg_quality_all:.3f}")
# A model trained only on 'logs' has zero examples from 80% of the catalog —
# it cannot know whether those items are better or worse than what it's seen.python3 main.py