Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Every one of these projects fails the same way: someone points it at a folder of unrelated blog posts and discovers in week three that there is nothing to extract, nothing to compare, and nothing to measure. Real data has structure — filings reference other filings, wiki pages link, papers cite, product docs have dependency chains — and that structure is what the project is actually about. It also has mess: duplicates, near-duplicates, inconsistent naming, missing fields. The mess is not an obstacle to the project, it is most of the work, and handling it is what separates your repository from the thousand that used the same tutorial dataset.
Before committing to a corpus, run this check. If the entity graph is nearly edgeless, or every document is independent, pick something else — you will not recover later.
# corpus_smell_test.py — run before you commit three weeks to a dataset
import collections, itertools, re
def smell_test(docs: list[str]) -> dict:
"""Cheap signals that a corpus has extractable structure."""
# 1. Do documents refer to shared proper nouns?
caps = [set(re.findall(r"\b[A-Z][a-zA-Z]{2,}\b", d)) for d in docs]
shared = collections.Counter()
for a, b in itertools.combinations(range(len(caps)), 2):
if caps[a] & caps[b]:
shared[a] += 1
shared[b] += 1
connected = sum(1 for d in range(len(docs)) if shared[d] > 0) / max(len(docs), 1)
# 2. Is there vocabulary depth, or is every doc about the same thing?
vocab = collections.Counter(w.lower() for d in docs for w in re.findall(r"\w+", d))
depth = sum(1 for _, n in vocab.items() if n > 2) / max(len(vocab), 1)
return {
"docs": len(docs),
"pct_docs_sharing_an_entity": round(connected * 100, 1), # want > 60
"vocab_depth": round(depth, 3),
"verdict": "usable" if connected > 0.6 else "TOO SPARSE — pick another corpus",
}
# Good candidates: SEC filings, an internal wiki, arXiv papers with citations,
# product docs with dependency chains, your own company's runbooks.
# Bad candidate: 200 unrelated blog posts.python3 main.py