The Index and What Gets Excluded

Why this matters

Not every URL Google crawls ends up in the index — pages can be excluded for dozens of reasons, and most site owners do not know it is happening. Understanding what triggers exclusion (noindex, soft 404, duplicate content, low quality signals) gives you a diagnostic framework for pages that should rank but do not.

Try it yourself

1. In Google Search Console, open Index → Pages and look at the 'Why pages aren't indexed' breakdown — identify the top three exclusion reasons for your site and write down what each one means.
2. Pick one URL listed under 'Excluded' and run it through the URL Inspection tool to see Google's live assessment — note whether the exclusion reason is a tag, a server response, or a quality signal.
3. Check five of your lowest-traffic blog posts for a noindex meta tag using curl https://yourpost | grep 'noindex' — confirm none were accidentally excluded by a plugin or CMS setting.

Prompt your AI

Use these three in order. Each builds on the one before.

1. Basics & terminology

In one paragraph, explain the difference between being crawled and being indexed, and why a page can be one without the other.

2. Why it works (the mechanism)

Walk me through the main categories of reasons Google would crawl a URL but not add it to the index, from technical blocks to quality assessments.

3. Advanced — application & what's next

Given an e-commerce site with 10,000 product pages, 4,000 of which are flagged as 'Crawled — currently not indexed' in Search Console, design a systematic triage process to identify which exclusion causes to fix first.

References

• Google Search Central — Reasons a Page May Not Be Indexed