URL Discovery: Sitemaps, Links, and Fetch

Why this matters

Googlebot cannot crawl pages it does not know exist — URL discovery is the gate between your content and the index. Knowing the three discovery channels (sitemap submission, followed links, and manual fetch) lets you ensure every important page has a reliable path to being found.

Try it yourself

1. Navigate to Google Search Console → Sitemaps and verify your XML sitemap is submitted, has no errors, and shows a 'Discovered URLs' count close to your actual page count.
2. Pick a newly published page and check whether it appears as a linked destination in your internal link structure — use a site crawler like Screaming Frog free tier to confirm at least one internal link points to it.
3. Use the URL Inspection tool in Search Console to manually request indexing for that new page and record the response status and estimated timeline Google provides.

Prompt your AI

Use these three in order. Each builds on the one before.

1. Basics & terminology

In one paragraph, explain the three main ways Google discovers new URLs on the web.

2. Why it works (the mechanism)

Walk me through how a well-structured XML sitemap interacts with Googlebot's crawl queue — what happens after you submit it and how Google decides which URLs to prioritize.

3. Advanced — application & what's next

Given a news site that publishes 50 articles per day and needs each article indexed within an hour of publication, design the URL discovery strategy combining sitemaps, internal links, and Indexing API calls.

References

• Google Search Central — Sitemaps Overview