Website Page Discovery for Scraping: Build the Inventory First

Discovery is an inventory problem, not an extraction detail

A scraper can have excellent selectors and still produce incomplete data. The usual cause is not a parsing bug: it is an incomplete understanding of which URLs exist, which are in scope, and which actually represent useful documents.

Treat website page discovery for scraping as a separate phase that produces a durable inventory. Extraction begins only after you can answer a few operational questions:

• Which URLs did we observe, and from which source?
• Which URLs are allowed by our crawl policy?
• Which ones resolve to a useful response rather than a login screen, download, error, or action endpoint?
• Which URLs are likely duplicates of the same resource?
• Which important URL families have weak or no internal-link coverage?

This changes the shape of a scraping project. Rather than starting with three visible category pages and expanding the crawler only when someone notices missing data, you first build evidence for crawl scope.

Search crawlers discover URLs through more than one channel, including links, sitemaps, and redirects. That is a useful model for scraper design as well: gather candidates from several channels, then apply your own access and relevance rules to the union. Google’s developer guide describes these discovery mechanisms and the value of pages being reachable from other findable pages.

Define the URL record before collecting URLs

A plain set of strings is not enough. Discovery sources disagree, redirects alter destinations, and normalization can accidentally erase distinctions that matter to a server. Use a record that retains the observation alongside later decisions.

A practical schema might look like this:

UrlRecord
  observed_url: string
  comparison_key: string
  source: html_link | sitemap | redirect | rendered_link | manual_seed
  source_page: string | null
  discovered_at: timestamp
  robots_decision: allowed | disallowed | unknown
  fetch_status: integer | null
  final_url: string | null
  redirect_chain: list[string]
  content_type: string | null
  classification: pending | content | duplicate | excluded | error | review
  notes: list[string]

The key design choice is keeping both observed_url and comparison_key. The observed value is evidence: it tells you exactly what a page linked to or what a sitemap declared. The comparison key supports queue deduplication, but should not become an irreversible rewrite of the original.

For example, you may decide that host-name casing and a default port do not create a new crawl target. But do not automatically discard every query parameter, reorder parameters, or collapse trailing slashes without testing the site. Those transformations can change what the server returns.

Start with access constraints, then use several discovery channels

Fetch and inspect the applicable robots.txt file before scheduling a crawl. The Robots Exclusion Protocol is a way for a site to communicate crawler access rules, and its scope is tied to the host, protocol, and port where it is served. It may also advertise sitemap locations. It is not an authorization system, and it is not a complete inventory of public URLs. See RFC 9309 for the protocol details.

Next, collect candidates from independent channels. Each one compensates for blind spots in the others.

1. Crawl ordinary document links

Begin with stable seed pages: a home page, section indexes, known category pages, documentation roots, or explicitly supplied URLs. Extract links from the raw response and record the source page for each edge.

Ordinary anchor links are the most dependable starting point because they create a visible document graph. Google similarly recommends standard links for crawlability and notes that links available in the initial response can be found sooner than links added later by script. Its link guidance is a useful reminder that a page’s visual navigation and its immediately crawlable navigation are not always the same.

Apply scope rules at enqueue time, not after downloading a large number of irrelevant pages. Typical rules include:

• allowed hosts and path prefixes;
• explicit exclusions for account, checkout, logout, search, or administrative paths;
• accepted media types;
• query-parameter budgets for faceted navigation;
• a maximum depth or per-pattern page limit;
• a rule never to trigger state-changing operations.

The goal is not to make the crawler maximally aggressive. It is to make its boundary explicit and reviewable.

2. Parse sitemap indexes and URL sitemaps

Sitemaps are often the fastest way to reveal a site’s intended content inventory, especially when deep content is lightly linked from navigation. They can also expose pages that have been omitted from an internal-link traversal.

Sitemap entries should be treated as candidates, not automatic truths. Google recommends absolute, generally canonical URLs in sitemaps, but also characterizes sitemap submission as a hint rather than a guarantee. Its sitemap documentation supports both parts of that approach.

For every sitemap URL, retain the sitemap source, apply your scope policy, and fetch it like any other candidate. It may redirect, fail, return an unexpected media type, or no longer be relevant to the dataset you intend to collect.

3. Promote in-scope redirect targets into the inventory

A redirect response supplies useful evidence. Record the whole chain, preserve the initial URL, and add an in-scope final target as a candidate if it is not already known. Do not silently replace one with the other: the source URL may be an old alias worth tracking, while the destination may reveal the current URL convention.

Redirect behavior is also one signal among several when reasoning about duplicates. Google’s canonicalization guidance distinguishes redirects, canonical annotations, and sitemap inclusion rather than treating them as identical evidence. Your scraper should do the same: collect signals first, then make a documented merge decision.

4. Model pagination as edges between documents

Pagination is where seemingly complete scrapers commonly stop after page one. Treat each page in a sequence as a URL node and the next/previous links as graph edges. The inventory should show both the individual pages and the relationship between them.

Do not assume that a visible “Load more” control yields crawlable URLs. Crawlers generally do not perform browser interactions, and pages identified only by URL fragments are not a dependable way to represent distinct content. Google recommends unique URLs and sequential crawlable links for paginated content in its pagination guidance.

When you find a page sequence, verify it empirically:

1. Fetch several early, middle, and late pages.
2. Confirm that the item set changes as expected.
3. Check whether the final page loops, redirects, or becomes empty.
4. Record the observed end condition rather than relying on a guessed page count.

This avoids generating thousands of speculative URLs such as ?page=99999.

5. Compare raw and rendered link sets when necessary

A raw HTTP response is usually the right first pass: it is faster, simpler to debug, and shows what a non-rendering client receives. But a JavaScript-heavy site may only add route links after scripts run.

Use rendering as a targeted second pass for representative templates or URL families where the raw response appears incomplete. Capture links from both forms and calculate the difference:

render_only_links = rendered_links - raw_response_links

Those render-only links identify routes that need additional handling. They do not automatically mean every page requires rendering. Google documents a crawl-render-index process in which rendering can reveal additional links, while also noting that JavaScript execution is not universal among crawlers. Its JavaScript guidance provides the underlying model.

Normalize cautiously and classify aggressively

After the initial collection, you will have a noisy candidate set. This is the point to normalize for comparison, validate responses, and classify records—not the point to begin page-specific extraction.

A conservative sequence is:

1. Resolve relative references against the source page.
2. Remove fragments for fetch identity, while retaining the original observed URL as metadata.
3. Apply only well-understood comparison normalization.
4. Fetch permitted candidates with rate limits and retry rules appropriate to the target.
5. Record status, content type, final URL, redirect chain, and a lightweight content signature.
6. Classify each response.

Useful classifications include:

• content: in scope and a likely extraction target;
• duplicate candidate: same or near-identical resource reached through another URL;
• excluded: outside policy, disallowed, or a known non-content route;
• error: unavailable or invalid response;
• review: ambiguous route that needs a human decision.

Do not merge URLs just because their paths look similar. A canonical annotation, a redirect, a sitemap listing, and matching content are different observations. Store the reasons for a merge so you can revisit the decision if a later fetch contradicts it.

Find gaps with coverage reports

The most valuable output of discovery is not merely a queue. It is a set of reports that make omissions visible.

Build at least these views:

• candidate counts by source and URL pattern;
• fetched, excluded, errored, and pending counts;
• response type and status distributions;
• redirect chains grouped by destination;
• pagination sequences and detected endpoints;
• URLs appearing in sitemaps but not reached through internal links;
• URL families with a high number of parameter variants.

A sitemap URL or externally supplied URL with no inbound edge in your observed site graph is a useful orphan-page candidate. That is an implementation heuristic, not proof that the page is truly orphaned: your crawl may have missed a source link, and some links may be inaccessible under your scope rules. Still, comparing inventories is a direct way to prioritize investigation. The rationale follows from the separate roles of links and sitemaps described in Google’s developer guide.

Here is a compact outline of the workflow:

inventory = seed_records()

inventory.add_all(parse_robots_for_sitemaps())
inventory.add_all(parse_sitemaps())

for record in inventory.pending():
    if not allowed_by_policy(record.observed_url):
        record.classification = "excluded"
        continue

    response = fetch(record.observed_url)
    record.store_response_metadata(response)

    if response.is_redirect:
        inventory.add_redirect_target(response.final_url, record)

    if response.is_html:
        inventory.add_links(extract_raw_links(response), source=record)

for template in templates_needing_render_check(inventory):
    inventory.add_links(compare_rendered_and_raw_links(template))

classify_and_report(inventory)

The critical property is provenance. Every queued URL should be traceable to a sitemap, source page, redirect, or seed. Every exclusion should be attributable to a policy rule. That makes crawl scope explainable when a downstream dataset has a surprising gap.

Ready to validate seed pages before building a larger inventory? Create a PagePith account at Sign up.

Tooling pattern: separate discovery from extraction

You can implement this workflow in any HTTP and parsing stack. The architectural separation matters more than the framework:

• Discovery job: collects and validates candidates, stores graph edges and metadata, and produces reports.
• Extraction job: consumes only records classified as content and applies template-specific parsing.

For a Scrapy implementation, this separation maps cleanly to its provided spider types. CrawlSpider supports link-following rules built with LinkExtractor, while SitemapSpider supports sitemap URLs, nested sitemaps, sitemap discovery from robots.txt, and sitemap-oriented rules. The relevant capabilities are documented in the Scrapy spiders documentation.

You do not need to choose only one. Run a sitemap-oriented pass and a graph-oriented pass, write both results into the same inventory, and compare the sets before enabling expensive extraction callbacks. This is often clearer than mixing traversal decisions and record parsing inside one callback.

A limited PagePith demonstration

The supplied PagePith proof shows a fetch request for Google’s SEO Guide for Web Developers. For that request, the recorded result includes:

• the requested URL;
• the page title, SEO Guide for Web Developers | Google Search Central | Documentation | Google for Developers;
• the fetch tier;
• a content length of 9412;
• a Markdown excerpt beginning with the guide’s “Get started with Search” section.

That is a useful seed-page retrieval outcome: it provides a title and Markdown-formatted content that can be stored beside a URL record for inspection or subsequent parsing.

The proof does not establish that PagePith parses sitemaps, extracts every link, renders JavaScript, enforces robots rules, crawls a site, or retrieves every URL successfully. Those steps should remain explicit parts of your discovery design unless you have separate evidence for them.

Finish discovery with a crawl contract

Before writing selectors, save a short crawl contract in the repository. It should name the allowed hosts and paths, prohibited routes, source channels, parameter rules, rendering triggers, validation fields, and the definition of a successful content record.

That document turns “scrape the site” into a testable plan. More importantly, it lets you distinguish a parsing failure from a discovery failure. When a record is missing, you can determine whether it was never linked, only present in a sitemap, blocked by policy, hidden behind rendering, redirected elsewhere, or excluded deliberately.

Build that inventory first, then write extraction logic against a known and defensible set of pages.

Want a retrieval starting point for your discovery workflow? Sign up for PagePith.