How to Build a Crawl Budget for Large Websites Without Missing High-Value Pages

A large scraper rarely fails because it cannot make enough HTTP requests. It fails because it spends requests in the wrong places: faceted URLs, tracking variants, stale pages, redirect chains, retries, and unchanged documents.

The remedy is an internal crawl budget: an explicit policy that decides which URLs to fetch, when to revisit them, and when to stop. The goal is not maximum request volume. It is maximum useful, timely coverage within constraints imposed by your infrastructure and the target host.

Google uses “crawl budget” to describe the URLs that Googlebot can and wants to crawl, combining crawl capacity with crawl demand. That definition is specific to Googlebot, but it is a helpful model for scraper design: capacity depends on conditions such as latency, concurrency, errors, and rate limiting; demand reflects the relative value and freshness of URLs. Google’s crawl budget guidance describes this distinction in detail.

Define the budget as capacity plus demand

Start by making two quantities explicit.

Fetch capacity is the work your system can safely perform over a time window. Measure it per host rather than treating it as one global number. It includes:

• maximum concurrent requests
• request delay or rate ceiling
• connection and response latency
• retry allowance
• response-size and bandwidth limits
• 429 and 5xx error thresholds
• rules from robots.txt, permissions, terms, and internal policy

Fetch demand is the set of candidate URLs competing for that capacity. A URL earns demand when it has a meaningful chance of producing useful new data, an important change, or a needed validation result.

This framing prevents a common error: setting a blanket target such as “one million URLs per day.” A million requests may be too aggressive for one host and wasteful for another. More importantly, the target says nothing about whether you captured the pages that matter.

A useful operational definition is:

crawl budget = host-safe fetch capacity × time window
allocation = ranked candidate URLs selected within that budget

The first term protects hosts and your own workers. The second protects product value.

Build an inventory before fetching page bodies

Do not let link depth become your only discovery strategy. First create a durable candidate inventory from sources such as:

• XML sitemaps and sitemap indexes
• feeds and published update endpoints
• known database exports or partner-provided URL lists
• internal links discovered during prior crawls
• pages already stored in your crawl state
• redirect targets and canonical URLs observed in responses

For each candidate, record enough metadata to make an initial decision without immediately downloading its body:

url
normalized_url
host
url_class
source
first_seen_at
last_seen_at
last_fetch_at
last_changed_at
http_status
etag
last_modified
canonical_url
content_hash
estimated_value
estimated_change_rate
estimated_cost

Classification is essential. Separate product or article pages from category listings, paginated archives, internal search results, faceted navigation, assets, account paths, and URLs containing session-like or tracking parameters. Google’s guidance on crawling errors identifies duplicate URL variants, parameterized navigation, infinite spaces, and low-value pages as classes that can consume crawl resources with little return. Review the URL-management guidance here.

The inventory is not a one-time spreadsheet. It is the stateful input to your scheduler. A URL can move between classes as evidence changes: a category page may become a valuable discovery source, while a supposedly canonical item page may start redirecting permanently.

Normalize and deduplicate before scheduling

A scheduler cannot prioritize well when ten variants represent one resource. Normalize before inserting a candidate into the queue.

Typical normalization rules include:

1. Lowercase the hostname and remove the fragment identifier.
2. Resolve relative links against the response URL.
3. Apply a documented policy for trailing slashes, default ports, and query-parameter ordering.
4. Remove known non-content tracking parameters where that is appropriate for the target.
5. Treat redirects as observations and update the preferred destination after validation.
6. Store declared canonical URLs as signals, not unquestioned truth.
7. Deduplicate by both normalized URL and content fingerprint when duplicate content is likely.

Be conservative with parameter removal. A parameter named color may represent a useless filter on one site and a unique product configuration on another. Prefer allowlists and per-site rules over a universal regex.

Scrapy provides duplicate filtering and priority-aware scheduling, but your strongest deduplication logic generally belongs in a shared URL policy layer. Its scheduler documentation also makes clear that queue order is configurable; the default behavior need not dictate your crawl strategy. See Scrapy’s scheduler model.

Score URLs by expected utility, not link depth

Depth is useful as a discovery signal, but it is weak as a decision rule. A deeply linked record may be more valuable than a shallow tag archive. Assign a score that reflects your actual extraction objective.

Here is a practical starting formula:

score(url) =
  0.35 × value
+ 0.25 × freshness_need
+ 0.20 × change_probability
+ 0.10 × discovery_value
- 0.10 × fetch_cost
- risk_penalty

Normalize each positive feature to a common range, such as 0 to 1. The precise weights are less important than documenting them, logging the feature values, and revising them using results.

Inputs to the score

Value reflects the importance of the page if it yields data. Examples include an item in a priority catalog, an authoritative documentation page, or a listing tied to a downstream alert.

Freshness need rises as a page approaches its acceptable staleness limit. A page that must be checked daily should eventually outrank a page with a 30-day service-level objective, even if the latter has a higher base value.

Change probability is an estimate learned from previous observations. A page that changed in six of its last eight checks should be revisited more often than a stable policy page. Accurate sitemap lastmod values can be an additional signal, but not a command. Google treats sitemaps as hints and notes that its priority and changefreq fields are not used for scheduling decisions. Its sitemap documentation explains these limits.

Discovery value rewards pages that reliably reveal unseen, eligible URLs. A paginated listing might have modest extractable value but high discovery value.

Fetch cost includes recent latency, typical response bytes, rendering cost if applicable, and retry history. It is an engineering estimate, not a reason to exclude expensive pages automatically. A high-value slow page can still win; it simply competes honestly for constrained capacity.

Risk penalty should sharply demote URLs that are disallowed by your crawler policy, produce repeated errors, resemble infinite navigation spaces, or exceed retry limits.

Build a queue you can inspect, not merely tune. If you are evaluating a fetch layer for a prioritized workflow, create a PagePith account and test it against a representative set of allowed URLs. Compare returned content and failures with your own queue metadata before changing production allocation rules.

Use separate queues for different jobs

A single priority queue makes it easy for a large discovery wave to starve critical recrawls. Use logical queues with reserved capacity instead:

Queue	Purpose	Example allocation
Critical recrawl	High-value pages nearing freshness limits	Reserved first
New discovery	Sitemap, feed, and link candidates	Fixed share
Validation	Redirect, canonical, and metadata checks	Small steady share
Backfill	Lower-priority historical coverage	Uses spare capacity
Retry	Temporary failures with backoff	Strictly capped

The allocations need not be permanent. If the critical queue is empty, lend its unused capacity to discovery or backfill. If a host begins returning 429 responses, reduce all queues for that host rather than allowing retries to consume the remaining budget.

This design follows a useful separation of concerns: scheduling selects work, downloading performs network activity, and result processing updates future decisions. Scrapy’s architecture describes those roles separately, which is a useful reference even if you build your own workers. Read the architecture overview.

Enforce host-aware limits and respectful access

Global concurrency is not a politeness policy. A crawler running at 100 requests per second may be harmless across thousands of hosts but disruptive when most work lands on one origin.

Maintain a per-host or per-download-slot controller with:

• current concurrency
• delay between starts
• recent median and tail latency
• 429, 5xx, timeout, and connection-error rates
• active retry count
• a circuit-breaker state

Use additive increases cautiously and reduce concurrency or increase delay quickly when error signals appear. Scrapy’s AutoThrottle is one established example of adaptive behavior: it adjusts delay using response latency and target concurrency, and it does not use non-200 responses to make the crawler speed up. The AutoThrottle documentation describes the calculation.

Check applicable robots.txt rules before scheduling a URL. The Robots Exclusion Protocol is a request for crawler behavior, not an authorization system; it does not grant access to private material. Authentication, contractual permissions, terms, and legal review remain separate concerns. RFC 9309 specifies this distinction.

Revalidate pages without repeatedly downloading bodies

A revisit does not always require a full transfer. Persist ETag and Last-Modified headers where available, then send conditional headers on eligible recrawls:

If-None-Match: "stored-etag"
If-Modified-Since: Tue, 14 Oct 2026 08:15:00 GMT

If the representation is unchanged, the server can respond with 304 Not Modified and omit the resource body. Conditional requests can reduce transfer and processing work, although they still consume a request and must remain inside your host budget. MDN’s conditional-request guide covers the relevant validators and response behavior.

Treat a 304 as useful evidence: it updates your observation of stability and can reduce that URL’s near-term change probability. Conversely, a meaningful content hash change should increase the likelihood of a prompt future revisit until the page’s pattern is clearer.

A concrete scheduling loop

Run this loop continuously or in bounded batches:

while budget.has_capacity():
    candidates = inventory.eligible(now)
    candidates = normalize_and_deduplicate(candidates)
    candidates = apply_robots_and_policy_rules(candidates)

    for url in candidates:
        url.score = score(url, crawl_history, now)

    url = choose_highest_score_with_available_host_slot(candidates)
    if url is None:
        break

    response = fetch(url, conditional_headers=url.validators)
    outcome = process(response)

    inventory.record(outcome)
    host_controller.update(url.host, outcome)
    reschedule(url, outcome, now)

The details matter. eligible(now) should exclude URLs that are not due, blocked by policy, or currently in flight. choose_highest_score_with_available_host_slot() should be queue-aware, so a vast backfill collection cannot monopolize the worker pool. reschedule() should use observed outcomes, not fixed intervals alone.

For example, after three consecutive unchanged validations, double a low-risk page’s revisit interval up to a defined maximum. After a meaningful change, shorten the interval temporarily. After repeated errors, apply exponential backoff and eventually require manual review rather than retrying forever.

Measure coverage and yield, not raw throughput

A crawler reporting “10 million requests completed” may be performing poorly. Pair volume with efficiency and freshness measures:

• high-value URL coverage: eligible priority pages fetched within their target window
• freshness lag: time since the last successful observation of each important page
• useful records per 1,000 requests
• duplicate or canonical-collapse rate
• bytes transferred per useful record
• 304 rate for revalidation traffic
• median and p95 latency by host
• timeout, 429, and 5xx rates by host
• retry share of total requests
• discovery-to-useful-page conversion rate

Review these metrics per host and per URL class. A high duplicate rate in one class indicates an inventory or normalization problem; a high 304 rate may indicate that recrawl intervals are too short for stable pages; rising 429s mean the capacity model is too optimistic.

PagePith demonstration: validate retrieval separately from scheduling

The supplied PagePith proof shows a fetch request for Google’s crawl-budget documentation at https://developers.google.com/crawling/docs/crawl-budget. It returned the page title, a Markdown excerpt beginning with navigation and content links, and a reported content length of 10,684.

That is a useful narrow demonstration of retrieval: a requested documentation URL produced structured content that could be stored alongside crawl metadata or passed to an extraction stage. It is not proof that PagePith performs URL scoring, deduplication, robots evaluation, adaptive throttling, sitemap processing, or a complete crawl scheduler. Those remain responsibilities for your orchestration layer unless separately verified in your own integration.

Make the budget a feedback system

The first version of a crawl budget is a hypothesis. Keep it observable and revise it from evidence. When a host slows down, lower its local capacity. When a low-priority listing repeatedly reveals valuable new pages, increase its discovery score. When query variants collapse to identical content, improve normalization and stop paying for them.

The durable principle is simple: protect host health, reserve capacity for important pages, and use every response to make the next fetch decision better.

Ready to test a retrieval step against your own permitted URLs? Sign up for PagePith.