How to Convert Web Scraping Failures into Actionable Error Categories

A failed scrape is not one kind of event. A DNS lookup that fails, a 429 Too Many Requests response, a changed CSS selector, and a database write error may all appear as “job failed” in a dashboard—but they require radically different responses.

That difference is the foundation of reliable web scraping error handling. Instead of catching a broad exception and retrying every URL, classify the failure by the lifecycle stage that produced it. Then attach a deliberate action: retry, back off, repair, skip, quarantine, or alert.

This approach reduces needless traffic, makes regressions visible sooner, and leaves enough evidence to reproduce a failure later.

Start with the scraping lifecycle

An error taxonomy should describe where work stopped before it describes the exception text. A useful lifecycle has seven stages:

1. Request construction — normalize a URL, select headers, validate credentials, and create the request.
2. Transport — resolve DNS, connect, negotiate TLS, and receive bytes.
3. HTTP and access — interpret an HTTP response, redirects, authentication, permissions, and rate limits.
4. Representation — verify that the returned body is actually the expected page, document, JSON payload, or rendered result.
5. Parsing — turn a representation into fields, links, records, or structured data.
6. Validation — check extracted records against business and schema requirements.
7. Persistence and downstream processing — deduplicate, publish, write, index, or enqueue the valid record.

This lifecycle prevents misleading labels. If a response was retrieved successfully but a selector found no product price, that is not a network outage. If extraction succeeded but a unique-key constraint rejected the record, that is not a parsing failure.

A taxonomy that leads to a decision

Use a stable top-level category for aggregation and a more precise subtype for diagnosis. The following model is small enough to operate and specific enough to guide automation.

Category	Typical subtypes	Default action	Why it is distinct
request	invalid URL, unsupported method, missing required configuration	Repair or reject before dispatch	No remote request should be sent.
transport	DNS failure, connection refused, reset, connect timeout, read timeout	Bounded retry when safe	An application response may never have arrived.
http	unexpected redirect, 4xx, 5xx, malformed protocol response	Decide by status semantics	The server did respond, so retryability depends on that response.
access	authentication required, insufficient permission, robots-policy decision, suspected block	Stop, escalate, or change authorized configuration	More requests can worsen the situation.
rate_limit	429, retry window supplied, quota exhausted	Slow down and honor server guidance	This is a pacing signal, not ordinary unreliability.
representation	wrong content type, challenge page, empty body, truncated document	Inspect and repair assumptions	Retrieval can work while the content is unusable.
parse	JSON decode error, missing selector, malformed markup, unexpected layout	Repair parser and replay saved input	Repeating the same request often returns the same incompatible input.
validation	missing required field, invalid type, out-of-range value	Quarantine record or repair mapping	Extraction produced data, but it does not meet the contract.
downstream	duplicate, storage outage, queue rejection, transformation failure	Retry or quarantine independently	Fetching and extraction may have completed correctly.

The category is your operational contract. Subtypes can evolve as your crawler encounters new conditions, but parse should not silently become transport just because both throw exceptions in the same worker process.

Separate connection time from response time

Transport failures need more detail than timeout. In Requests, a connection problem, a connection-establishment timeout, and a response-read timeout are separate exception types; ConnectTimeout is documented as safe to retry. Requests also notes that a timeout is not applied unless the caller configures one. Requests API documentation and the Requests quickstart are useful references when defining these boundaries.

Track at least two timer values:

• Connect timeout: how long the client allows to establish a connection.
• Read timeout: how long the client waits for the server to provide response data.

Those fields make incidents interpretable. A surge in connect timeouts can suggest a routing, DNS, or remote availability problem. A surge in read timeouts may indicate slow origin processing, oversized responses, or a threshold that is too aggressive for a particular target.

Do not label every timeout “retryable” without context. For an idempotent retrieval request, a bounded retry can be reasonable. For a request with side effects, or one where partial work matters, make the retry decision explicitly and record why.

Interpret HTTP outcomes instead of flattening them

An HTTP status is already structured information. Keep it intact.

A 404 usually means that the target resource is not available at that location; repeatedly requesting it is usually wasteful. A 401 means authentication is needed, while a 403 indicates that the current identity does not have sufficient permission. These are configuration or authorization paths, not transient server errors. See the HTTP authentication guide and RFC 9110 for the relevant semantics.

By contrast, 502, 503, and 504 may be temporary gateway or service conditions. A 503 can include Retry-After, which is a concrete signal to delay work rather than immediately retry. 429 is specifically a rate-limit response and can also include Retry-After; respect it at the host or account scope, not only for the individual URL. MDN’s 429 reference describes that response, while RFC 9110 defines the related HTTP semantics.

A practical mapping looks like this:

connect_timeout       -> retry with bounded exponential backoff
read_timeout          -> retry only if request semantics permit it
429 + Retry-After     -> pause the relevant target scope until that time
502 / 503 / 504       -> retry with backoff; cap attempts
401                   -> stop and refresh or correct authorized credentials
403                   -> stop and escalate; do not brute-force retries
404                   -> mark unavailable; revisit only under a deliberate recrawl policy

Keep robots-policy decisions separate from access denials. The Robots Exclusion Protocol provides crawler rules; it is not an authorization mechanism. That distinction is explicit in RFC 9309. A compliant crawler may choose to skip a URL due to policy even when the server would have returned a successful response.

Build fewer blind retries. If you are designing a new extraction workflow, create a PagePith account and make error categories part of the contract from the first job: sign up.

Treat successful retrieval and successful parsing as separate facts

One of the most expensive observability mistakes is overwriting a successful response with “scrape failed.” Instead, represent both facts:

{
  "fetch": { "outcome": "success", "http_status": 200 },
  "parse": {
    "outcome": "failure",
    "category": "parse",
    "type": "missing_required_selector"
  }
}

This lets you answer an important question: did the target change, or did the network fail?

For parsing errors, retain controlled diagnostic evidence: content type, byte count, a content hash, parser version, selector or extraction rule identifier, and a redacted sample of the response. Requests exposes JSON-decoding failures separately from transport and HTTP exceptions, which supports preserving the response context while recording a parser-specific problem. Requests’ API reference documents these distinct exception families.

Avoid storing raw bodies indiscriminately. Responses can include personal information, credentials, or proprietary content. Define retention limits, redact known secret patterns, and restrict access to failure samples.

Make validation failures replayable

Parsing answers “can I extract a value?” Validation answers “can I trust this record?” Those are different tests.

For example, a parser might extract "price": "0" from a page. It is syntactically valid, yet it may violate a rule that prices must be positive. Or a field may be present but have the wrong type because a source changed from a number to a formatted string.

Store validation failures with field-level paths and failed constraints:

{
  "category": "validation",
  "type": "schema_violation",
  "record_id": "hash:...",
  "parser_version": "catalog-v17",
  "errors": [
    {
      "instance_path": "/price",
      "rule": "minimum",
      "message": "must be greater than 0"
    }
  ]
}

JSON Schema treats validation as an assertion of constraints over an instance and specifies structured output that can identify validity, instance locations, keyword locations, and errors. That makes it a strong model for error records, even if your validation layer uses another implementation. See the JSON Schema Core specification and validation specification.

Quarantine invalid records with their input version and parser version. Then you can fix a mapping and replay the quarantined set without re-fetching every source page.

Define a policy table in code, not scattered catch blocks

A category becomes actionable only when it maps to a policy. Centralize that mapping so workers, queues, and alerts agree.

POLICY = {
    "transport.connect_timeout": {"action": "retry", "max_attempts": 3},
    "http.503": {"action": "retry", "max_attempts": 3},
    "rate_limit.429": {"action": "backoff", "scope": "host"},
    "access.403": {"action": "escalate", "max_attempts": 0},
    "parse.missing_required_selector": {"action": "repair", "max_attempts": 0},
    "validation.schema_violation": {"action": "quarantine"},
    "downstream.storage_unavailable": {"action": "retry", "max_attempts": 5}
}

The values are examples, not universal defaults. Tune attempts, delays, and alert thresholds for the request’s idempotency, permission model, freshness needs, and the target’s published guidance.

Capture the evidence needed to operate the system

Every terminal or retried outcome should produce a structured event. At a minimum, capture:

• stable category and specific type;
• target URL or a privacy-safe target identifier;
• HTTP method and status when available;
• attempt number and prior outcome;
• connect, read, and total durations;
• retry or backoff decision and its reason;
• worker, parser, and schema versions;
• redacted exception message, stack trace, or response sample reference.

This makes aggregate charts meaningful: transport failure rate by host, rate-limit events by credential scope, parse failures by parser release, and quarantined records by schema rule. It also makes alerting calmer. Alert on a sudden increase in a category that needs human intervention, such as parse or access; do not page a team for every isolated timeout.

A limited PagePith demonstration

The supplied PagePith proof shows a successful fetch of https://requests.readthedocs.io/en/stable/api/ at the fetch tier. It records the retrieved document title as “Developer Interface — Requests 2.34.2 documentation,” a content length of 101,771, and a Markdown excerpt beginning with the Requests interface documentation.

That is useful evidence for a lifecycle-aware record: the retrieval stage can be marked successful with target metadata and a content artifact available for subsequent analysis. It does not prove that PagePith automatically categorizes scrape errors, retries requests, renders browsers, bypasses access controls, validates schemas, or persists extracted records. Those application-level decisions still need an explicit error model such as the one described above.

Turn failures into a feedback loop

The goal is not to eliminate every failure. Websites change, networks are unreliable, permissions expire, and downstream systems have incidents. The goal is to ensure each failure produces the correct next action and enough evidence to improve the pipeline.

Start with a small set of stable categories, preserve the original cause, make retry decisions policy-driven, and isolate parser, validation, and persistence outcomes from retrieval. Once those boundaries exist, failures stop being a noisy count in a dashboard and become prioritized work: slow down, repair, authorize, replay, or investigate.

Ready to build a more observable extraction workflow? Sign up for PagePith.