How to Recover Useful Data From an Incomplete Scraped Page

An HTTP 200 is not a completeness guarantee

A scraper can receive a successful response and still produce an unusable record. The body may be cut short, a JavaScript application shell may contain none of the actual content, an expected field may have moved, or a source may return a degraded variant under load.

The dangerous outcome is not a visible exception. It is a plausible record that quietly enters a downstream database with a missing price, stale availability, truncated description, or incorrect association between fields.

Treat recover incomplete scraped page data as a staged decision process rather than a single retry button:

1. Classify the failure layer.
2. Preserve evidence from the attempt.
3. Choose the least expensive, policy-compliant recovery path.
4. Validate the recovered record against explicit rules.
5. Publish, retry later, or send the record to review.

This approach prevents two common mistakes: repeatedly retrying pages that cannot improve, and accepting partial data merely because a parser returned an object.

Classify the failure before choosing a fallback

The same symptom—title present but price absent—can arise at several layers. Recovery is much more reliable when these layers are kept separate.

1. Transport-level incompleteness

First ask whether the response itself may be incomplete. A connection that closes early is not always distinguishable from a complete response without an explicit transport error; HTTP semantics make this an important distinction. A 206 Partial Content response is different: it explicitly communicates a byte-range representation. Its Content-Range metadata identifies the returned interval and the resource size when known. See RFC 9110 and MDN's range request guide.

Record enough evidence to investigate later:

• Requested and final URL after redirects
• Status code
• Response headers, especially Content-Length, Content-Range, ETag, and Retry-After
• Received byte count
• Content type and character encoding
• Timeout, connection, and decompression errors
• A body hash or retained raw response where your retention policy permits it

Do not blindly concatenate a 206 response onto a normal response. A range response is valid only in the context of a deliberate range request and a verified Content-Range contract.

2. Server-level degradation

A valid HTML response may be an error template, an interstitial, a regional variant, or a throttled version that omits the content you expect. Look for source-specific fingerprints such as a maintenance heading, an empty results region, an authentication prompt, or a template whose main record container has disappeared.

HTTP failures deserve a different path from parser failures. In Scrapy, network and HTTP-processing failures can be handled through errbacks, while non-success responses can be routed there by its HTTP error middleware. That separation lets an application decide whether it should retry, defer, or preserve a diagnostic result. Scrapy's request and response documentation describes this error-handling model.

3. Rendering-level incompleteness

If the initial HTML is mostly scripts and empty containers, the content may be populated after client-side execution. The recovery step is not simply “wait longer.” Wait for evidence that the data you need exists.

For example, a browser recovery pass might wait until a product heading is visible and the price region has non-empty text. Playwright's locator assertions repeatedly check expected conditions until they pass or time out, and its documentation discourages treating networkidle as a universal readiness signal. Use conditions tied to your record instead. Playwright assertions are useful reference behavior for this pattern.

4. Extraction-level incompleteness

Sometimes the response and rendered page are fine, but the extractor is no longer aligned with the source. A selector may match the wrong repeated element, a label may have changed, or a field may now be represented in structured metadata rather than visible text.

Prefer durable page contracts where they exist: accessible roles, labels, text associated with a field, and intentional test IDs. Structural CSS or XPath chains tend to fail when the DOM is rearranged. Playwright's locator guidance explains why these user-facing or explicit contracts are generally more resilient.

Preserve an attempt ledger, not just the final item

A recovery system needs provenance. Store an attempt record alongside the extracted item, even if only briefly:

attempt = {
    "url": final_url,
    "fetched_at": now_iso8601(),
    "status": response.status,
    "content_type": response.headers.get("content-type"),
    "received_bytes": len(response.body),
    "content_length": response.headers.get("content-length"),
    "etag": response.headers.get("etag"),
    "extractor_version": "product-v12",
    "missing_required_fields": ["price"],
    "recovery_path": ["http", "json_ld"],
}

This ledger answers questions that a final JSON object cannot:

• Did the source change, or did the parser change?
• Was this a timeout, an empty shell, or a selector mismatch?
• Did a fallback supply the value?
• Did two representations disagree?
• Is the same URL failing repeatedly?

It also enables targeted remediation. A widespread HTML parser failure calls for an extractor update; repeated 429 responses call for less pressure, not another selector.

Use bounded, status-aware retries

Retries are appropriate for transient events, not as a generic response to missing data. Timeouts, connection problems, and selected server errors can improve on a later attempt. In contrast, retrying a consistently missing DOM field three times usually adds load without improving the record.

Scrapy's RetryMiddleware is designed for temporary failures and retryable response statuses; its documented defaults include responses such as 408, 429, 500, 502, 503, and 504, with a bounded retry count. Review its current behavior and configure it for the target rather than treating the default as a universal policy. Scrapy downloader middleware provides the details.

A practical policy looks like this:

Signal	Typical action
DNS failure, connection reset, timeout	Retry a small number of times with exponential backoff and jitter
429 or 503 with Retry-After	Defer until the requested time, then reduce pressure
500, 502, 504	Retry within a fixed budget; retain diagnostic evidence
206 without an intentional range workflow	Mark transport-incomplete and investigate or refetch
Valid document, required field absent	Try representation or rendering fallbacks once
Required field still absent after fallbacks	Keep as incomplete; queue for review or a later refresh

Honor Retry-After when it is present. HTTP defines it as an indication of when a follow-up request can be made, including in association with 503; 429 Too Many Requests may also include it. RFC 9110 covers the header semantics. Adaptive throttling also matters: Scrapy notes that rapid non-200 responses can otherwise lead a crawler to accelerate into an error condition. Scrapy AutoThrottle explains this failure mode.

Recover from lighter representations first

Browser rendering is valuable, but it is usually not the first fallback to attempt. It adds operational cost and can make diagnosis less clear. Start with representations that are already supplied by the page.

Inspect structured data

A page may expose JSON-LD in a script block even when the visible DOM is incomplete or difficult to select. JSON-LD is a standardized JSON-based Linked Data serialization, making it a reasonable candidate for supplementing or cross-checking HTML extraction. It should not automatically override visible content: validate its type, context, and agreement with other fields. The format is specified in the JSON-LD 1.1 recommendation.

For example, if visible HTML yields a name and SKU but misses availability, a product-oriented JSON-LD block may offer an availability value. Mark that field's provenance as json_ld, rather than presenting it as if it came from the same selector as the name.

Revalidate a previous representation

For refreshes of a previously incomplete page, save validators such as ETag when available. Conditional requests can ask whether a stored representation is still current, helping distinguish a changed source from a repeated response. MDN's conditional request guide explains validator-based revalidation.

This is especially useful when an item was last accepted with low confidence. If the resource is unchanged, another extraction run may simply reproduce the same gap. If it changed, run the normal extraction and validation path again.

Build recovery into the extraction workflow, not as an afterthought. If you are evaluating PagePith for source retrieval and Markdown-oriented inspection, create an account to try it with your own permitted URLs.

Make partial acceptance explicit

A partial record can still be useful, but only if its state is visible to consumers. Define a schema with three categories:

• Required fields: Without these, the record cannot be published as complete.
• Recommended fields: Missing values lower confidence but may not block publication.
• Consistency rules: Relationships that must hold, such as a sale price not exceeding a list price unless the source explicitly signals a different currency or unit.

Here is a simple, application-owned policy:

def assess_product(item):
    required = ["source_url", "name", "canonical_id"]
    missing = [field for field in required if not item.get(field)]

    errors = []
    if item.get("price") is not None and item["price"] < 0:
        errors.append("negative_price")
    if item.get("currency") and len(item["currency"]) != 3:
        errors.append("invalid_currency")

    completeness = 1 - (len(missing) / len(required))
    if missing or errors:
        return {"state": "incomplete", "score": completeness, "reasons": missing + errors}
    return {"state": "accepted", "score": 1.0, "reasons": []}

The score itself is not a truth claim. It is an operational signal. Keep the reasons, field provenance, and recovery history so a reviewer or downstream process can make an informed choice.

Feed export and item pipelines can carry these fields through to separate accepted and review datasets. Scrapy's export system supports structured item output, but the definition of “complete enough” remains an application decision. Scrapy feed exports documents the export layer.

An honest PagePith demonstration

The supplied PagePith proof shows a fetch-tier retrieval of https://www.rfc-editor.org/rfc/rfc9110.html. It reported the title “RFC 9110: HTTP Semantics”, a content length of 554,021, and a Markdown excerpt containing the document table, the HTTP Semantics heading, and its abstract.

That is a useful example of the first recovery step: preserve a retrievable representation and inspect whether expected document structure is present before relying on downstream extraction. This proof does not establish how PagePith handles JavaScript rendering, retries, alternate representations, confidence scoring, or review queues. Those remain workflow decisions to test against your own permitted targets.

Close the loop with observability and review

Measure incomplete records as a first-class outcome. Useful counters include:

• Completion rate by source and extractor version
• Missing required fields by field name
• Retry count and recovery path distribution
• Browser-render recovery rate
• Structured-data versus visible-HTML disagreements
• Age of records awaiting review

Alert on changes, not isolated failures. A sudden increase in missing price fields after an extractor deployment is actionable. One malformed page may simply belong in a review queue.

Finally, keep recovery within the source's applicable access rules, terms, authentication boundaries, and rate limits. Technical persistence is not permission. A good recovery policy protects both data quality and the systems being queried.

Turn incomplete pages into observable states

Reliable scraping does not mean every request succeeds. It means every result has a clear state, evidence, and next action. Separate transport, server, rendering, and parser failures; retry only transient conditions; choose light fallbacks before heavy ones; and never let an unvalidated partial object masquerade as complete.

To test a retrieval-oriented workflow with URLs you are permitted to access, sign up for PagePith.