How to Build an Incremental Web Data Ingestion Pipeline

Repeatedly downloading every known page is not a synchronization strategy. It is a full recrawl with a timer.

A production-grade incremental web data ingestion system instead treats each source as a stateful synchronization problem. It discovers likely changes, asks the origin whether stored representations are still current, writes results idempotently, and maintains enough state to resume safely after interruption.

That framing solves the common recurring-ingestion failures:

• unchanged pages consume bandwidth and crawl capacity;
• a failed run loses its place in a paginated source;
• retried work creates duplicate records;
• a temporary missing page is mistaken for a deletion;
• rate limits cause a whole source to fail repeatedly.

This guide presents a practical architecture for recurring collection from websites, feeds, and HTTP APIs.

Start with a synchronization model

The pipeline should answer two distinct questions:

1. What resources might have changed since the last run?
2. For each candidate, is the stored representation still valid?

Discovery answers the first question. HTTP validators, source version fields, and content hashes answer the second.

Keep these stages separate. A sitemap's lastmod timestamp is useful for prioritizing or narrowing work, but it should not be the only basis for overwriting stored content. The sitemap protocol describes lastmod as the linked page's modification date, while Google notes that use of this field depends on it being consistently accurate. Validate candidate pages again when correctness matters. Sitemaps XML format · Google sitemap guidance

A useful lifecycle looks like this:

Discover -> Schedule -> Fetch conditionally -> Parse -> Persist atomically
                     |                         |
                     v                         v
                 Retry later               Reconcile deletion

The important property is durability: every transition should be recoverable from stored state rather than dependent on in-memory progress.

Choose stable identities before writing code

A record needs an identity that survives repeated runs. For web pages, this is commonly:

source_id + canonical_url

For an upstream API, prefer the provider's stable object ID when one exists. Do not use a crawl timestamp, response body, or queue message ID as the primary identity. Those values change across observations and will turn retries into duplicates.

Canonicalize URLs consistently before calculating the key. For example, define how the pipeline handles fragments, default ports, trailing slashes, query parameters, redirects, and tracking parameters. Keep both the original requested URL and the final canonical URL when redirects matter to your audit trail.

A minimal table design might include:

create table source_document (
  source_id text not null,
  canonical_url text not null,
  body_hash text,
  etag text,
  last_modified text,
  parsed_payload jsonb,
  first_seen_at timestamptz not null,
  last_checked_at timestamptz not null,
  last_changed_at timestamptz,
  status text not null,
  primary key (source_id, canonical_url)
);

Use atomic upserts for writes. PostgreSQL documents INSERT ... ON CONFLICT DO UPDATE as an atomic insert-or-update operation, which makes it appropriate for concurrent workers processing the same stable identity. PostgreSQL INSERT documentation

insert into source_document (
  source_id, canonical_url, body_hash, etag, last_modified,
  parsed_payload, first_seen_at, last_checked_at, last_changed_at, status
)
values (
  :source_id, :canonical_url, :body_hash, :etag, :last_modified,
  :payload, now(), now(), now(), 'active'
)
on conflict (source_id, canonical_url) do update
set
  body_hash = excluded.body_hash,
  etag = excluded.etag,
  last_modified = excluded.last_modified,
  parsed_payload = excluded.parsed_payload,
  last_checked_at = now(),
  last_changed_at = now(),
  status = 'active';

In a real implementation, avoid updating last_changed_at when the new normalized content hash equals the stored hash. A 200 response can still contain semantically unchanged content.

Discover candidates without recrawling the entire web inventory

Maintain a source-specific discovery adapter. Different inputs expose different notions of progress:

Source type	Useful checkpoint	Typical candidate signal
Sitemap index	last processed index revision or observation time	changed sitemap file or lastmod
RSS or Atom feed	newest observed item ID/date	recently published or updated item
Paginated API	page token, cursor, or continuation URL	records after a cursor
Known URL inventory	last checked time	scheduled revalidation

XML sitemaps, sitemap indexes, RSS, and Atom feeds can all contribute URLs. Google documents these formats as discovery mechanisms and notes that feeds generally contain recent URLs rather than the complete historical inventory. Google sitemap guidance

For APIs, persist the opaque continuation mechanism supplied by the source rather than trying to calculate your own offset. For example, GitHub's REST API pagination documentation directs clients to follow URLs in the Link response header. That pattern generalizes well: treat cursor values and next-page URLs as source-owned state. GitHub pagination documentation

Commit a discovery checkpoint only after the associated batch is durably persisted. If a worker stores the next cursor first and crashes before writing the records, those records may never be revisited.

A simple batch transaction boundary is:

1. Read the current cursor.
2. Request one page or discovery unit.
3. Process and durably write all resulting records.
4. Store the returned next cursor in the same transaction or an equivalent recoverable workflow.

If the source cannot support a transactional coupling with your database, design for replay. Reprocessing a page is acceptable when writes are idempotent; silently skipping it is not.

Use conditional HTTP requests for known pages

For every previously fetched URL, persist the latest response validators:

• ETag
• Last-Modified
• normalized body hash as a fallback verification signal

On the next fetch, send conditional headers:

GET /docs/widget HTTP/1.1
Host: example.org
If-None-Match: "f8a4c-7d"
If-Modified-Since: Tue, 07 Oct 2026 10:00:00 GMT

HTTP defines ETag with If-None-Match and Last-Modified with If-Modified-Since for conditional requests. When the representation is unchanged, a conditional GET can return 304 Not Modified, allowing the client to retain its stored representation rather than transfer the body again. RFC 9110 describes If-None-Match as primarily intended to make GET updates efficient; it also specifies that If-Modified-Since is ignored when If-None-Match is present. Store both when available, but prefer the entity tag in the request. RFC 9110

Treat a 304 as a successful no-op:

304 Not Modified -> update last_checked_at -> do not parse -> do not rewrite content

For a 200 OK response:

1. validate the response and final URL;
2. calculate a normalized content hash;
3. parse only when necessary;
4. atomically upsert content and validators;
5. record observation metadata.

Not every site emits reliable validators. In that case, schedule a normal fetch and use a normalized body hash to avoid downstream writes when content is equivalent. This does not avoid the transfer, but it still avoids needless parsing, embedding, indexing, or change events.

Evaluating recurring extraction workflows? Build the state model independently of the fetcher so that discovery, identity, validators, retries, and persistence remain testable components. Create a PagePith account to explore a workflow for URL retrieval.

Keep retry state separate from content state

A document's last successful representation and a failed request are different facts. Store them separately.

create table fetch_attempt (
  source_id text not null,
  canonical_url text not null,
  attempt_count integer not null default 0,
  last_error text,
  last_status integer,
  next_retry_at timestamptz,
  request_id text,
  updated_at timestamptz not null,
  primary key (source_id, canonical_url)
);

This separation prevents a temporary timeout from erasing a valid document or changing a source checkpoint prematurely.

Classify failures deliberately:

• Network failures and 5xx responses: retry with a bounded policy.
• 429 responses: honor Retry-After when provided, then apply per-host concurrency limits.
• 401/403 responses: usually require configuration or permission review rather than blind retries.
• 404/410 responses: record missing evidence, then apply the source's deletion policy.
• Parse failures: retain the fetched raw representation or enough diagnostic metadata to investigate without losing the prior good record.

HTTP defines 429 Too Many Requests for rate limiting and permits a Retry-After field. Retry-After can express either a delay in seconds or a date. RFC 6585 · RFC 9110

A bounded exponential-backoff policy with jitter is a reasonable default:

base delay: 5 seconds
attempt cap: 6
maximum delay: 15 minutes
actual delay: random value from 0 to calculated delay

Jitter reduces synchronized retry bursts when many workers see the same outage. Retry controls such as backoff rate, maximum attempts, maximum delay, and jitter are documented in AWS Step Functions as configurable workflow behavior. AWS retry handling documentation

Reconcile deletions as an explicit state transition

A missing page is not always deleted. It may be temporarily unavailable, redirected, protected by changed access rules, or returning an incorrect response during an outage.

Therefore, do not immediately delete a stored document after one 404. Instead, model a tombstone or verification state:

active -> suspected_missing -> tombstoned

Example policy:

1. First 404: set suspected_missing, retain prior content, schedule verification.
2. Repeated missing responses across a configured time window: create a tombstone.
3. Deletion marker in an authoritative source feed: tombstone immediately.
4. Valid 200 response: restore active and clear missing evidence.

A tombstone should retain the stable identity, source URL, observation time, reason, and evidence. This is safer than silently removing the row, especially when downstream systems need to remove search records or derived entities. The same principle appears in versioned object storage: Amazon S3 represents a deletion with a delete marker rather than assuming the object's history should disappear. Amazon S3 delete marker documentation

Add scheduling, access, and observability controls

Incremental efficiency can be lost if all candidates are scheduled at once. Use a scheduler that understands:

• per-host concurrency;
• crawl intervals by source importance and observed volatility;
• retry queues separate from normal work;
• robots policy checks before recurring scheduling;
• a dead-letter or review path for exhausted failures.

The Robots Exclusion Protocol standardizes the rules published in robots.txt that crawlers are requested to honor. It is not authorization; access controls, terms, and other obligations remain separate concerns. RFC 9309

Measure every stage independently. At minimum, emit counters for:

candidates_discovered
conditional_requests_sent
responses_304
responses_200_changed
responses_404
responses_429
retries_scheduled
parse_failures
upserts_completed
tombstones_created

These metrics tell you whether the system is actually incremental. For example, a lower fetch count could mean validators are working—or it could mean a broken discovery cursor is skipping pages. Pair counters with source lag, queue depth, retry age, and the age of the oldest unverified record.

An honest PagePith demonstration

The supplied PagePith proof shows a fetch request for RFC 9110. It returned the title “RFC 9110: HTTP Semantics”, a content length of 554,021, and a Markdown excerpt beginning with the RFC's document header and abstract.

That is a concrete example of obtaining a web document as Markdown-oriented content for a downstream ingestion process. The proof does not establish behavior beyond that fetch result, so it should not be read as evidence of rendering, crawling, scheduling, change detection, or extraction performance. Those pipeline responsibilities still need to be implemented and validated in your architecture.

A practical implementation checklist

Before operating at scale, verify that your design can answer yes to each question:

• Does every source have a durable checkpoint?
• Is a checkpoint advanced only after its data is safely written?
• Are record identities stable across retries and redirects?
• Are ETag and Last-Modified retained per canonical URL?
• Does 304 update observation metadata without rewriting content?
• Are retry records separate from the latest successful content?
• Are rate limits handled with Retry-After, backoff, jitter, and host-level limits?
• Is deletion represented as a tombstone or confirmed state rather than a single failed fetch?
• Can operators distinguish discovery failures from fetch, parse, and persistence failures?

An incremental pipeline is successful when it can safely run forever: it spends work on likely changes, preserves prior truth through temporary failures, and can replay any batch without creating duplicate data.

Ready to evaluate a retrieval component for your ingestion architecture? Sign up for PagePith.