A Freshness Strategy for Web Data That Changes at Different Speeds

Freshness is a policy, not a polling interval

A web-data pipeline becomes unreliable when every source is assigned the same recrawl schedule. Polling a price page and a slowly revised standards document every 15 minutes wastes requests. Checking a fast-moving availability feed once a day can leave downstream users with materially old data.

A practical freshness strategy for web data starts by defining expectations per source or source class. The goal is not to fetch everything as frequently as possible. It is to make a deliberate promise about how old each dataset may be, how changes are detected, what happens when refreshes fail, and what consumers see when the promise cannot be met.

This distinction matters operationally. Workflow scheduling systems can separate the interval represented by a run from the time at which the run actually executes, and can support custom scheduling logic rather than only a global cadence. Apache Airflow's timetable documentation is a useful reference point for that model. Your ingestion service should make the same distinction: a source's freshness target is data policy; a fetch job is merely one attempt to satisfy it.

Model each source as state plus policy

For every URL, feed, API endpoint, or publisher domain, maintain two related records:

1. Policy: the target freshness, priority, allowed request pattern, retry rules, and stale-data behavior.
2. Observed state: the last check, last detected change, last successful fetch, last successful parse and load, validators, content fingerprint, and current status.

A minimal policy could look like this:

source: supplier-inventory
priority: critical
check_every: 15m
warn_after: 30m
stale_after: 60m
retry:
  attempts: 3
  backoff: exponential
on_refresh_failure: retain_last_known_good
signals:
  use_etag: true
  use_last_modified: true
  inspect_sitemap: false

A monthly public report might have a very different policy:

source: standards-library
priority: normal
schedule: "0 6 * * 1"
warn_after: 14d
stale_after: 35d
retry:
  attempts: 2
on_refresh_failure: retain_last_known_good
signals:
  use_etag: true
  use_last_modified: true
  inspect_sitemap: true

The example values are policy choices, not universal defaults. What matters is that the values are explicit, versioned, and visible to the teams that consume the data.

Use tiers to make policy manageable

Most teams do not need a bespoke schedule for every URL on day one. Begin with a small set of source tiers, then override individual sources when evidence justifies it.

Tier	Typical source behavior	Scheduling approach	Example consequence
Critical	Changes often and affects a user-facing decision	Short rate-based checks	Flag stale quickly and prioritize recovery
Active	Usually changes daily or several times a week	Daily or several scheduled checks	Refresh within the agreed daily window
Slow	Changes weekly, monthly, or irregularly	Cron-style or signal-led checks	Avoid unnecessary body downloads
Archive	Expected to be stable	Occasional audit validation	Preserve provenance and check for unexpected revisions

A scheduler may support rate, cron, and one-time schedules, as well as retry handling and dead-letter queues. Those primitives are documented for Amazon EventBridge Scheduler, but the tiering approach applies regardless of the orchestrator you use.

Do not classify sources based only on intuition. Start with a conservative tier, capture observed change history for several cycles, and then adjust. A source that has not changed in six months may still need a stricter policy if a rare change is business-critical. Conversely, a source that changes many times a day may not need minute-level ingestion if your product only publishes daily snapshots.

Validate before downloading full content

The highest-leverage efficiency improvement is conditional HTTP revalidation. Store response validators after a successful fetch, then send them on the next request:

GET /resource HTTP/1.1
Host: example.org
If-None-Match: "rev-8d291"
If-Modified-Since: Tue, 02 Sep 2026 15:00:00 GMT

When the representation has not changed, the origin can respond with 304 Not Modified and omit the response body. HTTP defines If-None-Match and If-Modified-Since for this purpose; when both are available, If-None-Match is the more accurate validator. See RFC 9110 for the protocol semantics.

In ingestion state, persist at least:

• final resolved URL and canonical identity, where applicable;
• ETag and Last-Modified values exactly as returned;
• time of the last validation attempt;
• time of the last successful full fetch;
• a normalized-content hash;
• parser version and schema version used for the loaded record.

MDN's HTTP caching guide explains the ETag/If-None-Match and Last-Modified/If-Modified-Since pairs, including the precedence of ETags when both mechanisms are used. Not every publisher provides trustworthy validators, so retain a fallback: fetch the representation according to policy, normalize the content in a deterministic way, and compare a stored hash. That fallback costs a body transfer, but it still prevents needless parsing and downstream writes when the content is unchanged.

A 304 is successful freshness work. Record it as a completed validation, not as a no-op. It proves that the source was checked at a specific time, even though there was no new body to ingest.

Build the source inventory before scaling the crawler. A clear policy table, persisted validators, and a visible stale state will usually improve data trust more than a shorter global interval. Set up PagePith when you are ready to test fetches as part of that workflow.

Treat sitemaps as prioritization signals

Publisher sitemap metadata can reduce discovery work, particularly for broad sites with many mostly stable pages. The Sitemap protocol allows optional lastmod metadata and supports sitemap index files, which can help identify sitemap files worth retrieving incrementally. It also makes an important distinction: the modification time of a sitemap index entry is not automatically the modification time of every page listed by that sitemap. Review the Sitemaps protocol for those semantics.

That leads to a safe sequence:

1. Fetch the sitemap or sitemap index on its own schedule.
2. Use lastmod, newly discovered locations, and removed locations to rank candidates.
3. Revalidate the actual page before declaring its record changed.
4. Record sitemap observations separately from page-level validation.

This avoids an easy mistake: treating an updated sitemap as proof that all listed content changed. Sitemap signals are useful for choosing where to spend requests; HTTP validators or page content determine whether a specific record should be updated.

Define stale-data behavior before failures happen

Refresh failures are normal: a server may time out, return an error, temporarily block a request, or publish malformed content. The unsafe response is to silently replace a valid record with an empty or partial result. The equally unsafe response is to keep serving old data without telling anyone.

For each policy, define these states:

• Fresh: the latest successful validation or ingestion is within the target.
• Warning: the target has been missed, but the last known-good record is still acceptable for a limited period.
• Stale: the maximum age has been exceeded; retain the known-good record if appropriate, but expose its age and status.
• Unavailable: no valid record exists, or a consumer cannot safely use the stale result.

Persisted workflow results make it possible to retrieve prior outputs while retries run. Prefect's results documentation describes persisted results in the context of caching, retries, and reusing task outputs. The design decision is yours: preserve the last known-good value, attach last_success_at and freshness_status, and prevent failed or partial parses from becoming the current canonical record.

A consumer-facing record might carry explicit metadata:

{
  "product_id": "A-104",
  "availability": "in_stock",
  "observed_at": "2026-09-08T10:15:00Z",
  "last_success_at": "2026-09-08T10:15:00Z",
  "freshness_status": "warning",
  "stale_after": "2026-09-08T11:15:00Z"
}

This is more useful than a Boolean is_current. It lets an API, dashboard, or downstream job choose an appropriate response: continue with a warning, defer an action, or reject results beyond a stricter service-level requirement.

Propagate source freshness downstream

Freshness should not stop at extraction. If a derived table, search index, or feature set depends on a source with an hourly warning threshold, it should not be treated as indefinitely cacheable.

There is a concrete pattern for this in Prefect's dbt orchestrator documentation: cache expiration can be derived from the freshness thresholds of upstream sources. The broader principle is straightforward: downstream materializations should inherit the most restrictive relevant freshness constraint, unless you intentionally document a different contract.

For example, suppose a recommendation dataset depends on:

• catalog metadata with a 7-day warning threshold;
• inventory with a 30-minute warning threshold; and
• exchange rates with a 24-hour warning threshold.

If inventory is part of the output's meaning, a week-long cache for the recommendation dataset would conceal an inventory miss. Either refresh or invalidate the derived output based on the tightest dependency, or label it with its dependency freshness so consumers can make an informed decision.

Monitor the pipeline as a chain of separate events

A single metric called “last updated” is usually ambiguous. It may mean a publisher changed a page, your fetcher observed it, your parser accepted it, or downstream consumers received it. Track each event separately:

Signal	Question it answers
Last source validation	When did we last check the origin?
Last source change detected	When did validators or content show a change?
Last successful fetch	When did we obtain a usable representation?
Last successful parse/load	When did canonical data become available?
Current data age	How old is the record consumers are using?
Policy status	Is this source fresh, warning, stale, or unavailable?

Alert on missed freshness targets and repeated failures, not simply on every unchanged response. An unchanged source may be exactly what you expect. A source that cannot be validated before its stale deadline is the operational problem.

An honest PagePith demonstration

The supplied PagePith proof shows a fetch request for RFC 9110. It returned the title “RFC 9110: HTTP Semantics” at the fetch tier, with a reported content length of 554021 and a Markdown excerpt beginning with the RFC header and abstract.

That is a useful example of the first ingestion step: retrieve a source and preserve content suitable for inspection or further processing. This proof does not establish conditional-request support, recurring scheduling, change detection, parsing accuracy, or freshness guarantees for that URL. Those require additional requests, stored state, and a policy like the one described above.

Make freshness an explicit contract

A resilient pipeline does not promise that every web record is always current. It promises that every source has a documented freshness target, an efficient validation path, a recovery plan, and an honest stale-data signal.

Start with a small tier taxonomy, persist HTTP validators and content fingerprints, use sitemap metadata to prioritize rather than prove changes, and retain last known-good results with visible age. As change history accumulates, refine schedules based on observed behavior and business impact.

If you want to begin testing web retrieval in that policy-driven workflow, sign up for PagePith.