How to Build a Source Registry for a Web Data Pipeline

A web data pipeline rarely fails because a team cannot fetch a URL. It fails because the details surrounding that URL live in scraper code, a ticket, an analyst’s memory, or an untracked configuration value.

When a publisher changes a page, moves a download, adjusts access guidance, or silently alters a table, engineers need quick answers: What is this source meant to represent? Who owns its integration? Which access route is authoritative? What did the extractor assume? How fresh must the resulting data be? What changed since the last good run?

A web data source registry is a versioned control plane for those answers. It is not another crawler queue, and it is not a substitute for runtime telemetry. It is the durable metadata layer that connects a conceptual source to its representations, retrieval rules, extraction contract, expectations, and history.

Start with the right unit: source, representation, and access method

Avoid making “one URL equals one source” the core model. A publisher may offer the same logical information through a landing page, a downloadable CSV, an RSS feed, a sitemap, or an API. Those paths have different stability, permissions, formats, and failure modes.

This distinction is consistent with DCAT 3, which separates a cataloged dataset from its distributions and data services. It also distinguishes an access URL, a direct download URL, and service-based access. For a web pipeline, that suggests a practical hierarchy:

• Source: the conceptual asset your pipeline needs, such as a public procurement notices collection.
• Representation: a concrete form of that asset, such as an HTML listing, a CSV export, or a JSON endpoint.
• Access method: how the pipeline retrieves that representation, including request method, authentication mode, rate assumptions, and conditional-request state.

This prevents a common maintenance error: a landing page redesign gets treated as the disappearance of the data itself, even when a stable download remains available.

Give every source a stable internal ID. URLs are important attributes, but they are poor primary keys: they can redirect, change paths, or be replaced. A source ID such as public-notices.example lets representations evolve without breaking lineage or dashboards.

Define a small but complete registry contract

A useful registry can begin as YAML or JSON in a Git repository. Group fields by the decisions they support, rather than creating an unstructured document of notes.

1. Identity and stewardship

At minimum, capture:

• id, title, and description
• publisher or external owner
• internal technical owner and escalation contact
• source status and contract version
• canonical landing page, when one exists
• created, reviewed, and last-modified dates

DCAT 3 includes identifiers, titles, descriptions, publishers, contact points, lifecycle status, versions, landing pages, and modification metadata in its catalog resource model. You do not need RDF to benefit from the underlying discipline: make identity and accountability explicit.

Use lifecycle states that describe operational reality. For example:

• proposed: being evaluated; no production dependency
• active: approved and scheduled
• degraded: still used, but currently violating an expectation
• deprecated: replacement exists; new consumers should not depend on it
• retired: no longer retrieved

A state is more actionable than a comment like “maybe broken.” It lets schedulers, alerts, and downstream users make predictable decisions.

2. Rights and access assumptions

Keep permission-related context separate from selector logic. Record the license or policy reference, known access-rights notes, the review owner, and the date the assumption was last reviewed. Also record the pipeline’s user-agent identity and any authentication mode—but never commit secrets. Store only a secret reference or credential class.

Robots guidance belongs here as an explicit, time-bound assumption. RFC 9309 defines the Robots Exclusion Protocol, including user-agent groups, allow/disallow rules, the top-level /robots.txt location, and caching behavior. It also makes an important boundary clear: robots.txt is not access authorization or a security mechanism.

For each host, preserve:

• robots retrieval timestamp and retrieval outcome
• the user-agent evaluated
• relevant matched rule or parser result
• the policy decision your pipeline made
• a recheck interval

Do not reduce this to robots_allowed: true. A boolean cannot explain why a path was allowed, which user-agent was considered, or whether the policy has since changed.

3. Retrieval state and representations

A representation needs enough detail for an engineer to reproduce retrieval without opening the scraper implementation. Store its type (html, csv, json, feed, or similar), primary access route, fallback route if one is approved, request method, content expectations, pagination strategy, and timeout or cadence class.

Maintain retrieval state separately from static configuration:

• last successful retrieval time
• last response status
• ETag, if observed
• Last-Modified, if observed
• content hash and normalized-record hash
• redirect target, when applicable

RFC 9110 describes Last-Modified as a representation modification timestamp and recommends sending it when it can be determined consistently; it supports conditional requests that can avoid unnecessary transfers. Retaining validators and response state lets a poller use conditional requests while making its behavior inspectable.

4. Extraction and schema contract

Extraction configuration is an interface contract. Make it versioned and reviewable rather than embedding every assumption in application code.

Record the endpoint or document region, expected format and encoding, selector or parser name, pagination method, normalization rules, target schema reference, and transformation version. For HTML, include a semantic fallback strategy where practical—for instance, identify a table by heading context plus required column labels rather than by a brittle positional selector alone.

This mirrors a useful separation in the Open Data Contract Standard: physical connection details can be distinct from logical schema, ownership, quality rules, and service levels. In a source registry, runtime credentials and network details should not obscure what the pipeline expects the content to mean.

Example: a Git-friendly source record

The following is deliberately compact. It shows the shape of a source record, not a universal schema.

id: municipal-notices
version: 1.3.0
status: active
identity:
  title: Municipal public notices
  publisher: Example Municipality
  internal_owner: data-platform@company
  description: Notices published for downstream monitoring.

rights:
  policy_reference: publisher-terms-reviewed-2026-09
  reviewed_at: 2026-09-10
  robots:
    user_agent: AcmeDataBot
    retrieved_at: 2026-09-11T08:00:00Z
    decision: allowed
    matched_rule: "Allow: /notices/"

representations:
  - id: notices-json
    type: json
    role: primary
    access:
      route: notices-api
      authentication: none
      pagination: cursor
    retrieval_state:
      last_success_at: 2026-09-11T08:05:12Z
      last_status: 200
      etag: '"a1b2"'
      content_hash: sha256:example
    extraction:
      parser: notices_v3
      schema_ref: schemas/notices-v2.yaml
      transformation_version: 3.1.0

expectations:
  declared_update_frequency: daily
  freshness:
    max_age: 30h
  validation_suite: validation/notices-core
  failure_route: team-data-ingestion

observations:
  last_changed_at: 2026-09-10T17:30:00Z
  median_change_interval: 24h
history:
  - at: 2026-09-01
    change: Switched primary representation from HTML to JSON.

Two details matter here. First, route is an internal label rather than a hard-coded URL in every component. The registry can point to an environment-specific resolution mechanism without checking sensitive context into Git. Second, static intent (max_age) and dynamic evidence (last_success_at, hashes, observed change interval) are separate. One says what should happen; the other says what did happen.

Build the registry before the crawler fleet grows. A small set of reviewed records can reveal missing owners, ambiguous access paths, and untestable freshness assumptions early. If you are evaluating a browser-based way to inspect source material during that work, try PagePith.

Make freshness an executable expectation

“Updates daily” is useful context, but it is not a service-level objective. Convert it into a threshold that a job can test, such as “the newest accepted record must be no older than 30 hours” or “the primary representation must be retrieved successfully at least once every 12 hours.”

Keep three categories separate:

1. Publisher-declared signals: sitemap lastmod, changefreq, page timestamps, and stated release schedules.
2. Pipeline observations: fetch time, response validators, content hashes, and intervals between meaningful changes.
3. Your service expectation: the maximum acceptable age, retrieval cadence, and alerting policy.

The Sitemaps protocol defines optional lastmod and changefreq values, but cautions that lastmod should represent the linked page’s actual modification date rather than sitemap generation time. Treat these as signals, not guarantees. A sitemap can inform scheduling; it should not silently override observed behavior.

The Data Contract CLI service-level documentation provides a useful model: freshness, frequency, and retention expectations can be declared and tested. Apply that idea to web ingestion. A registry field should map to a concrete check and a response when it fails.

For example:

• Freshness failure: alert the source owner and mark the source degraded after the threshold is exceeded.
• Retrieval failure: retry according to a documented policy, then preserve the response outcome.
• Schema failure: quarantine the affected run instead of publishing partially mapped records.
• Meaningful content change: open or route a review event when an expected structure hash shifts.

Link every source to validation and evidence

A scraper that returns HTTP 200 has not necessarily succeeded. The response could be an error page, a login page, an empty payload, or a structurally changed document.

Each registry record should reference reusable checks: required fields, allowed values, minimum record count, uniqueness rules, and source-specific assertions. Also keep the most recent validation status, validation run ID, and failure route in operational state.

Great Expectations’ validation workflow describes reusable checkpoints that associate data with expectation suites and retain validation results. The implementation can vary, but the registry pattern is durable: configuration should point to validation; runtime systems should persist the outcome.

This creates a clean division of responsibility:

• The registry defines what a source is expected to provide.
• The retriever captures what was received.
• The extractor produces normalized data.
• The validator determines whether the run met the declared contract.
• Observability systems track trends and route failures.

Operate the registry as code

Put records, schemas, and validation references under version control. Require pull-request review for changes that alter access behavior, extraction rules, lifecycle state, ownership, or freshness thresholds. The Open Data Contract Standard documentation describes the broader practice of storing contracts in Git, reviewing them, linting them, and testing them in CI/CD; a source registry benefits from the same workflow.

Add lightweight CI checks before making the registry authoritative:

• schema validation for every record
• unique source and representation IDs
• required owner, status, and policy-review fields
• no embedded credentials or tokens
• valid references to parsers, schemas, and validation suites
• lifecycle transition rules, such as requiring a replacement before retirement

Then add a scheduled reconciliation job. It can re-evaluate robots guidance, verify access routes, compare observed and expected freshness, and flag representations whose validators or content hashes changed unexpectedly. The result should update operational state or create a review event—not overwrite the reviewed contract without human approval.

A practical rollout sequence

Do not attempt to catalog every historical integration first. Start with sources that are business-critical, volatile, or frequently repaired.

1. Create an initial record with identity, owner, lifecycle state, and one primary representation.
2. Record rights and robots assumptions, including when they were evaluated.
3. Move extraction details out of opaque application settings and into a versioned contract reference.
4. Add retrieval state for the next successful run, including validators and hashes when available.
5. Turn the first freshness and schema assumptions into executable checks.
6. Add review gates and a change log as the registry becomes the system of record.

The outcome is not perfect certainty about the web. It is a maintained record of the assumptions your pipeline depends on, plus evidence that lets engineers diagnose change without guesswork.

PagePith demonstration: capturing a source document as evidence

The supplied PagePith proof shows a browser-tier retrieval of the W3C DCAT 3 specification. The returned title was “Data Catalog Vocabulary (DCAT) - Version 3,” the captured content length was 410,188, and the Markdown excerpt included the document’s abstract explaining that DCAT supports interoperable web data catalogs.

That is a useful example of the evidence a registry workflow can preserve during source assessment: requested URL, retrieval mode, document title, a captured excerpt, and retrieval metadata. The proof does not establish broader PagePith behavior such as monitoring, extraction, or automated validation, so those should be evaluated separately for your environment.

A registry makes this evidence useful by associating it with a stable source ID, an owner, an access decision, and a review date—rather than leaving it as an isolated fetch result.

Ready to make web-source assumptions reviewable before they become incidents? Sign up for PagePith.