How to Extract Dates, Authors, and Source Metadata from Web Pages Reliably

Metadata is often treated as a scraping afterthought: find one CSS selector for the byline, parse a date-shaped string, and save the URL. That approach works until a redesign, an incomplete template, a syndicated article, or a contradictory structured-data block reaches production.

A more dependable approach is to extract metadata from web pages as evidence reconciliation. Collect each plausible signal, normalize it carefully, preserve where it came from, then choose a field value with an explainable policy. This is useful whether you are building a research archive, content pipeline, monitoring system, or search index.

The central rule is simple: never let a convenient signal erase the evidence that supports—or contradicts—it.

Define the fields before writing selectors

Start with a model that distinguishes concepts which pages routinely blur together:

• Observed URL: the URL requested, including any tracking parameters or redirect entry point.
• Final URL: the location actually returned after redirects.
• Canonical candidate: a publisher's preferred URL, usually from a canonical link relation.
• Publication date: when the article first became available.
• Editorial modification date: when the publisher says the article was meaningfully updated.
• Representation modification date: when the server says the returned response was last modified.
• Author: a person or organization responsible for the content.
• Retrieval facts: fetch time, response headers, parser results, and the source location of every candidate.

These definitions prevent a common failure mode: labeling every timestamp as publishedAt and every visible name as author.

For example, a page could contain a visible “Updated” date, a JSON-LD datePublished, an Open Graph publication time, and an HTTP Last-Modified header. Those are not interchangeable. Google explicitly recommends distinguishing datePublished from dateModified, keeping markup consistent with visible dates, and avoiding dates that belong to events merely described in the article. It also notes that multiple date-related factors may be used rather than one signal alone. Its publication-date guidance is a useful validation checklist even if search is not your use case.

Collect candidates from independent channels

A resilient extractor gathers a set of candidates per field. Each candidate should include a raw value and a locator, not just a parsed result.

Semantic HTML and visible context

The semantic time element is the best HTML-level date signal. When it has a datetime attribute, that attribute is the machine-readable value; otherwise, its text can supply the value under the HTML rules. The standard permits several forms, including dates and date-times. See the HTML specification for time.

That does not mean every time value is an editorial date. Capture surrounding context before assigning a role:

• Is the element near the article heading or byline?
• Does nearby text say “Published,” “Updated,” or “Last reviewed”?
• Is it inside an event card, a comment, a related-story module, or the article body?
• Is its date implausibly in the future relative to fetch time?

A practical candidate record might look like this:

{
  "field": "datePublished",
  "rawValue": "2026-08-24T09:30:00Z",
  "normalizedValue": "2026-08-24T09:30:00Z",
  "sourceType": "semantic-html",
  "locator": "time[datetime] near article header",
  "pageUrl": "https://publisher.example/article",
  "retrievedAt": "2026-08-27T12:00:00Z",
  "parseStatus": "valid",
  "confidence": "medium"
}

The locator need not be a fragile CSS selector alone. A selector plus a short structural description, DOM path, or captured nearby label makes later audits far easier.

JSON-LD structured data

JSON-LD is often the highest-value source because it can express typed relationships rather than isolated strings. For article-like objects, look for Article, NewsArticle, and BlogPosting nodes, including nodes nested in graph-shaped data. Extract datePublished, dateModified, author, headline, mainEntityOfPage, and identifiers when available.

Google's Article structured-data documentation illustrates author data alongside publication and modification dates, including author names and identifying URLs. Still, JSON-LD is an input to reconciliation, not a verdict. A site may emit stale template data, duplicate article nodes, or structured data for a related item.

Parse JSON-LD defensively:

1. Collect every JSON-LD script block.
2. Parse each block independently so one malformed block does not discard the rest.
3. Flatten top-level arrays and graph containers into nodes.
4. Identify article candidates by type and their connection to the current page.
5. Record the JSON pointer or script-block index for each extracted field.

Keep the raw object fragment as well as the normalized candidate. When a publisher changes its markup, that retained evidence is often enough to diagnose the issue without another crawl.

Open Graph metadata

Open Graph article properties provide a separate channel for article:published_time, article:modified_time, and repeated article:author values. The Open Graph Protocol defines these as first-publication time, most recent modification time, and profile links for authors.

Treat author profile URLs as identity clues, not guaranteed display names. Fetching profile pages may be unnecessary and can create avoidable crawl volume. Preserve the URL, associate it with any corroborating author name, and let downstream policy decide whether it is sufficient.

Link relations and response headers

For canonicalization, examine both the document and the HTTP response. The HTML standard defines rel="canonical" as the preferred URL for the current document and rel="author" as a link to information about the page or nearest article author. The link-type definitions are useful for implementing both checks.

Also capture HTTP Link headers, redirects, status code, content type, and Last-Modified. A canonical declaration is an important preference signal, but it should not overwrite the observed or final URL. Google describes redirects and canonical annotations as strong canonicalization signals in its duplicate-URL guidance; for an extractor, that is a reason to record and assess the signal, not to assume every declaration is correct.

Last-Modified deserves a separate field such as representationLastModified. RFC 9110 defines it as the time the origin server believes the selected representation was last modified. Since a representation can be assembled from many parts, this may reflect a template, ad slot, or other non-editorial change. It is excellent for change detection and cache workflows, but is not automatically dateModified. See RFC 9110.

Build extraction results that can be inspected, not merely consumed. If your pipeline needs a repeatable record of what a page returned, create a PagePith account and evaluate it against your own metadata sources.

Normalize without destroying meaning

Normalization should make values comparable while retaining the original value. For dates, parse only formats you can validate, retain the original timezone offset, and convert to UTC only as an additional representation.

A safe date result has at least these states:

• Valid instant: an offset or UTC marker makes the moment unambiguous.
• Valid local date-time: syntactically valid, but no timezone is declared.
• Date only: suitable for publication-day display, not for ordering within a day.
• Unparsed: preserved as raw text for inspection.
• Rejected: invalid or implausible under a documented rule.

Do not silently attach the crawler's timezone to a timezone-less value. That turns uncertainty into a false instant. Likewise, do not infer a publication date from a URL path unless you explicitly classify it as a low-confidence heuristic.

Authors need similar care. Schema.org allows author to be either a Person or an Organization; reducing all values to strings loses useful semantics. The Schema.org author definition supports both types. A normalized author object can preserve a display name, type, identity URL, source-specific identifier, and the evidence that connected it to the page.

{
  "name": "Example Editorial Team",
  "type": "Organization",
  "url": "https://publisher.example/about",
  "evidence": ["json-ld:$.author", "link-rel-author"],
  "confidence": "high"
}

Deduplicate authors conservatively. Matching names alone can merge different people; matching identity URLs alone can miss a publisher's alternate profiles. Keep source records separate and merge only when your policy has sufficient evidence.

Reconcile conflicts with explicit policy

A deterministic policy is more valuable than a universal ranking. Your priorities will depend on page type and downstream needs, but a typical policy might be:

1. Prefer an article-scoped JSON-LD date that agrees with a visible, header-area semantic date.
2. Otherwise prefer a clearly labeled visible or semantic date associated with the article header.
3. Use Open Graph as corroboration or a fallback when its value is internally consistent.
4. Store HTTP Last-Modified separately rather than promoting it to an editorial date.
5. If credible candidates conflict, return the selected value and a conflict status with all alternatives.

For canonical URLs, resolve relative links against the final response URL, normalize only safe syntactic differences, and retain the original declaration. Do not discard a canonical candidate just because it crosses hostnames—but lower confidence or flag it if it conflicts with redirects, page identity, or other facts your policy considers material.

Confidence should be explainable. Rather than a mysterious score of 0.83, store reasons such as:

• article JSON-LD and visible time agree
• Open Graph date conflicts by 24 hours
• candidate occurs in related-content region
• header timestamp is later than editorial modification date

This turns bad metadata from a silent data-quality problem into a reviewable result.

Test with disagreement, not ideal pages

Selector tests built from perfect examples create a false sense of reliability. Build fixtures for cases such as:

• a publication date and a later update date;
• an event date in the article body;
• two JSON-LD article nodes;
• an organization author plus individual contributors;
• a canonical URL that differs from the fetched URL;
• malformed structured data alongside valid Open Graph tags;
• no timezone, conflicting timezones, and date-only values;
• an HTTP Last-Modified date newer than the editorial update date.

Assertions should check both selection and provenance. It is not enough to assert that the chosen date equals a value; assert that the system retained competing candidates, their locators, and the decision reason.

A limited PagePith demonstration

The supplied PagePith proof shows a successful fetch of the WHATWG HTML text-level semantics page. The result reported the title “HTML Standard”, a content length of 151,834, and a Markdown excerpt containing the text-level-semantics table of contents, including the time section.

That is useful evidence of source retrieval: a pipeline can retain the requested URL, returned title, fetched content, and an excerpt that helps identify relevant material. It is not proof that PagePith extracted publication dates, authors, canonical URLs, JSON-LD, response headers, or conflict scores from that page. Those capabilities are not established by the supplied proof and should be validated against your own pages and requirements.

Make provenance part of the contract

The final output should never be just:

{ "author": "A. Writer", "date": "2026-08-24" }

Instead, return selected values, candidate lists, raw values, locations, parsing outcomes, retrieval time, and conflict information. That record is what lets research tools cite evidence, monitoring systems explain changes, and indexing pipelines improve policies without re-fetching every historical page.

Reliable web metadata extraction is not about discovering the one correct selector. It is about representing uncertainty faithfully, preferring corroborated evidence, and making every decision reproducible.

Ready to test a source-retrieval workflow against the pages you care about? Sign up for PagePith.