How to Extract Data from Similar Pages with Different Content Types

Many websites reuse a shell: the same header, breadcrumb, sidebar, footer, and content card appear on every URL. That visual consistency is useful, but it can hide a data-model problem. A blog post, author profile, product page, and documentation reference may all look nearly identical while carrying fundamentally different fields.

If you apply one global selector map to this site, results tend to fail quietly. A selector intended for an article author might capture a product brand. A price may be stored as a normal paragraph on one page and machine-readable metadata on another. A documentation page may have many headings and code blocks but no publication date at all.

The reliable approach is not a separate scraper for every URL. It is a layered extraction pipeline that first identifies what the page represents, then extracts a common set of fields plus an optional type-specific payload.

Start with content type, not CSS selectors

Treat content_type as an extraction result. It should be present in every record, even when its value is unknown.

This follows the way structured-data vocabularies distinguish entities. Articles commonly describe a headline, authors, images, and publication dates. Profile pages focus on a primary person or organization. Product pages can include offers, availability, reviews, and ratings. Google documents these as distinct structured-data use cases rather than variations of one universal record shape. Article structured data, ProfilePage structured data, and Product structured data are useful references for the fields that differ.

A simple classifier can combine several signals:

1. JSON-LD @type values such as Article, ProfilePage, Person, Product, or TechArticle.
2. Microdata itemtype and itemprop attributes.
3. URL conventions such as /products/, /authors/, or /docs/.
4. Page landmarks and visible cues: price controls, an author card, a version picker, or an article byline.
5. Metadata such as Open Graph type, only as a supporting signal.

Avoid treating any one signal as unquestionable. A product page might embed an Organization entity for the seller, while an article might include Person objects for several authors. Classification should score evidence, select a primary type, and retain the signals that led to that decision.

Model a stable envelope and typed payloads

A single flat database table becomes awkward quickly. Either it fills with irrelevant nullable columns, or fields acquire vague meanings: does author mean a writer, an account owner, or a manufacturer?

Instead, use a stable envelope for fields meaningful across page types and nested objects for type-specific data.

type Extraction = {
  url: string;
  canonicalUrl?: string;
  contentType: "article" | "profile" | "product" | "documentation" | "unknown";
  title?: Field<string>;
  description?: Field<string>;
  image?: Field<string>;
  extractedAt: string;
  article?: {
    authors?: Field<string[]>;
    datePublished?: Field<string>;
    dateModified?: Field<string>;
    body?: Field<string>;
  };
  profile?: {
    name?: Field<string>;
    role?: Field<string>;
    organization?: Field<string>;
    links?: Field<string[]>;
  };
  product?: {
    name?: Field<string>;
    sku?: Field<string>;
    price?: Field<string>;
    currency?: Field<string>;
    availability?: Field<string>;
  };
  documentation?: {
    version?: Field<string>;
    headings?: Field<string[]>;
    codeBlocks?: Field<string[]>;
  };
};

type Field<T> = {
  value: T | null;
  state: "present" | "missing" | "unsupported" | "ambiguous";
  source?: "jsonld" | "microdata" | "dom" | "metadata";
  confidence?: "high" | "medium" | "low";
};

This design has two practical advantages. First, consumers can depend on url, contentType, and normalized shared fields regardless of the page type. Second, the absence of product.price does not look like a failed article extraction: it is explicitly unsupported for an article, or missing for a product where no price was found.

Keep arrays as arrays. Multiple authors, images, tags, ratings, headings, or code blocks are meaningful repeated values, not formatting mistakes. Article examples can include repeated authors and images, and the underlying page may legitimately contain several values for a property. Google’s Article guidance illustrates these type-specific, sometimes repeated fields.

Use a layered source strategy

For each field, resolve sources in a deliberate order. A typical precedence order is:

1. JSON-LD
2. Microdata
3. Scoped rendered or static DOM
4. Page metadata
5. Carefully validated heuristic fallback

Do not replace the entire output just because one layer is incomplete. A page can have an excellent JSON-LD headline and author list but omit dateModified; the visible time[datetime] element can fill only that missing field.

Parse JSON-LD as a graph, not a single object

JSON-LD is a JSON serialization for linked data, standardized by the W3C. See JSON-LD 1.1. In practice, a page can include multiple application/ld+json scripts, arrays of nodes, or a graph under @graph.

Your parser should therefore:

• Parse every JSON-LD script independently and capture parse errors.
• Expand top-level arrays and @graph values into candidate nodes.
• Normalize @type to an array before matching.
• Preserve identifiers and relationships instead of immediately flattening nested objects.
• Resolve references where your implementation supports them.

The hard part is selecting the primary entity. Do not merge every detected object into one record. A page may declare its publisher, breadcrumb list, website, author, and page subject simultaneously. Schema.org defines mainEntity as the primary entity described by a page, making it a strong signal when present.

When mainEntity is unavailable, score candidates using canonical URL alignment, title or h1 similarity, expected page type, and whether the candidate has fields appropriate to that type. Preserve the runner-up candidates if ambiguity remains rather than silently choosing a plausible but wrong product or person.

Use Microdata as a structured fallback

Not every site publishes JSON-LD. Microdata provides another structured source through attributes including itemscope, itemtype, itemprop, itemid, and itemref. The HTML Living Standard’s Microdata section defines these mechanics, including nested items and repeated properties.

A robust Microdata extractor should recursively collect itemprop values and use the element’s appropriate machine-readable value where applicable:

• content for metadata-like elements
• href for links
• src for media
• datetime for time elements
• nested item objects for nested itemscope elements
• normalized text as a final fallback

This matters because a visually correct text scrape can lose the actual canonical URL, publication timestamp, or image source encoded in the element attributes.

Build the pipeline before multiplying selectors. PagePith’s homepage presents a /v1/api/scrape endpoint and describes a single API for website, PDF, and social-video content, with rendering and extraction among the listed steps. If you want to evaluate that workflow against your own URLs, sign up for PagePith.

Scope DOM extraction to the content root

DOM extraction is indispensable, but global selectors are risky on repeated layouts. A global .author query can match navigation, a recommended article card, and the primary byline. Find the primary content container first, then query within it.

const root = document.querySelector(
  "main, [role='main'], article, [itemprop='mainEntity'], .content"
);

const headings = root
  ? [...root.querySelectorAll("h2, h3")].map((node) => node.textContent?.trim())
  : [];

The DOM selector APIs distinguish between querySelector(), which returns the first match or null, and querySelectorAll(), which returns all matches. That distinction is central to repeated values such as tags, headings, gallery images, and authors. MDN’s selector and traversal guide covers these behaviors.

Prefer semantic hooks before presentation classes: itemprop, role, accessible labels, stable data-* attributes, element semantics, and heading hierarchy. Framework-generated class names are often an implementation detail rather than a durable extraction contract. If dynamic identifiers must be queried, remember that selector input has CSS syntax requirements; MDN’s querySelector() documentation notes that invalid identifier values may need escaping.

Render only when the static response is insufficient

Static HTML is a sensible first pass because it is simpler to fetch and parse. But some sites send an application shell first and insert the useful content only after JavaScript executes. Lazy-loaded sections can create a similar issue.

Escalate to rendered-page extraction when one or more conditions hold:

• the initial document has no usable main content;
• required structured data is absent and the page has strong client-rendering indicators;
• the document contains placeholders but no resolved values;
• required images, prices, or sections appear only after interaction or load;
• static and rendered content produce an explainable discrepancy.

Google’s explanation of JavaScript SEO basics describes the difference between initial HTML and rendered HTML and why JavaScript-generated and lazy-loaded content requires inspection. Treat rendering as a targeted escalation rather than a default for every URL, and record which representation produced each field.

Resolve field conflicts with provenance

Extraction is not only about values. It is also about why you trust those values.

For every resolved field, retain:

• source layer (jsonld, microdata, dom, or metadata)
• raw candidate value
• normalized value
• parser or selector used
• content root or JSON-LD block identifier
• confidence level
• conflicts and rejected alternatives

For example, a record might contain headline = "Release Notes 4.2" from JSON-LD at high confidence while dateModified comes from a scoped time[datetime] element at medium confidence. If the title metadata conflicts with the visible h1, retain both candidates and mark the selected result as ambiguous when no rule can safely choose one.

This evidence makes schema drift diagnosable. When a site redesign changes a selector, you can see whether structured data still worked, whether the content root changed, and whether a fallback began producing a different value.

Test the matrix, not one representative page

A scraper can succeed on the first page you test and still fail across the site. Build a fixture matrix with examples of every known type and difficult variation:

Case	What to verify
Article with multiple authors	Arrays remain intact; byline is not confused with a sidebar author card.
Profile with organization data	The primary profile entity wins over publisher metadata.
Product with no offer	price is missing, not an invented zero or empty string.
Documentation page	Headings and code blocks are collected without forcing article dates.
JavaScript-heavy page	Rendering is triggered only when the static result lacks required content.
Conflicting values	Provenance and ambiguity state are retained.

Run these fixtures whenever you modify a selector, parser, normalization rule, or classifier. Also keep raw source snapshots where permitted by your retention policy; normalized output alone is rarely enough to explain an extraction regression.

A practical extraction sequence

The complete workflow can be expressed as a compact sequence:

1. Fetch static HTML and capture response metadata.
2. Parse JSON-LD and Microdata into candidate entities.
3. Identify the primary entity and classify page content type.
4. Locate a primary content root for DOM-based evidence.
5. Resolve common fields independently through the source order.
6. Resolve the selected type’s optional payload fields.
7. Escalate to rendered extraction only when required evidence is absent.
8. Validate formats and visible-page consistency.
9. Emit values, states, confidence, provenance, warnings, and raw evidence references.

The key is that the page layout is only one input. A repeated template should lead to shared crawling and shared evidence collection, not an assumption that every URL has the same data contract.

When you classify first, preserve type-specific shapes, and make fallbacks field-level and observable, one pipeline can handle similar website pages without flattening their meaningful differences.

Ready to test a URL-based extraction workflow in your own stack? Sign up for PagePith.