How to Extract Embedded JSON and Structured Metadata from Web Pages

Web pages often expose the records you need without displaying them as ordinary text. Product attributes, article authors, event dates, canonical URLs, and application state may live in structured metadata, non-executing data blocks, or JSON responses requested after the page loads.

The practical consequence is simple: do not begin by flattening the rendered page into text. Treat the page as a set of data layers, moving from the least expensive and most standardized source to the most dynamic one:

1. Server-delivered HTML
2. JSON-LD and standardized metadata
3. Generic embedded JSON payloads
4. Browser-rendered DOM
5. Observed XHR or fetch responses

This sequence gives you useful provenance, avoids unnecessary browser automation, and makes failures easier to diagnose.

Start with the document you received

Fetch the document and retain the original response body before extracting anything. The raw HTML is your evidence: it lets you reproduce a parse, compare server and rendered states, and inspect failures later.

An HTML script element is not necessarily executable JavaScript. The HTML Standard permits data blocks when a script has a non-JavaScript MIME type, which makes them useful containers for structured payloads that tools can read without executing page code. HTML Standard: the script element

At this stage, collect a small amount of context alongside every candidate value:

• Requested and final URL
• Fetch timestamp and HTTP status
• Response content type
• Source location, such as json-ld, open-graph, or script-state
• The raw candidate string or a content-addressed reference to it
• Parser result, parser error, and extraction version

That context prevents a common operational problem: a downstream record looks wrong, but no one can determine whether the site changed, the parser regressed, or the source itself was inconsistent.

Extract standards-based metadata first

JSON-LD

JSON-LD is usually the best first target because it is explicit JSON and often maps directly to a recognizable vocabulary. JSON-LD uses graph-oriented concepts such as @context, @id, @type, and @graph. JSON-LD specifications JSON-LD 1.1 Syntax

Look for scripts selected by script[type="application/ld+json"]. They may appear in either document head or body, and a page can contain more than one payload. Google's structured data introduction

Do not assume one script means one record. Valid payloads can be:

• One object
• An array of objects
• A graph container with an @graph array
• A nested object that contains entities relevant to your target type

Here is a deliberately strict Python starting point using an HTML parser and the standard JSON decoder:

import json
from bs4 import BeautifulSoup


def jsonld_candidates(html: str):
    soup = BeautifulSoup(html, "html.parser")

    for index, node in enumerate(
        soup.select('script[type="application/ld+json"]')
    ):
        raw = node.get_text(strip=True)
        if not raw:
            continue

        try:
            yield {
                "source": "json-ld",
                "script_index": index,
                "raw": raw,
                "value": json.loads(raw),
                "parse_error": None,
            }
        except json.JSONDecodeError as error:
            yield {
                "source": "json-ld",
                "script_index": index,
                "raw": raw,
                "value": None,
                "parse_error": str(error),
            }

Strict parsing is intentional. A JSON-looking block may actually be JavaScript, a templated fragment, or invalid data. Preserve failed candidates and continue scanning rather than allowing one bad block to discard every other record.

Next, normalize containers before applying a schema-specific mapper:

def iter_jsonld_nodes(value):
    if isinstance(value, list):
        for item in value:
            yield from iter_jsonld_nodes(item)
        return

    if not isinstance(value, dict):
        return

    graph = value.get("@graph")
    if isinstance(graph, list):
        for item in graph:
            yield from iter_jsonld_nodes(item)

    # Keep objects that represent an entity, including envelopes
    # that may carry an identifier or type outside a graph.
    if "@type" in value or "@id" in value:
        yield value

Your mapper can then select only types you expect, such as an article, product, organization, or event. Keep the original node with the mapped result. It is much easier to update a mapper when the source object remains available.

Open Graph, Microdata, and RDFa

JSON-LD is not the only structured layer. Open Graph uses document metadata fields such as og:title, og:type, og:image, and og:url. The Open Graph Protocol

A simple collector can preserve repeated properties rather than silently overwriting them:

from collections import defaultdict


def open_graph(html: str):
    soup = BeautifulSoup(html, "html.parser")
    values = defaultdict(list)

    for node in soup.select('meta[property^="og:"]'):
        key = node.get("property")
        value = node.get("content")
        if key and value:
            values[key].append(value)

    return dict(values)

Also inspect Microdata and RDFa when the document is structured through attributes instead of a JSON payload. Schema.org describes JSON-LD, Microdata, and RDFa as supported approaches, so a JSON-LD-only crawler will miss valid structured information on some pages. Getting Started with Schema.org

These formats are especially useful as corroborating sources. For example, a canonical URL from metadata, a title from Open Graph, and a name from JSON-LD may agree. If they conflict, retain all values and make the conflict visible to your pipeline rather than choosing an arbitrary winner.

Handle application state without guessing

Modern applications sometimes serialize initial state into a script block. The block may have an identifiable type or ID, but naming is framework- and site-specific. First collect likely data blocks by their MIME type, identifier, or unusually large text content. Then determine whether the content is strict JSON before parsing it.

Avoid broad regular expressions that try to extract an object from arbitrary JavaScript. Braces can occur inside strings, nested objects, and unrelated source code. If the page exposes only an assignment expression rather than JSON, prefer a browser-based inspection of the application state or observe the API response that supplied the data.

The same rule applies to entities embedded in a larger payload: define what you want before you flatten everything. A catalog-page extractor might emit one record per product and retain the page-level breadcrumb separately. An article extractor might select one primary article node while keeping author objects as linked supporting records.

Escalate to a browser only when HTML is insufficient

Some pages insert JSON-LD after load or construct their content entirely in the browser. Google notes that structured data can be injected dynamically with JavaScript. Generating structured data with JavaScript

When the initial HTML does not contain the target record, use a browser to answer two distinct questions:

1. Did rendering add useful structured elements or data blocks to the DOM?
2. Did the page request a JSON endpoint that already contains the record?

For the second question, collecting a JSON response is often less brittle than reading rendered cards, labels, and pagination controls. Playwright supports observing responses, checking request resource types such as XHR and fetch, and parsing response bodies as JSON. Playwright network documentation

from playwright.async_api import async_playwright


async def capture_json_responses(url: str):
    captured = []

    async with async_playwright() as pw:
        browser = await pw.chromium.launch()
        page = await browser.new_page()

        def record(response):
            content_type = response.headers.get("content-type", "")
            resource_type = response.request.resource_type
            if resource_type in {"xhr", "fetch"} and "json" in content_type:
                captured.append(response)

        page.on("response", record)
        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_timeout(1000)

        payloads = []
        for response in captured:
            try:
                payloads.append({
                    "url": response.url,
                    "status": response.status,
                    "json": await response.json(),
                })
            except Exception as error:
                payloads.append({
                    "url": response.url,
                    "status": response.status,
                    "json": None,
                    "parse_error": str(error),
                })

        await browser.close()
        return payloads

In production, replace the fixed wait with an event or response predicate tied to the data you expect. Filter endpoints carefully, redact secrets from logs, and do not turn browser network capture into a way to bypass authentication or access restrictions.

Build a repeatable evidence trail: If you want a place to start testing URL-based retrieval in your own workflow, sign up for PagePith.

A limited PagePith demonstration

The supplied PagePith proof provides one concrete example of why source-layer inspection matters. A fetch-tier request to the HTML Standard scripting page returned a document titled HTML Standard, with a reported content length of 131,915. Its supplied Markdown excerpt includes the navigation path to the scripting section and its script-element subsection.

That result demonstrates that PagePith retrieved the server-delivered document and surfaced content relevant to locating the script-element specification. It does not demonstrate browser rendering, JSON-LD extraction, API-response capture, automatic schema mapping, or a successful parse of a particular embedded payload. Those steps still need explicit extraction logic such as the workflow above.

The underlying reference is the HTML Standard scripting section, which is the appropriate source when deciding whether a script block should be treated as executable code or a data container.

Validate before publishing or storing records

Structured does not automatically mean correct. Markup can be stale, incomplete, or inconsistent with what a user sees. Google's structured-data policies emphasize that markup should represent the page's main content and should not be misleading or inaccurate. General structured data guidelines

Build validation at three levels:

1. Syntax validation: Record JSON decode failures, unexpected primitive values, and unknown shapes.
2. Schema validation: Enforce required fields and expected types for your internal record model. Normalize dates, URLs, prices, and arrays deliberately.
3. Semantic validation: Compare important extracted values with visible page content and related metadata. Flag conflicts instead of silently replacing one source with another.

Finally, operate within the site's access rules, authentication boundaries, terms, rate expectations, and applicable law. Cache responses when appropriate, identify your crawler where required, and use bounded concurrency. A technically successful extractor is not necessarily an operationally appropriate one.

Use the least dynamic layer that works

A resilient extractor is a small escalation ladder, not a single selector. Start with raw HTML and standardized metadata. Parse strict JSON with per-candidate error handling. Normalize arrays and graph containers. Escalate to a browser only for dynamic rendering, then prefer observed JSON responses over fragile visual scraping when they are legitimately available.

This approach produces records that are easier to validate, explain, and maintain as pages evolve.

Ready to test a URL-centric retrieval step in your process? Sign up for PagePith.