How to Ground an AI Agent’s Answer in the Exact Page State It Used

A web-connected agent is only as trustworthy as its evidence trail. If an answer cites a URL but cannot show which representation of that URL it retrieved, when it retrieved it, and which passage supported the wording, the citation is closer to a reading suggestion than an audit record.

This matters because a web resource is not necessarily one fixed document. HTTP selects a representation through request and response semantics, and content negotiation can produce different representations for different requests. A changing page can also have a different state tomorrow than it had at retrieval time. RFC 9110 and the W3C Web Annotation Data Model both provide useful concepts for treating state as part of the target.

The practical design shift is simple:

Treat each retrieval as an immutable evidence object, not as a URL plus some chunk text.

That object should remain attached to the extracted passage, the model context, generated claims, and audit log. This is the foundation for teams that need to ground AI agents with live web data without losing the ability to explain what the agent actually used.

Why a URL is not sufficient grounding

Consider an agent asked, “What is the current return policy?” It fetches https://example.com/returns, extracts one paragraph, and answers with a link to the page.

Several things can go wrong:

• The page changes after retrieval.
• The server supplies a different representation based on language, Accept headers, cookies, or other request context.
• JavaScript modifies the visible content after the original HTTP response.
• The agent cites the whole page even though only one sentence supported one part of the answer.
• The model generates a detail that is plausible but absent from the extracted passage.

A retrieval timestamp helps, but it is not a permanent version identifier. RFC 7089 distinguishes an original resource from a resource state at a specific time; the Annotation model similarly includes time- and request-header-based state descriptions. If historical versions are not available, you may not be able to reproduce an old page from its URL alone.

Therefore, preserve both what you requested and what you received. The goal is not to promise perfect reconstruction of every third-party website. The goal is to make your system’s own record precise enough to verify the evidence it captured.

Define an immutable evidence object

An evidence object represents one captured page state and one or more passages derived from it. Assign it an immutable ID that is separate from the URL. URLs are identifiers for resources; evidence IDs identify the artifact your agent used.

A compact record might look like this:

{
  "evidence_id": "ev_01JQ8A7K9F3X",
  "captured_at": "2026-09-07T12:00:42.802Z",
  "request": {
    "method": "GET",
    "requested_url": "https://www.rfc-editor.org/rfc/rfc9110.html",
    "representation_headers": {
      "accept": "text/html",
      "accept-language": "en"
    }
  },
  "response": {
    "final_url": "https://www.rfc-editor.org/rfc/rfc9110.html",
    "status": 200,
    "content_type": "text/html",
    "etag": "<when supplied>",
    "last_modified": "<when supplied>"
  },
  "acquisition": {
    "mode": "http_fetch",
    "javascript_executed": false
  },
  "artifact": {
    "stored_bytes_sha256": "<digest of stored bytes>",
    "extracted_markdown_sha256": "<digest of extracted text>"
  },
  "passage": {
    "passage_id": "psg_01JQ8A7M2R",
    "exact": "The selected supporting passage.",
    "prefix": "Text immediately before the passage.",
    "suffix": "Text immediately after the passage.",
    "start": 1842,
    "end": 1874
  }
}

Do not manufacture missing values. If the server did not send an ETag, store null; if your capture did not use a browser, do not imply that it represents the rendered page. Completeness is valuable, but honest absence is better than inferred provenance.

Capture the selected representation

Store the requested URL and the final URL after redirects, status code, content type and encoding, plus request and response headers that influenced representation selection. That last point is easy to overlook: a captured response is the selected representation for a particular request, not a universal truth for every visitor. This model follows HTTP’s representation semantics in RFC 9110.

Avoid retaining every header by default. Cookies, authorization fields, and identifiers may create security or privacy problems. Instead, define an allowlist of representation-relevant fields, redact secrets, and separately record a policy version explaining the redaction decision.

Preserve validators and integrity information

When available, record ETag and Last-Modified. HTTP defines these as validators that can help differentiate or characterize representations. On refresh, use conditional requests such as If-None-Match and If-Modified-Since where appropriate, then record whether the server reported no change or sent new content. See RFC 9110.

Also calculate a digest over the bytes you actually store. A local SHA-256 digest is enough to detect accidental or unauthorized changes to your retained artifact, provided the digest is protected by the same audit controls as the artifact metadata. If the response provides standard digest fields, preserve them as received and validate them when your implementation supports the algorithm. RFC 9530 defines Content-Digest and Repr-Digest for validating message content and representation data.

A digest does not prove that a page was correct, authoritative, or unchanged on the publisher’s server. It proves that a retained artifact still matches the bytes that your system recorded.

Anchor the claim to an exact source span

A document-level citation is often too broad. The answer should point to the passage that supports each material claim.

Store two complementary anchors:

1. Text position: start and end offsets within a normalized extraction.
2. Text quote: the exact text, with short prefix and suffix context.

The Web Annotation Data Model specifies TextPositionSelector and TextQuoteSelector patterns that map directly to this design. Position offsets are efficient but can break if extraction rules evolve. Quote matching is more resilient to offset shifts but can become ambiguous in repeated text. Keeping both allows a verifier to locate the original span in the stored extraction and to re-anchor it if the representation is processed again.

Normalize deliberately. For example, define whether offsets count Unicode code points, UTF-16 code units, or bytes; whether whitespace is collapsed; and whether navigation text is removed. Store the extraction pipeline version alongside the output. Without that detail, start: 1842 has no stable meaning.

For an answer with multiple claims, require claim-to-passage links rather than attaching one list of sources to the entire response:

{
  "answer_id": "ans_01JQ8B0Q",
  "claims": [
    {
      "claim_id": "clm_1",
      "text": "The server returned the selected representation at capture time.",
      "evidence_ids": ["ev_01JQ8A7K9F3X"],
      "passage_ids": ["psg_01JQ8A7M2R"],
      "support_status": "supported"
    }
  ]
}

This does not force every sentence to quote a source. It does force your agent to distinguish sourced facts from explanations, uncertainty, and recommendations.

Model the full provenance chain

Evidence becomes more useful when it records transformations, not just inputs. A raw response becomes a parsed document; the document becomes extracted text; text becomes chunks; chunks may be reranked; selected chunks become model context; model context contributes to an answer.

The W3C PROV-O ontology offers a clear vocabulary for this: page snapshots and passages are entities, retrieval and extraction are activities, and your service, model run, or user-directed agent can be agents. Relations such as used, wasGeneratedBy, and wasDerivedFrom are a natural fit.

You do not need to deploy a graph database on day one. A relational audit schema with immutable identifiers and parent references is sufficient:

retrieval_run -> captured_representation -> extracted_document
             -> passage -> retrieval_context -> answer_claim

At each arrow, capture the actor, time, software or prompt version where relevant, and input/output IDs. This makes it possible to answer operational questions such as: “Which answers used this now-superseded evidence?” or “Which extraction version produced this passage?”

Treat rendered pages as a separate acquisition mode

Raw HTTP and rendered browser content are not interchangeable. A raw fetch captures server-delivered content. A browser capture can include JavaScript-created DOM, client-side routing, locale behavior, consent state, and viewport-dependent layout.

Represent these as different evidence modes:

• http_fetch for response bytes and server-side extraction.
• rendered_dom for a DOM captured after a defined browser lifecycle event.
• visual_capture for an image-based page state when visual presentation is the evidence.

For rendered modes, store browser or renderer identity, viewport, locale, user agent, JavaScript status, capture timing rule, and any permitted session assumptions. The Web Annotation model’s treatment of resource state and rendering context supports recording this distinction rather than collapsing every result into a generic “web page.” Web Annotation Data Model

Generate with evidence IDs, then validate claims

Retrieval-augmented generation made external passages a useful conditioning mechanism, while also surfacing provenance as an unresolved challenge. The original RAG work is a useful reminder that retrieval alone is not the same as attribution.

Pass stable evidence and passage IDs into the model context. Instruct the model to attach one or more IDs to each factual claim and to say when the evidence is insufficient. Then run a second validation step that compares each claim to the cited spans.

Use a small, explicit status vocabulary:

• Supported: the cited span directly entails the claim.
• Partially supported: the span supports only part of the claim or requires a clearly labeled inference.
• Unsupported: cited material does not justify the claim.
• Uncited: no evidence ID is attached.

Do not treat model-generated citations as proof. Research on answer attribution notes that self-citation can be unfaithful and argues for finer-grained attribution tied to the retrieved context. Model Internals-based Answer Attribution and LAQuer both reinforce the value of mapping output back to localized source segments.

A production policy can be straightforward: remove unsupported factual claims, visibly qualify partially supported ones, and retain the validation result in the answer audit record.

Build the evidence layer before the agent becomes a dependency. Capture page state, spans, and IDs from the first retrieval onward. Create a PagePith account to explore a workflow for live web extraction.

An honest PagePith demonstration from the supplied proof

The supplied PagePith proof shows a fetch retrieval of RFC 9110. It reports the page title as “RFC 9110: HTTP Semantics,” a content length of 554021, and a Markdown excerpt beginning with the RFC header and abstract.

That is enough to illustrate the first layer of an evidence object: requested URL, retrieval tier, extracted representation, and a captured excerpt. It does not demonstrate rendered-browser capture, HTTP response headers, redirect handling, validators, response digests, exact source-span selectors, or claim-level attribution. Those fields should therefore remain absent or unknown in any record created solely from this proof.

The lesson is practical: record what your tool actually observed, then add the metadata your architecture captures. Never fill provenance gaps with assumptions.

A practical rollout plan

Start narrow and make every stage durable:

1. Persist raw retrieval metadata and extracted text with immutable IDs.
2. Add a digest and record ETag and Last-Modified whenever received.
3. Create passage records with quote, prefix, suffix, offsets, and extraction version.
4. Pass passage IDs through retrieval, reranking, prompting, and generated claims.
5. Validate support before displaying citations to users.
6. Add rendered capture only where raw HTTP cannot represent the relevant evidence.

This design will not stop pages from changing or models from making mistakes. It gives your system a disciplined way to detect changes, limit unsupported statements, and show the exact material behind an answer.

When an agent must answer from the live web, grounding is not a link at the end of a paragraph. It is a preserved chain from the request that selected a page state to the source span that supports a claim.

Start building an auditable live-web evidence workflow with PagePith.