How to Give AI Agents Structured Access to Web Pages Without Overfetching

AI agents need web access, but “give the model the page” is rarely a sufficient tool contract. A complete HTML document can contain navigation, repeated UI, scripts, legal copy, unrelated recommendations, and thousands of tokens of content that have nothing to do with the task.

The alternative is not blindly compressing every page into a tiny summary. An agent may need a product’s price and availability, the rows of a comparison table, a specific documentation section, or links from a filtered result list. The right design is structured web access for AI agents: page-specific operations that return bounded, typed, task-relevant results.

This is primarily a tool-design problem. Define what an agent can ask for, constrain what it receives, validate the result, and provide an explicit escalation path when the compact view is insufficient.

Why full-page retrieval is a weak default

An agent loop repeatedly processes system instructions, tool descriptions, conversation history, and prior tool results. Large, unbounded observations therefore have a compounding cost: they consume context, obscure relevant facts, and make it harder for the model to identify the next action. OpenAI describes this accumulation as context bloat and discusses output caps and deferred discovery as practical mitigations in agentic systems (OpenAI).

A raw page response also creates an ambiguous interface. If one tool returns HTML, another returns cleaned text, and a third returns prose generated by an extractor, downstream agent behavior becomes harder to test. The agent must infer structure that the tool could have exposed directly.

For example, consider an agent answering: “Which plans include single sign-on?” A full document dump requires it to:

1. Locate the pricing content among global navigation and footer links.
2. Identify the plan comparison table.
3. Match feature rows to plan columns.
4. Preserve the source URL and enough evidence to support its conclusion.

A better contract makes the task explicit:

page.get_table(url, selector_hint="pricing", max_rows=30)

The response should contain the requested table, a stable representation of its headers and cells, source location information, and an indicator of whether truncation occurred. The model spends its context on reasoning, not reconstructing a page’s layout.

Treat web access as a set of bounded operations

Avoid a universal fetch_page(url) tool that always returns the same bulky representation. Instead, model the operations agents actually perform. A practical baseline includes:

• Inspect page metadata: title, canonical URL, description, language, publication signals, and available representations.
• List semantic regions: major sections, headings, tables, forms, and content blocks with identifiers.
• Extract a section: content under a heading or a supplied region ID.
• Retrieve a table: normalized headers, rows, captions, and source location.
• Enumerate links: filtered by host, text, URL pattern, or region.
• Query repeated items: cards, products, search results, articles, or listings with pagination.
• Request bounded detail: a selected item’s fields, a limited text range, or a raw fallback for a named region.

This decomposition gives each call a clear purpose and lets output limits match the operation. Guidance on agent tools recommends pagination, range selection, filtering, and truncation for outputs that can become large (Anthropic).

Design inputs that make scope explicit

Every potentially large operation should expose limits as normal parameters—not hidden implementation behavior. For example:

{
  "url": "https://docs.example.com/api/authentication",
  "section": "Rate limits",
  "include": ["text", "links"],
  "maxChars": 8000,
  "linkLimit": 20
}

Useful input controls include:

Control	Purpose
cursor and limit	Page through repeated results predictably.
offset and length	Read a bounded text range.
include	Request only fields needed for the next decision.
filter	Restrict links, items, rows, or regions before retrieval.
maxChars / maxRows	Set an understandable response budget.
regionId / section	Target a previously discovered part of the page.
mode	Choose a summary, semantic, detailed, or raw representation.

Do not silently return an arbitrary beginning of the page when a limit is reached. Return truncated: true, explain which limit was applied, and provide a continuation mechanism. That makes an incomplete response observable and actionable.

Make output schemas part of the product contract

A web tool should return machine-readable data, not only an HTML string or a narrative description. JSON Schema is a useful baseline; the JSON Schema project identifies version 2020-12 as its current specification version (JSON Schema).

For MCP integrations, an outputSchema can advertise the expected structured result. MCP SDK documentation describes structuredContent as application-facing data and documents client-side validation against the advertised schema (MCP Python SDK, MCP TypeScript SDK).

Here is a simplified response contract for a section extraction operation:

{
  "type": "object",
  "required": ["url", "title", "section", "blocks", "truncated"],
  "properties": {
    "url": { "type": "string", "format": "uri" },
    "title": { "type": "string" },
    "section": {
      "type": "object",
      "required": ["id", "heading", "level"],
      "properties": {
        "id": { "type": "string" },
        "heading": { "type": "string" },
        "level": { "type": "integer", "minimum": 1, "maximum": 6 }
      }
    },
    "blocks": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["kind", "text"],
        "properties": {
          "kind": { "enum": ["paragraph", "list", "table", "code"] },
          "text": { "type": "string" }
        }
      }
    },
    "truncated": { "type": "boolean" },
    "nextCursor": { "type": ["string", "null"] }
  }
}

Schema fidelity matters because agents rarely operate alone. A planner may call an extractor; a validation step may consume selected fields; a UI or audit logger may present the evidence later. A stable schema gives each stage a testable boundary.

When model-facing compatibility matters, return a compact textual rendering alongside the structured object. The MCP Python SDK explicitly distinguishes structured data for application code from content intended for model consumption (MCP Python SDK). Keep that text representation small and derived from the structured response so the two cannot drift.

Extract semantic signals before scraping arbitrary text

Not every page offers clean machine-readable metadata, but many provide useful signals. JSON-LD can be embedded in HTML in application/ld+json script blocks (W3C JSON-LD 1.1), and Schema.org defines shared vocabularies for common web entities and properties (Schema.org).

A robust extractor can prioritize evidence in roughly this order:

1. HTTP and document metadata, including canonical URL and language.
2. JSON-LD and other structured page data.
3. Semantic HTML: headings, tables, lists, forms, labels, and links.
4. Rendered content regions and repeated components.
5. Raw HTML only when a bounded semantic representation cannot preserve what the task needs.

This is a prioritization strategy, not a guarantee of truth. Publisher markup can be incomplete, stale, or inconsistent with visible content. Preserve the originating URL and region context so an agent or downstream application can inspect conflicts rather than treating extracted fields as unquestionable facts.

Build the contract before the crawler. If your team is exploring a structured web layer for agent workflows, sign up to evaluate PagePith for your own pages and tool boundaries.

Use progressive disclosure rather than one fixed representation

The smallest output is not always the best output. Research on web-agent observations reports that the preferred representation can vary by model capability and reasoning budget; compact accessibility-oriented representations can help in some settings, while detailed HTML and layout information can matter in others (Read More, Think More).

That argues for staged access:

1. Discover

Return inexpensive page identity and available regions:

{
  "title": "Authentication guide",
  "canonicalUrl": "https://docs.example.com/auth",
  "regions": [
    { "id": "overview", "heading": "Overview" },
    { "id": "rate-limits", "heading": "Rate limits" },
    { "id": "errors", "heading": "Errors" }
  ],
  "tables": [],
  "truncated": false
}

2. Select

Request one region, table, or filtered link set. Keep the initial detail limit conservative and report when it clips the result.

3. Expand

Use a cursor, range, or item identifier only if the selected representation leaves an unanswered question.

4. Escalate

Allow an explicit raw or higher-fidelity fallback for a bounded region. Record why it was requested and keep it separate from the default flow.

This pattern also applies to the tool catalog itself. Instead of exposing every possible extraction operation in every agent turn, offer a small discovery surface and load specialized tools only when the task requires them. Deferred discovery is one way to reduce always-present tool context (OpenAI).

A practical tool suite

A concise agent-facing API might look like this:

web.inspect(url)
web.get_region(url, region_id, include, max_chars, cursor)
web.get_table(url, table_id, row_filter, limit, cursor)
web.list_links(url, region_id, filter, limit, cursor)
web.get_item(url, collection_id, item_id, fields)
web.get_raw_region(url, region_id, max_chars, reason)

The names communicate scope. The parameters reveal cost and output boundaries. The result schemas make it possible to validate tool outputs before passing them to another component.

Keep failure states structured as well. Distinguish:

• not_found: the requested section, table, or item was absent.
• unsupported: the extractor cannot reliably represent that content type.
• blocked: access was denied or requires authentication.
• partial: a representation was produced but is truncated or incomplete.
• invalid_output: extraction completed but failed schema validation.

An empty array is not a sufficient substitute for these states. An agent needs to know whether “no links found” means there are no links, the page did not load, or the target region could not be identified.

Choose an integration boundary deliberately

Teams commonly have three implementation paths:

Custom extraction API

A direct API is suitable when the web workload is narrow and your application owns the orchestration. It offers maximal control over auth, caching, page policies, and data retention. The tradeoff is that every agent framework needs an adapter.

MCP server

An MCP server is useful when multiple MCP-capable clients should access the same tool contract. Its structured output model and declared schemas fit web extraction well, especially when client-side validation matters (MCP TypeScript SDK).

Framework integration

A framework-native tool can be the fastest path for one existing agent stack. Keep the underlying request and response contracts framework-neutral, however. That preserves the option to expose the same capability through an API or MCP later.

The decision should follow your integration surface, not marketing labels. In all three approaches, bounded outputs, explicit fallbacks, and validation are the durable design choices.

Evaluate preservation, not compression alone

A smaller response is useful only if it still supports the intended task. Evaluate structured web access with a representative set of pages and tasks, then track:

• Task completion and correctness.
• Recall of required fields, rows, and links.
• Preservation of canonical URLs and source locations.
• Schema-valid response rate.
• Truncation and raw-fallback frequency.
• Tool calls per successful task.
• Input and output tokens per task.
• Latency by operation and page type.

Add adversarial examples: a pricing table with missing cells, a page with duplicated headings, a long legal page, a client-rendered listing, and structured data that conflicts with visible text. These cases reveal whether your compact representation is genuinely reliable or merely small.

An honest PagePith demonstration

The supplied PagePith proof records a fetch of Anthropic’s article at https://www.anthropic.com/engineering/writing-tools-for-agents. The result is identified as fetch tier, reports the page title as “Writing effective tools for AI agents—using AI agents,” and records a content length of 24,608. Its included Markdown excerpt begins with the article’s discussion of MCP and then lists tool-design themes.

That proof demonstrates a concrete, inspectable page retrieval result with a title, a retrieval tier, a measured content length, and Markdown content. It does not establish a universal token reduction rate, automatic schema extraction, or outcome improvement for any particular agent workflow. Those properties should be tested against your pages, tasks, and output contracts.

Start by defining the smallest useful response for one high-value agent task. Give it a schema, an explicit limit, a continuation path, and a raw fallback. Then measure whether the agent still succeeds. That is how structured web access becomes dependable infrastructure rather than another large blob in the context window.

Ready to explore a more deliberate web-access workflow? Sign up for PagePith.