How to Build an Evaluation Set for AI Extraction from Real Web Pages

AI extraction is not adequately evaluated by asking whether a model returned plausible JSON on a handful of pages. To evaluate AI web extraction accuracy in a way that supports deployment decisions, teams need a repeatable system: representative pages, explicit extraction requests, verified expected outputs, and metrics that reveal why an output failed.

That is a different task from testing whether a scraper can download HTML. A model may retrieve the page successfully yet select the wrong region, omit one item from a repeated list, normalize a value incorrectly, or invent a value that is not present. A single aggregate “accuracy” score conceals each of those risks.

This guide describes a practical evaluation set design for extracting structured data from real, changing web pages.

Treat the eval as a dataset and a procedure

Start with a useful mental model: an evaluation is not just a folder of URLs and expected answers. It is a versioned dataset plus the instructions, grading logic, normalization rules, and reporting process used to assess a particular extractor version. This aligns with the evaluation design guidance in OpenAI Evals, which frames an eval as examples paired with reference answers or evaluators and repeatable runs.

Write an evaluation contract before you sample pages. It should answer:

• What pages are in scope? For example, product detail pages, job listings, event pages, documentation pages, or research articles.
• What is the extraction task? Define the fields, expected output shape, cardinality, and normalization rules.
• What input does the extractor receive? Rendered page, raw HTML, Markdown conversion, screenshot, or another representation.
• What counts as success? Exact string match may be appropriate for an ID, but a price may require numeric normalization and a list needs set-level scoring.
• What is the release gate? State the minimum quality thresholds and which slices must not regress.

This prevents a common failure mode: changing prompts, schemas, source representation, and graders simultaneously, then treating the resulting number as a meaningful comparison.

Build a corpus that reflects production traffic

Your corpus should represent the conditions in which the extraction system will operate. NIST recommends realistic test sets that reflect expected-use conditions, documented methodology, and disaggregated results rather than one undifferentiated score (NIST AI RMF: Accuracy and Robustness).

The practical implication is to stratify page selection. Do not let a large set of clean articles drown out the difficult pages that matter to your application.

Create a sampling matrix

Define the dimensions that plausibly affect extraction behavior. A typical matrix includes:

Dimension	Example buckets
Page type	Product, article, listing, profile, event, table-heavy document
Layout complexity	Single-column, multi-column, repeated cards, tabs, accordions
Site family	Major domains, long-tail domains, template variants
Field shape	Scalar text, date, currency, nested object, multi-value list
Cardinality	Exactly one, optional one, zero or more, one or more
Content condition	Complete, missing field, conflicting values, stale-looking content
Render condition	Server-rendered, client-rendered, content behind interaction

Sample from each meaningful cell, with enough examples to expose a pattern. The point is not statistical perfection on day one. It is ensuring that known hard cases cannot disappear inside a favorable average.

Research on multi-type web extraction supports this approach: WCXB covers multiple page types and reports that systems can appear strong on articles while showing wider variation on more structured pages. That is a useful warning against article-only test sets.

Include intentionally negative examples as well. If the requested field is absent, the correct answer is not a guessed substitute. If a page contains multiple prices, an extractor should follow the task definition instead of selecting a visually prominent number.

Preserve the source state

For every example, retain enough evidence to reproduce a judgment:

• Canonical URL and capture timestamp
• Domain, page-type, and template labels
• Input artifact used by the system under test
• Screenshot or rendered capture when visual position matters
• Content hash and extraction pipeline version
• Notes about interaction, localization, authentication, or dynamic content

URLs alone are not stable test fixtures. Websites change markup, reorder modules, and replace content. LiveWeb-IE specifically examines the gap between static benchmark performance and live pages, including degradation when site structures evolve. Keep frozen captures for reproducibility, then separately recapture a live subset to measure drift.

Define labels at the field level

A page-level label such as “correct” is too coarse for extraction. Each evaluation example should state what to retrieve and how to interpret it.

Use a compact record like this:

{
  "example_id": "product-042",
  "page_type": "product_detail",
  "request": "Extract the product name, current price, currency, and available sizes.",
  "schema": {
    "name": "string, required",
    "price": "number, required",
    "currency": "ISO 4217 string, required",
    "sizes": "array of strings, zero or more"
  },
  "gold": {
    "name": "Trail Jacket",
    "price": 129.0,
    "currency": "USD",
    "sizes": ["S", "M", "L"]
  },
  "acceptable_variants": {
    "name": ["Trail Jacket", "Trail Jacket®"]
  },
  "evidence": {
    "name": "visible product heading",
    "price": "current-price region"
  }
}

The schema should specify cardinality because it changes the grading question. A request for “the author” expects one value; “all authors” requires completeness as well as correctness. For dates, currencies, units, names, and titles, declare your canonical representation and any accepted variants up front.

This attribute-level approach resembles the query construction used in LiveWeb-IE, which varies requests by attribute count and cardinality. It makes an evaluation set useful across simple and compound extraction tasks rather than only one fixed schema.

Produce gold labels with evidence and review

Gold labels deserve engineering discipline. A label that is ambiguous, outdated, or silently normalized inconsistently will create false regressions and false confidence.

A workable labeling flow is:

1. Draft labels from the captured page and task specification.
2. Attach evidence for each nontrivial field: visible text, a region description, or an annotated source snippet.
3. Run deterministic checks for parseability, required fields, valid enums, duplicate values, and schema conformance.
4. Review disagreements with a second annotator or domain reviewer.
5. Record adjudications when reasonable reviewers disagree, including why a variant is accepted or rejected.

For a larger set, automation can speed the drafting and verification stages, but it should not erase accountability. WCXB describes a multi-stage process that combines model assistance, automated verification, review passes, scripts, and human review. The lesson is not that every team needs the same pipeline; it is that labels should be checked through more than one mechanism.

Mid-project checkpoint

Before expanding the corpus, run a pilot across every planned page type. Review the first failures manually. If evaluators cannot explain the expected output and evidence for a field, revise the schema and labeling guidance before producing hundreds of examples.

If you are building a workflow around real-page inputs, create a PagePith account to explore the product in your own evaluation process.

Measure correctness, support, and validity separately

A useful report distinguishes three questions:

1. Was the extracted value correct?
2. Was it supported by the page?
3. Was the output usable by downstream systems?

For multi-value fields, calculate precision, recall, and F1:

• Precision: of values returned, how many are correct?
• Recall: of gold values, how many were returned?
• F1: the harmonic mean of precision and recall.

For scalar fields, use a field-level exact or normalized match rate. Then separately count:

• Omissions: a required gold value was not returned.
• Unsupported values: a returned value is absent from the source or does not satisfy the request.
• Wrong substitutions: a nearby but incorrect value was selected.
• Format failures: invalid JSON, wrong type, invalid enum, unparsable date, or schema violation.

This separation matters because an extractor can have respectable exact-match results while still producing unsupported values often enough to damage trust. NEXT-EVAL reports precision, recall, F1, and hallucination-oriented measures, illustrating why extraction correctness and unsupported output should not be collapsed into one number.

Report every metric by slice: page type, domain family, field, cardinality, complexity, and capture age. An overall F1 is a headline, not a diagnosis.

Use an error taxonomy that leads to fixes

When an example fails, assign one primary failure category and optional secondary categories. Keep the taxonomy small enough that reviewers use it consistently.

A practical starting taxonomy is:

Failure category	Example
Request interpretation	“Current price” is treated as the original list price
Target attribute identification	The system extracts a rating instead of review count
Region grounding	It uses a recommendation card rather than the primary record
Element selection	It chooses the first repeated item, not the requested one
Completeness	It returns two of four listed authors
Normalization	It drops a unit or changes a date incorrectly
Unsupported extraction	It returns a plausible value not present on the page
Serialization	The values are right but the schema is invalid
Source or capture issue	The relevant content did not render in the test artifact

The categories mirror the staged view of extraction in LiveWeb-IE: interpreting attributes, locating relevant page regions, pinpointing elements, and producing the final output can each fail independently.

This taxonomy makes regression review efficient. A rise in serialization failures suggests schema or prompt changes. A rise in region-grounding failures after recapture suggests layout drift. A cluster of normalization issues points toward deterministic post-processing or clearer acceptance rules.

Split, version, and recapture the set

Use three partitions:

• Development: examples engineers may inspect while iterating.
• Validation: examples used to tune prompts, tools, or thresholds without repeatedly touching the final test set.
• Held-out test: examples reserved for release decisions.

Avoid splitting near-duplicates across partitions. Pages from the same template family can leak layout knowledge, making held-out results overly optimistic. Maintain similar page-type distributions across splits, but reserve some domains and structural variants exclusively for testing.

Version the dataset as carefully as the extractor. A version should identify the page captures, schemas, labels, accepted variants, grader code, and taxonomy. When any of these change, document the reason and decide whether to rerun the prior extractor for an apples-to-apples comparison. The OpenAI Evals guidance recommends explicit versioning when an eval changes.

Also validate the grader itself. If a model-based evaluator is used for fuzzy equivalence, compare its decisions against human-labeled cases before using it as a release gate. Meta-evaluation is necessary because an unreliable grader can manufacture improvements or regressions.

An honest PagePith demonstration: use retrieved evidence as a fixture input

The supplied PagePith proof shows a fetch request for the LiveWeb-IE arXiv page. The recorded result includes the page title, a content length of 7,660, and a Markdown excerpt beginning with the arXiv subject area, identifier, submission date, title, and authors.

That proof is enough to demonstrate one evaluation practice: record the exact retrieved artifact or excerpt that your test used, alongside the source URL and capture metadata. For example, a fixture for this page could ask for the paper title and arXiv identifier, with the supplied Markdown excerpt as evidence for the expected values.

It does not establish that PagePith can extract every field on arbitrary sites, handle live-page changes, or meet any accuracy threshold. Those are claims an evaluation set must test with representative pages and explicit labels.

Run regression checks as an operational loop

A release report should compare the candidate extractor with a named baseline and answer:

• Did overall field-level quality change?
• Which page types or attributes moved materially?
• Did unsupported values or invalid outputs increase?
• Are changes concentrated in one failure category?
• Did results change on recently recaptured pages?
• Are score differences larger than expected evaluation noise?

Run frozen fixtures frequently for repeatability. Recapture a rotating live subset on a scheduled basis to detect website evolution. NIST’s Measure function guidance emphasizes documented, repeatable measurement and ongoing monitoring; that is especially relevant when the input environment is the public web.

A practical release checklist

Before relying on a reported extraction score, verify that you can say yes to these questions:

• Does the corpus cover the page types, fields, and layouts that matter in production?
• Are labels field-level, evidence-backed, and reviewed?
• Are required, optional, and repeated fields scored appropriately?
• Do reports separate correctness, omissions, unsupported values, and format validity?
• Can every failure be assigned to a useful category?
• Are development, validation, and held-out material separated by meaningful structural variation?
• Are captures, labels, graders, and results versioned?
• Do frozen regressions and live recaptures both exist?

A strong evaluation set does not promise that extraction will never fail. It gives your team a reliable way to see where it fails, decide whether a change is safe, and improve the system without hiding risk behind a single number.

Ready to build a repeatable real-page evaluation workflow? Sign up for PagePith.