How to Build a Validation Layer for Extracted Web Data

Web extraction produces observations, not automatically trustworthy records. A product page may express a weight as 1,250 g, 1.25kg, or approximately 1 kilogram. Availability may appear as In stock, Available now, or a publisher-specific badge. A field that looks numeric may contain —, N/A, or an empty string.

A validation layer turns those observations into a controlled downstream contract. Its job is not to make every record pass. Its job is to make every decision explainable:

1. preserve what was extracted,
2. apply explicit normalization rules,
3. validate a canonical representation,
4. accept, warn, or reject with a reason, and
5. retain rejected input for remediation and replay.

This approach prevents the most expensive failure mode in ingestion: plausible-looking bad data silently entering analytics, search, or operational systems.

Define validation as a post-extraction contract

Keep extraction and validation separate. The extractor should capture what the source presented, while the validation layer decides whether that value conforms to your data contract.

A useful record has three representations:

• Raw value: the value as extracted, such as "1,250 g".
• Normalized value: an intermediate value produced by a documented rule, such as { "value": 1250, "unit": "g" }.
• Canonical value: the value that downstream systems use, such as { "value": 1.25, "unit": "kg" }.

Store the raw value even when canonical validation succeeds. It gives engineers evidence when a source changes its formatting, makes audit and debugging easier, and allows you to improve a parsing rule without re-fetching every page.

For the canonical document, use a schema that is portable across services. JSON Schema provides a practical structural baseline: types, required properties, numerical constraints, patterns, array limits, and object structure can all be expressed in a machine-readable contract.

A simple extracted product contract might look like this:

{
  "source_id": "vendor-a:sku-481",
  "availability": "in_stock",
  "weight": {
    "value": 1.25,
    "unit": "kg"
  },
  "raw": {
    "availability": "Available now",
    "weight": "1,250 g"
  }
}

This is deliberately different from a scrape payload. The scrape payload can be broad and source-shaped. The validated record should be narrow, stable, and useful to consumers.

Make coercion a deliberate policy

Coercion is useful when it is narrow and predictable. It is dangerous when it is broad enough to hide semantic errors.

For example, converting the string "42" to the number 42 can be reasonable if the field is documented as a whole-number count and the string matches an accepted numeric grammar. Converting "about forty-two", "N/A", or "42 units" to 42 is a different decision with a greater risk of changing meaning.

Many validation libraries coerce values by default, while also offering strict modes when exact input types are required. Pydantic documents both the convenience and the risks of coercion, which is a useful framing for ingestion systems: decide field by field whether conversion is allowed rather than treating it as an implementation detail. Read the Pydantic validation documentation.

Create a coercion policy table before writing parsers:

Input condition	Example	Policy	Result
Exact integer string	"42"	Accept and parse	42
Decimal with known unit	"1,250 g"	Parse and normalize	1.25 kg
Empty source value	""	Treat according to field policy	missing or rejected
Explicit source null	null	Preserve as null	null if allowed
Unknown label	"Ships soon"	Do not guess	quarantine or review
Ambiguous numeric text	"around 10"	Reject unless explicitly supported	validation error

The table is as important as the parser. It lets reviewers challenge a business rule directly instead of reverse-engineering it from regular expressions.

Keep missing, null, and invalid distinct

These states should not collapse into one generic empty value:

• Missing: the property was absent from the extracted payload.
• Null: the property was present and explicitly represented as no value.
• Invalid: a value was present but failed the contract.

For example, an absent weight may mean the page did not provide it. A weight: null may mean the publisher explicitly lists no weight. A weight: "unknown" is a present value that your numeric quantity contract does not accept.

This distinction affects completeness metrics, update behavior, and downstream filtering. It also prevents a default from concealing an upstream defect. In JSON Schema, default is an annotation; validation does not automatically insert a missing value. If your pipeline supplies a default, implement that operation explicitly and record that it happened. JSON Schema’s annotation guidance explains this behavior.

Validate types and allowed values before business logic

Start each record with structural validation, then run semantic checks that require domain knowledge.

A structural schema can require a numeric quantity, ensure a positive value, and limit availability to a defined set:

{
  "type": "object",
  "required": ["source_id", "availability"],
  "properties": {
    "source_id": { "type": "string", "minLength": 1 },
    "availability": {
      "type": "string",
      "enum": ["in_stock", "out_of_stock", "preorder"]
    },
    "weight": {
      "type": ["object", "null"],
      "required": ["value", "unit"],
      "properties": {
        "value": { "type": "number", "exclusiveMinimum": 0 },
        "unit": { "type": "string", "enum": ["g", "kg"] }
      }
    }
  }
}

An enum is a contract, not a suggestion. JSON Schema defines enum as a finite set of acceptable values, so it is well suited to canonical statuses, currencies, categories, and workflow states. See the JSON Schema enum reference.

Do not put publisher labels directly into that enum unless they are truly your canonical states. Instead, maintain a source-aware mapping such as:

"Available now" -> "in_stock"
"Sold out"       -> "out_of_stock"
"Pre-order"      -> "preorder"

If a new label arrives, such as "Ships soon", leave it unmapped and surface it as a validation failure. Guessing that it means in_stock can be more damaging than temporarily withholding the record.

Model quantities as value plus controlled unit

A number alone is not a measurement. 12 could mean grams, kilograms, inches, days, or an arbitrary publisher-defined score. Use a quantity object with a numeric value and a machine-readable unit, then normalize before comparing or aggregating values.

The FHIR Quantity datatype similarly separates value and unit-related fields, and its guidance points to UCUM codes for standardized units. It also warns that a display label should not automatically be treated as a valid unit code. FHIR Quantity documentation and its UCUM unit value set provide useful reference models for a controlled vocabulary.

A practical unit pipeline is:

1. Parse a raw source string into a number and source label.
2. Map the source label to an allowed unit only when the mapping is explicit.
3. Convert to the canonical unit for that measurement dimension.
4. Validate bounds after conversion.
5. Preserve the original text and the applied mapping rule.

For example, the rule 1,250 g -> 1.25 kg is safe when the parser recognizes g as grams and the field is known to represent mass. The same parser should not run against a field called package_size without context; a value like 250 g might be a product weight, while 250 in another source could be a count.

This is where semantic validators belong. Structural validation can prove that unit is one of g or kg; a semantic validator can prove that the unit is appropriate for the field and that the value falls within a domain-specific range.

Apply rules only when they apply

Not every rule belongs on every row. A battery capacity requirement may apply to items whose category is battery, but not to books. A preorder_date may be required only when availability is preorder.

Express these dependencies explicitly:

IF availability == "preorder"
THEN preorder_date must be present and valid

IF category == "battery"
THEN capacity must be a positive quantity in an allowed unit

Conditional validation avoids two common mistakes: rejecting valid records with irrelevant requirements, and passing records because a global rule was too vague. This pattern is also reflected in data-quality tooling that supports row conditions for rules such as conditional null checks and allowed-value checks. Great Expectations documents conditional row validation.

Quarantine failures with enough context to fix them

A rejected record is not garbage. It is feedback about a source, parser, contract, or mapping table. Never silently drop it.

Write each failure to a quarantine store or dead-letter path with:

• a stable record and source identifier,
• extraction timestamp and parser version,
• schema or contract version,
• raw field value or a protected reference to it,
• normalized value, if produced,
• rule identifier and human-readable error,
• severity (warning or reject), and
• a replay correlation ID.

For example:

{
  "source_id": "vendor-a:sku-481",
  "field": "availability",
  "raw_value": "Ships soon",
  "normalized_value": null,
  "rule_id": "availability.enum.v3",
  "reason": "value does not map to an allowed canonical status",
  "action": "quarantine"
}

This makes remediation concrete. A maintainer can add a reviewed mapping, replay affected records, and measure whether the change resolved the failure. Dead-letter patterns exist for the same reason in message processing: retain unprocessable inputs for analysis and reprocessing rather than losing them. AWS describes this purpose for dead-letter queues.

Track failures by source, field, rule, and contract version. A sudden rise in weight.unit.unknown is often an early indicator that a publisher changed its page template or introduced a new label.

A small PagePith demonstration

The supplied PagePith proof shows a fetch of https://json-schema.org/understanding-json-schema/reference/type. The result is titled “JSON Schema - Type-specific Keywords,” has a reported content length of 10077, and includes a Markdown excerpt mapping JSON types to Python analogues. The excerpt also begins an explanation that the type keyword can take a single-string form.

That is useful input for a validation workflow: a fetched reference page can be retained alongside the rules it informs. But this proof does not show PagePith performing type validation, unit conversion, enum enforcement, or record quarantine. Those controls remain responsibilities of the validation layer described in this article.

If you want to try PagePith with a source in your own workflow, sign up.

Implementation checklist

Before sending extracted records to downstream systems, verify that your pipeline can answer these questions:

1. What raw value did the source provide?
2. Which normalization rule transformed it, if any?
3. Is coercion allowed for this field, and under what grammar?
4. Are missing, null, invalid, and defaulted values distinguishable?
5. Is the canonical value structurally valid?
6. Does it pass domain-specific checks for units, ranges, and dependencies?
7. If rejected, where is the record retained and how is it replayed?

A good validation layer is intentionally conservative. It accepts values that meet a documented contract, transforms only what it can justify, and exposes everything else for review. That discipline makes web data safer to aggregate and easier to operate as sources evolve.

Ready to put an observable extraction workflow behind your validation contract? Sign up for PagePith.