How to Detect Duplicate Entities Across Multiple Websites

Duplicate detection is an identity decision, not a string-cleaning task

When you collect organizations, products, people, or locations from several websites, the same real-world entity rarely arrives in the same shape twice. One site may list Acme Industrial Ltd. while another has ACME Industrial; a product can have a shared title but a different SKU; and two local businesses can have nearly identical names while being separate locations.

The goal is therefore not simply to remove repeated rows. It is to decide which source records refer to the same underlying entity, while preserving enough evidence to revisit that decision later.

A production workflow for deduplicate entities from web data should optimize for four properties:

• Precision: avoid incorrectly merging separate entities.
• Recall: find genuine duplicates despite formatting and source differences.
• Scale: avoid comparing every record with every other record.
• Auditability: retain source records, match evidence, and the logic used to make each decision.

A useful mental model is that ingestion produces observations, while entity resolution produces a separately managed identity layer.

1. Keep raw observations and provenance immutable

Start by treating each retrieved page, API response, or extracted listing as a source record. Do not overwrite it with a cleaned version or immediately collapse it into a canonical row.

Instead, keep at least three layers:

1. Raw source record — the fields exactly as collected, plus the original payload when feasible.
2. Normalized comparison record — carefully transformed fields used for matching.
3. Canonical entity record — a stable identity record connected to one or more source records.

For every raw observation, record provenance such as:

• source_url
• source-specific record or listing ID
• retrieval timestamp
• scraper, parser, or extraction version
• normalization version
• canonical entity ID, if assigned
• matching rule, score, and field-level evidence
• reviewer and review timestamp for manual decisions

This structure makes corrections reversible. If a parsing bug splits an address incorrectly, you can re-run normalization and matching without losing the original evidence. It also lets consumers answer an important question: “Why do we believe these records represent the same entity?”

The W3C provenance model describes provenance in terms of entities, activities, and agents, and connects it with assessing quality, reliability, trustworthiness, and reproducibility. That is a strong conceptual fit for a web-data pipeline where each entity decision may need to be inspected later. Read the PROV Model Primer.

A small record model

source_record
  source_record_id: source_b:listing:98173
  source_url: https://directory.example/listing/98173
  retrieved_at: 2026-10-21T12:00:00Z
  raw_name: ACME Indústrial, Ltd.
  raw_phone: +1 (415) 555-0198
  raw_address: 20 Market Street, Suite 4

comparison_record
  source_record_id: source_b:listing:98173
  name_compare: acme industrial ltd
  phone_digits: 14155550198
  address_compare: 20 market st suite 4
  website_domain: acme.example
  normalization_version: 3

entity_assignment
  source_record_id: source_b:listing:98173
  canonical_entity_id: org_01J...
  decision: accepted
  match_method: domain_and_phone_exact
  match_score: 0.99
  evidence: domain exact; phone exact; name compatible

The score in this example is an internal decision artifact, not a claim that identity can be mathematically proven.

2. Normalize for comparison, but preserve the original value

Normalization makes equivalent formatting easier to compare. Depending on the field, it may include:

• Unicode normalization and case folding for names
• whitespace collapsing and punctuation handling
• phone parsing into country code and national number where possible
• URL parsing to identify a registrable domain or canonical host
• address parsing into components such as number, street, locality, and postal code
• SKU cleanup that retains significant hyphens, prefixes, and check digits

Unicode normalization deserves particular caution. Unicode normalization forms can give canonically equivalent strings a consistent binary representation, which helps with differently encoded text. But compatibility normalization can discard formatting distinctions, so it should not be applied indiscriminately to fields where those distinctions might matter. Keep the raw value and create a field-specific comparison value instead. Unicode Standard Annex #15 explains the trade-off.

For example, it is usually reasonable to remove casing and repeated whitespace from an organization name for a comparison feature. It is much less safe to remove meaningful product punctuation, unit quantities, apartment numbers, or legal suffixes without understanding the domain.

Use field-specific normalizers

One global normalize(text) function is tempting, but weak in practice. Different fields carry different identity signals:

Field	Helpful comparison representation	Common risk
Organization name	Case-folded, punctuation-aware token form	Merging unrelated businesses with generic names
Website	Parsed host and canonical URL variants	Treating marketplace or social hosts as unique identifiers
Phone	Digits plus country context	Shared switchboards or call centers
Address	Structured components and standardized abbreviations	Collapsing suites, units, or separate branches
Product	Manufacturer part number, SKU, GTIN if available	Assuming title similarity identifies a product
Person	Name plus independent corroborating attributes	Conflating people with common names

Normalization should generate evidence, not erase uncertainty.

3. Generate candidate pairs with blocking

A naïve deduplication system compares every record with every other record. That becomes expensive quickly: pairwise comparison grows at approximately n(n−1)/2. Splink illustrates that one million records implies roughly 500 billion possible comparisons. Its blocking guidance explains why candidate reduction is essential.

Blocking creates a much smaller set of plausible record pairs before detailed scoring. A pair needs to satisfy at least one selective rule to be considered.

For organization data, useful blocking rules might include:

same normalized website domain
OR same normalized phone number
OR same postal code AND same first name token
OR same city AND same first 6 characters of normalized name
OR same address number AND same street token

Use multiple strict rules rather than one very permissive rule. Each rule catches a different kind of duplicate: a domain can survive name changes, while a locality-and-name rule can help when no site is available. Deduplicate the pairs produced by overlapping rules before scoring them; the same pair may satisfy more than one block. This approach is also consistent with Splink’s recommendation to combine a longer list of strict blocking rules. See its blocking-rule documentation.

Do not use blocks as final identity decisions. Shared domains, phones, postal codes, and address fragments all occur in legitimate non-duplicate records.

4. Score evidence across several fields

After generating candidate pairs, score each pair with field-specific evidence. Avoid a single global fuzzy-name threshold.

A practical hierarchy is:

1. Exact stable source or external ID, where its semantics are trusted.
2. Exact canonical URL or normalized domain plus corroborating data.
3. Exact product identifiers, phone numbers, or carefully parsed addresses.
4. Multi-field fuzzy comparison, such as compatible names plus address or geography.
5. Weak name-only similarity, generally routed to review or rejection.

For strings, compare at more than one level. Exact equality, token overlap, edit-distance measures, and Jaro-Winkler-style similarity each behave differently with abbreviations, transpositions, and short strings. Splink documents comparators including Levenshtein and Jaro-Winkler, along with multiple thresholds within a comparison. Review the comparator guidance.

Example: organization matching rules

Suppose these two observations arrive from different sites:

A: Acme Industrial Ltd.
   https://www.acme.example/about
   +1 415 555 0198
   20 Market St, Suite 4, Harbor City

B: ACME Indústrial
   https://acme.example/contact
   (415) 555-0198
   20 Market Street Ste 4, Harbor City

A defensible match explanation could be:

• domain matches after host normalization;
• phone matches after formatting removal;
• street number, street name, and suite are compatible;
• names match after cautious text normalization;
• neither record has contradictory location or identifier evidence.

Contrast that with two records sharing only Acme Industrial in different cities. Even a high name-similarity score is weak identity evidence. The system should not merge them merely because a string comparator says they are close.

Rare values should generally carry more weight than common ones. A distinctive domain, uncommon SKU, or full phone number is usually more discriminative than a common company name or generic business category. This is the reasoning behind term-frequency adjustments in probabilistic matching systems, which account for skewed values. Splink documents these adjustments in its guides index.

5. Route decisions to accept, reject, or review

Do not force every candidate pair into “same” or “different.” Use three outcomes:

• Accepted match: enough evidence to link records automatically.
• Rejected match: insufficient evidence or a material contradiction.
• Review: plausible but ambiguous pairs requiring human judgment or more data.

Thresholds should follow your error costs. For a compliance-sensitive organization directory, an erroneous merge may be much worse than leaving duplicates unresolved. For lead enrichment, the balance may differ. Start with clerically labeled examples where possible, measure precision and recall, and inspect false positives and false negatives before changing thresholds.

Splink’s evaluation tools support comparison of predicted matches with clerical labels, including precision, recall, false positives, false negatives, and F-scores. Its evaluation API documentation is a useful reference. Treat a score as decision evidence, not proof of identity.

Need a reproducible input trail for resolution work? Preserve the URLs and extracted source material that led to a match decision, then keep that provenance alongside your normalized records. Start with PagePith.

6. Build canonical entities without losing source truth

Once pairs are accepted, assign a stable canonical_entity_id. Keep identity resolution separate from survivorship, the process of choosing which value appears in a canonical display record.

For example, a canonical entity might use the newest verified address, the most complete display name, and a source-preferred description. Those choices do not change the fact that several source records belong to the same entity.

canonical_entity
  canonical_entity_id: org_01J...
  display_name: Acme Industrial Ltd.
  current_address: 20 Market Street, Suite 4, Harbor City

entity_membership
  canonical_entity_id: org_01J...
  source_record_id: source_a:company:441
  match_method: exact_domain_plus_address
  match_score: 0.98

entity_membership
  canonical_entity_id: org_01J...
  source_record_id: source_b:listing:98173
  match_method: domain_and_phone_exact
  match_score: 0.99

Rule-based entity-resolution workflows commonly retain a match identifier and the rule responsible for the decision. AWS documents these workflow outputs. Your own model should likewise preserve the membership evidence separately from the curated canonical fields.

7. Monitor clusters for transitive over-merges

Pairwise matches often become entity clusters. If record A matches B and B matches C, a connected-component algorithm may place all three in one entity group—even if A and C would not independently pass the threshold.

That is useful, but it creates a transitive risk. Monitor clusters that have:

• many source records;
• unusually diverse names, domains, or locations;
• low-confidence edges that connect otherwise distinct subgroups;
• contradictory stable identifiers;
• rapid growth after a rule or normalizer deployment.

Splink’s clustering API describes turning pairwise predictions into connected components. Read the clustering documentation. Keep the pairwise edges, rules, and scores so that a cluster can be split if later evidence shows an over-merge.

Order matching rules from strongest to weakest, and prevent weak signals from silently overriding strong contradictions. AWS likewise recommends arranging rules from the most specific and highest-confidence to the least specific, particularly when considering transitive matching behavior. See its rule-ordering best practices.

An honest PagePith demonstration

The supplied PagePith proof shows a browser-tier request for the W3C PROV Model Primer. The recorded result includes the page title, PROV Model Primer, a reported content length of 59302, and a Markdown excerpt beginning with the W3C link and publication metadata.

For an entity-resolution workflow, that kind of captured source context can be attached to a provenance record: the requested URL, when it was collected, and the extracted material used as input. The proof does not establish matching, normalization, scoring, or automatic canonicalization capabilities, so those steps should remain explicit parts of your own data pipeline.

A durable operating checklist

Before publishing deduplicated web data, verify that you can answer all of these questions:

1. Can you retrieve the untouched source observation for every canonical membership?
2. Are raw and normalized fields stored separately with versioned transformations?
3. Do candidate-pair rules constrain comparisons without excluding important duplicate patterns?
4. Is every accepted match explainable through field-level evidence?
5. Are uncertain cases retained for review rather than automatically merged?
6. Have thresholds been tested against labeled examples and actual business error costs?
7. Can you split an entity cluster and reprocess downstream records after an error?
8. Are canonical display choices distinct from the evidence establishing identity?

Duplicate detection becomes safer when it is designed as a reversible, evidence-based system rather than a one-time fuzzy matching pass. Preserve observations, normalize with care, compare only plausible candidates, combine independent evidence, and keep uncertainty visible.

Ready to build a source-aware ingestion workflow? Sign up for PagePith.