How to Extract Repeated Items Without Losing Parent-Child Relationships

Repeated records are easy to find and surprisingly easy to corrupt.

Consider a catalog with categories and products, a documentation portal with sections and articles, or a job board with departments and open roles. A page-wide query can collect every product, article, or role—but that output no longer says which parent owns each child. Once that context is gone, downstream joins become guesses.

The durable approach is container-first extraction:

1. Select one parent record container.
2. Read that parent’s fields from inside the container.
3. Select child records relative to that same container.
4. Repeat for every parent.
5. Emit a nested structure or a flat structure with explicit parent keys.

This follows the shape of the source document. The DOM is a tree with parent and child relationships, not a set of unrelated text fields. See the W3C DOM specification for the underlying node model.

The failure mode: correct fields, incorrect ownership

Suppose this is the relevant page structure:

<section class="category" data-category-id="cloud">
  <h2>Cloud</h2>
  <article class="product" data-product-id="vm-1">
    <h3>Virtual Machine</h3>
  </article>
</section>

<section class="category" data-category-id="security">
  <h2>Security</h2>
  <article class="product" data-product-id="fw-1">
    <h3>Firewall</h3>
  </article>
</section>

A tempting implementation does this:

const categories = [...document.querySelectorAll('.category')]
  .map((node) => node.querySelector('h2')?.textContent?.trim());

const products = [...document.querySelectorAll('.product')]
  .map((node) => node.querySelector('h3')?.textContent?.trim());

Both arrays are accurate individually. But categories[0] and products[0] are only associated because of their current order. Add a featured product, a hidden template, pagination controls, or a different layout, and positional matching breaks.

The more serious version of this bug happens inside a loop:

for (const category of document.querySelectorAll('.category')) {
  const title = category.querySelector('h2')?.textContent?.trim();
  const products = document.querySelectorAll('.product'); // Wrong scope
  // Every category now sees every product.
}

The selector is valid. The data may even look plausible in a small test. But the child lookup starts from document, so it has lost the current category as its boundary.

Treat repeated containers as the unit of extraction

Start by identifying the smallest stable container that represents one complete parent record. In the example, that is .category, not the heading and not the entire catalog.

This is more than a selector preference. Research on web record extraction uses recurring structural patterns and invariant paths to identify record containers, which supports treating the repeated subtree as the basic extraction unit rather than collecting individual fields globally. See Web Record Extraction with Invariants.

A useful inspection checklist is:

• Does each candidate parent contain one identity field, such as a name, URL, or ID?
• Does it contain the child list that belongs to that identity?
• Does the same subtree repeat for sibling records?
• Is the container narrow enough to exclude unrelated page regions?
• Does a child appear under exactly one selected parent?

Prefer semantic hooks when the markup provides them:

<section data-category-id="cloud">
  <article data-product-id="vm-1">...</article>
</section>

data-* attributes are standard HTML mechanisms for machine-readable custom data and can be read through dataset; see MDN’s dataset documentation. They are often less fragile than long selectors tied to presentation-oriented wrappers. Playwright similarly cautions that structure-dependent CSS and XPath selectors can become brittle as a DOM changes; its locator guidance is available in the Playwright locator documentation.

Scope every child query to its parent

Here is the same catalog extracted into nested JSON:

function text(root, selector) {
  return root.querySelector(selector)?.textContent?.trim() ?? null;
}

const categories = [...document.querySelectorAll('[data-category-id]')].map(
  (categoryNode) => ({
    id: categoryNode.dataset.categoryId,
    name: text(categoryNode, ':scope > h2'),
    products: [...categoryNode.querySelectorAll(':scope > .product')].map(
      (productNode) => ({
        id: productNode.dataset.productId,
        name: text(productNode, ':scope > h3'),
      })
    ),
  })
);

The important property is not the JavaScript syntax. It is the query root:

categoryNode.querySelectorAll(':scope > .product')

Element.querySelectorAll() searches descendants of the element on which it is called, so calling it on categoryNode retains context. MDN’s querySelectorAll() reference documents this scoped descendant behavior.

Decide whether children are direct or deep descendants

Record boundaries need a second decision: should a child be a direct child of the parent, or can it occur anywhere below it?

Use a direct-child selector when intermediary nested widgets must not count:

categoryNode.querySelectorAll(':scope > .product');

Use a descendant selector when the page legitimately wraps child records in layout elements:

categoryNode.querySelectorAll('.product');

The equivalent distinction matters in XPath. In Scrapy, p selects direct p children in the current context, while .//p searches descendants. Critically, //p begins at the document root even if it is called from a selected element. The Scrapy selector documentation explicitly describes this difference.

A Scrapy pattern that retains hierarchy

Scrapy’s selector chaining makes the boundary clear. Select parent containers first, then make every field and child selector relative to each parent selector.

import scrapy

class CatalogSpider(scrapy.Spider):
    name = "catalog"
    start_urls = ["https://example.invalid/catalog"]

    def parse(self, response):
        for category in response.css("[data-category-id]"):
            category_id = category.attrib.get("data-category-id")

            yield {
                "id": category_id,
                "name": category.css(":scope > h2::text").get(default="").strip(),
                "products": [
                    {
                        "id": product.attrib.get("data-product-id"),
                        "name": product.css(":scope > h3::text").get(default="").strip(),
                    }
                    for product in category.css(":scope > .product")
                ],
            }

For XPath, maintain the leading dot for children:

for category in response.xpath("//*[@data-category-id]"):
    product_names = category.xpath(
        ".//*[contains(@class, 'product')]/h3/text()"
    ).getall()

That initial . is a relationship-preserving detail, not cosmetic punctuation. The Scrapy documentation’s nested-selector examples explain why .xpath('.//p') is local to the selected subtree while .xpath('//p') is global. See Scrapy selectors.

Build extraction workflows around records, not isolated fields. If you want a repeatable place to fetch a page and inspect its content before designing those record boundaries, create a PagePith account.

Choose the output model before writing selectors

The same DOM can feed different consumers. Make the output model explicit up front.

Nested JSON for hierarchy-oriented consumers

Nested output mirrors the page and is convenient for APIs, snapshots, and document stores:

[
  {
    "id": "cloud",
    "name": "Cloud",
    "products": [
      { "id": "vm-1", "name": "Virtual Machine" }
    ]
  }
]

JSON supports objects and arrays as values, so arrays of child objects are a valid standards-compatible representation. RFC 8259 defines these JSON structures.

Parent-linked records for tables and warehouses

If the destination expects rows, do not flatten into anonymous lists. Emit stable IDs and parent IDs:

{
  "categories": [
    { "id": "cloud", "name": "Cloud" }
  ],
  "products": [
    {
      "id": "vm-1",
      "categoryId": "cloud",
      "name": "Virtual Machine"
    }
  ]
}

This is an engineering schema choice built on the DOM’s tree relationship: every child gets an explicit edge back to its selected parent. It makes joins deterministic and lets child records be processed independently without discarding ownership.

Avoid generating IDs from array positions when the page already provides durable identifiers. A URL, a data-* value, or a domain-specific identifier is generally a better key than category-3-product-7.

Dynamic pages: wait before you enumerate

Browser automation introduces a timing issue. A selector may be correctly scoped but still run while the list is incomplete.

In Playwright, first wait for an application-specific ready signal—such as a loaded-state marker or the disappearance of a loading indicator—then iterate the parent locator. Playwright warns that locator.all() does not wait for matching elements and can be unpredictable when a list is still changing. See the Locator API reference.

Once the list is ready, chain locators so each child query stays under its parent:

const categoryLocator = page.locator('[data-category-id]');

// Wait for the page's own completion condition before this point.
const categories = [];
for (const category of await categoryLocator.all()) {
  const products = [];

  for (const product of await category.locator(':scope > .product').all()) {
    products.push({
      id: await product.getAttribute('data-product-id'),
      name: await product.locator(':scope > h3').textContent(),
    });
  }

  categories.push({
    id: await category.getAttribute('data-category-id'),
    name: await category.locator(':scope > h2').textContent(),
    products,
  });
}

Locator chaining and descendant filtering are designed to preserve this kind of context. The Playwright locator guide documents that filtering is evaluated relative to the original locator rather than reset to the document root.

Recovering context when you begin with a child

Sometimes a task begins with a matching child: a product link, an “Apply” button, or a comment timestamp. In that case, recover the nearest intended record container before reading related fields:

const productLink = document.querySelector('a.product-link');
const product = productLink?.closest('[data-product-id]');
const category = product?.closest('[data-category-id]');

closest() checks the current element and then walks through ancestors until it finds a match. That makes it useful for upward recovery, provided the selector identifies the nearest correct container. MDN’s closest() reference describes this traversal.

Do not use an overly broad ancestor selector such as .card if cards can nest. Prefer a record-specific marker like [data-product-id].

Validate ownership, not just extraction

A scraper can return nonempty data and still be wrong. Add checks that specifically test hierarchy:

const seenProductIds = new Set();

for (const category of categories) {
  if (!category.id) throw new Error('Category is missing an ID');

  for (const product of category.products) {
    if (!product.id) throw new Error(`Product in ${category.id} has no ID`);
    if (seenProductIds.has(product.id)) {
      throw new Error(`Product ${product.id} was assigned more than once`);
    }
    seenProductIds.add(product.id);
  }
}

Also compare extracted counts with what the visible UI implies, inspect parents with zero children, and test pages where nested look-alike elements exist. The general principle is similar to list-count and descendant-filter checks described in the Playwright locator documentation: verify the structure you intended to select, not merely the presence of text.

A small PagePith demonstration

The supplied PagePith proof shows a fetch of the Scrapy selector documentation. It was retrieved at the fetch tier, reported the title “Selectors — Scrapy 2.18.0 documentation”, and reported a content length of 51,920. Its returned Markdown excerpt begins with Scrapy’s introduction to extracting data from HTML and mentions BeautifulSoup and lxml.

That demonstration establishes that PagePith retrieved this documentation page and produced Markdown content suitable for inspection. It does not by itself demonstrate automatic nested-record detection, selector generation, or parent-child extraction. Those relationships still need to be modeled deliberately using the container-first strategy above.

Keep the parent boundary intact

The rule to carry into every scraper is simple: once you identify a parent record, never restart child selection from the document root.

Select the repeated container, query within it, decide direct versus descendant boundaries intentionally, and produce either nested objects or parent-linked records. With those constraints, repeated content remains connected to the record that gives it meaning.

Ready to inspect pages and design a context-preserving extraction workflow? Sign up for PagePith.