How to Extract Tables from Web Pages When Rows Are Split Across Multiple Views

A table that appears to be one report in a browser may be several separately rendered datasets. A tab can replace the visible rows. Pagination can request a new server-side slice. A virtualized grid can expose only a small window of a much larger logical table.

If a scraper reads only the initial DOM and emits cell text, it can silently produce an incomplete or misaligned dataset. The safer model is to treat the page as a state machine:

1. Identify the table's semantic structure.
2. Capture a canonical schema.
3. Enumerate every reachable table view.
4. Wait for each view to reach a known state.
5. Extract rows with state metadata.
6. Merge with stable identities and validate completeness.

The result is not just a CSV-like collection of values. It is a reconstructed logical table with evidence for where each row came from.

Start by classifying the rendered table

Before writing selectors, determine what kind of table you are looking at.

• A native HTML table has table, row, header-cell, and data-cell semantics.
• An ARIA table is a static table represented through roles such as table, row, columnheader, rowheader, and cell.
• An interactive ARIA grid commonly uses grid, row, gridcell, columnheader, and rowheader; it may implement keyboard navigation, virtualization, or partial rendering.

Native tables are preferable when the page provides them, because the document semantics directly communicate row and cell structure. The W3C describes both native table structure and ARIA table/grid roles in its table pattern guidance.

This classification determines what to inspect next. For a native table, the header and body may be directly available. For a grid, a visible row set may be only a viewport into an indexed dataset. In the latter case, attributes such as aria-rowcount, aria-colcount, aria-rowindex, and aria-colindex can reveal the logical dimensions and positions of rendered fragments. The W3C documents these properties specifically for tables and grids that render only part of their logical content: Grid and Table Properties.

A useful initial record looks like this:

type TableProfile = {
  kind: "native" | "aria-table" | "aria-grid";
  locator: string;
  declaredRowCount?: number;
  declaredColumnCount?: number;
  visibleRowCount: number;
  controllerHints: string[];
};

controllerHints can include tab controls, page buttons, a “showing 21–40 of 194” label, a loading indicator, or an observed Fetch/XHR request. These are not incidental UI details; they define how to enumerate the underlying dataset.

Preserve schema before collecting data rows

A common failure mode is assuming the first visible row supplies the column names. That only works for the simplest tables.

Real reports can have multiple header rows, grouped headers, rowspan and colspan, or row headers that describe the first cell of each data row. The W3C’s tables tutorial covers simple, irregular, and multi-level header relationships, including explicit associations with scope, id, and headers where needed.

Build a canonical schema once per logical table:

type Column = {
  index: number;
  key: string;
  labelPath: string[];
  sourceHeaderIds?: string[];
};

type Schema = {
  columns: Column[];
  fingerprint: string;
};

For a two-level header, a leaf column's labelPath might be ["2026", "Revenue"], rather than just "Revenue". This avoids ambiguity when several groups contain a column with the same visible label.

The extraction algorithm should expand header spans into a rectangular coordinate grid, then derive one leaf-column definition per final column position. For native markup, prioritize explicit relationships where present, then use header placement and scope as appropriate. Store the original header text and normalized version separately: the normalized value is useful for stable matching, while the original is useful for output and audits.

Do not assume every tab has the same schema. Compare the schema fingerprint after each state transition. If a tab changes from monthly rows to a regional summary, that is probably a separate logical dataset—not another page of the same one.

Enumerate views as a product of state dimensions

“Multi-view” is broader than pagination. A page may have tabs, a date selector, filters, sort order, page size, and cursor or page controls. Conceptually, define a view as:

view = tab × filter × sort × page-or-cursor

You do not necessarily need every possible combination. For a target report, choose the intended fixed filter and sorting state, then enumerate all tabs and every page within each relevant tab. What matters is that your scope is explicit.

Tabs: follow the semantic relationship

A tabbed interface often leaves inactive panels in the DOM but hidden. Reading every matching table selector can therefore collect inactive, stale, or duplicate content. Conversely, reading only the active panel misses the other tab views.

The W3C tabs pattern defines a useful traversal path: a tab has role="tab", identifies its controlled panel with aria-controls, and the associated panel has role="tabpanel". The active tab is represented with aria-selected="true".

Use that relationship instead of a selector tied to a framework-specific class name:

const tabs = page.getByRole("tab");

for (let i = 0; i < await tabs.count(); i++) {
  const tab = tabs.nth(i);
  const tabId = await tab.getAttribute("id");
  const panelId = await tab.getAttribute("aria-controls");

  await tab.click();
  await expect(tab).toHaveAttribute("aria-selected", "true");

  const panel = page.locator(`#${CSS.escape(panelId ?? "")}`);
  await extractAllPagesInPanel(panel, { tabId, panelId });
}

This example uses role-based controls because those reflect the user-facing accessibility contract. Playwright recommends role and other user-visible locators over DOM-structure-dependent CSS or XPath chains in its locator guidance.

Pagination: inspect requests before clicking endlessly

A numbered paginator might be entirely client-side, or it might trigger a fresh server request for every page. The distinction changes your extraction plan.

DataTables, for example, documents server-side processing as an Ajax request per draw. Its requests include paging values such as start and length, as well as search and ordering information; responses include the page data plus recordsTotal, recordsFiltered, and a draw counter. See the server-side processing protocol. DataTables also distinguishes this behavior from client-side processing, in which the browser has the full dataset available: Data.

Inspect Fetch/XHR traffic while changing pages. Record request URL, method, query or form fields, response shape, and the current UI state. If the application exposes a permitted structured response, collecting its pages can be more reliable than repeatedly scraping repainting cells. If browser interaction is required, use the same observed values as audit metadata.

Do not use the asynchronous draw value as a page identity. It helps pair requests and responses, but start, length, search, ordering, and selected tab describe what a page actually means.

Build extraction workflows around evidence, not fragile selectors. Create a PagePith account to start documenting the pages and rendered content your team needs to inspect.

Wait for a state transition, then snapshot

Fixed delays are tempting: click “Next,” sleep 500 ms, read rows. They fail when the request is slow, cached, aborted, or replaced by a second request.

Instead, capture a pre-action signature and wait for an expected semantic change. A signature can contain the active tab ID, the first row's stable key, visible row count, paging label, and a short hash of the rendered rows.

type ViewSignature = {
  activeTab?: string | null;
  firstKey?: string | null;
  rowCount: number;
  pageLabel?: string | null;
};

async function readSignature(table: Locator): Promise<ViewSignature> {
  const rows = table.locator("tbody tr");
  return {
    rowCount: await rows.count(),
    firstKey: await rows.first().getAttribute("data-row-id"),
    pageLabel: await table.locator("..").getByText(/showing|page/i).first().textContent().catch(() => null),
  };
}

After activating a tab or page, wait for the appropriate signal: the active-tab attribute changes, a busy indicator disappears, the first row key changes, or a page label updates. Playwright provides actionability checks and retrying assertions for this style of synchronization; see auto-waiting.

For custom applications, a MutationObserver can monitor the table body or grid container for row replacement and relevant attribute changes. This is useful when no reliable loading indicator exists. Also, snapshot cells only after the table has stabilized. HTMLTableElement.rows is a live collection, so it can change while client-side code is replacing rows; MDN documents this behavior in HTMLTableElement.rows.

Attach provenance to each extracted row

Every emitted row should retain the state that gave it meaning. A practical record might be:

type ExtractedRow = {
  key: string;
  values: Record<string, string>;
  source: {
    tabId?: string | null;
    panelId?: string | null;
    start?: number;
    length?: number;
    pageNumber?: number;
    search?: string;
    order?: string;
    rowIndex?: number;
    capturedAt: string;
  };
};

Prefer a stable source-provided identity such as a row data-* attribute, link target, or response identifier. In DataTables server-side responses, DT_RowId can provide row-level identifiers, while DT_RowData can provide row metadata; both are documented in the server-side processing reference.

If there is no stable ID, use a composite key carefully. For example, account_id + reporting_date is safer than a hash of all visible cell values, because formatting changes can alter the latter. If neither exists, preserve request state and visual row position, flag the output as position-identified, and avoid aggressive deduplication.

Merge conservatively and prove completeness

Merging is a data-quality operation, not simple concatenation.

First, reject a view if its schema fingerprint differs from the expected schema. Then insert rows by stable key. When the same key appears twice, compare normalized values and provenance:

• Equal values may be a harmless overlap between cursor windows or repeated tab content.
• Different values may indicate a changing source, an insufficient key, different filters, or a parsing error.
• Missing keys require positional evidence and a warning rather than silent deduplication.

Finally, validate the assembled result against the page's own signals:

1. Coverage: Did you visit every discovered tab and every expected page or cursor state?
2. Counts: If the response reports recordsFiltered, does the count of unique extracted rows match it? If not, identify whether filtering, page size, or a failed request explains the difference.
3. Continuity: For offset pagination, are expected starts present without gaps or duplicate intervals?
4. Placement: For partial ARIA grids, do row and column indices cover the expected range indicated by declared counts?
5. Schema: Did header paths remain consistent across pages in the same logical table?
6. Repeatability: On a second run, do keys and row counts match within the expected source-update window?

A compact extraction manifest makes these checks reviewable:

logical table: quarterly-orders
schema fingerprint: 9e2c…
tabs visited: open, closed
page intervals: 0–49, 50–99, 100–149
expected filtered rows: 132
unique merged rows: 132
schema mismatches: 0
conflicting duplicate keys: 0

An honest PagePith demonstration

The supplied PagePith proof shows a browser-tier request to the W3C Tables Tutorial. It returned the page title Tables Tutorial | Web Accessibility Initiative (WAI) | W3C, a rendered-content length of 3,631, and Markdown explaining that header cells use th, data cells use td, and complex tables can require scope, id, and headers relationships.

That demonstration supports one narrow but useful point: rendered page content can preserve table-structure guidance in a Markdown-friendly form for inspection. It does not demonstrate extraction of a live paginated grid, automatic tab traversal, network replay, or dataset completeness validation. Those steps still require the state-aware process described above.

Treat the page as an interface, not a static document

The dependable way to extract multi-view web tables is to reconstruct the contract behind the display: headers define schema, controls define reachable states, requests define server slices, and row identifiers define merge behavior.

When you retain that context, a scraper can explain why it believes the table is complete. When you discard it, even a clean-looking export may contain hidden omissions or shifted columns.

Sign up for PagePith to begin capturing and reviewing the rendered web content that informs your extraction workflow.