How to Scrape Infinite-Scroll Pages Without Missing Records

Infinite-scroll interfaces are designed to hide pagination from people, not necessarily to eliminate it. A catalog may appear to be one continuous page, yet every new group of products, posts, or listings is usually delivered by a separate request with state that tells the server where to continue.

That distinction matters. If you only save the initial DOM, you will often collect the first visible records and silently miss the rest. If you automate scrolling with long fixed delays, you may get inconsistent results, duplicate batches, or stop before the final records arrive.

A more dependable approach is to scrape infinite scroll pages as a stateful pagination workflow:

1. Trigger one additional batch in a browser.
2. Identify the request that returned it.
3. Capture the continuation state.
4. Replay requests while carrying that state forward.
5. Stop based on an explicit terminal signal.
6. Verify uniqueness, ordering, and coverage before publishing data.

Why scrolling alone is not an extraction strategy

Infinite scroll and “load more” patterns commonly depend on JavaScript actions. The visible page can therefore be only a partial representation of the dataset; the next records may not exist in the initial HTML at all. Google’s guidance on incremental page loading makes the same practical point: content loaded after an action needs a discoverable delivery path rather than an assumption that the first response contains everything. Read the guidance.

Many implementations load the next batch when a small element near the bottom of a list enters the viewport. This element is often called a sentinel. The browser mechanism behind it may be IntersectionObserver, an API explicitly suited to infinite-scroll behavior. MDN’s Intersection Observer documentation is useful background when you need to understand why a scroll action triggered a request.

But a sentinel is a browser-side trigger, not the source of truth for extraction. The source of truth is usually one of these:

• A REST endpoint using page, offset, and limit
• A REST endpoint returning a next URL or opaque continuation token
• A GraphQL query using first and after
• A form or request body containing a cursor, filters, sort order, and session-specific state

Your goal is to discover which one the target uses.

Step 1: Discover the request behind one new batch

Start manually in a browser. Open developer tools, clear the Network panel, filter to Fetch/XHR traffic, and cause exactly one additional batch to load. Inspect the request that appears at that moment.

Chrome DevTools can show request headers, request payloads, response bodies, and initiators, and it can copy a request in formats such as cURL or fetch. Those features make it practical to compare the initial request with the request made after scrolling. See the DevTools network reference.

Record these details before writing a scraper:

• Request URL and HTTP method
• Query parameters or JSON/form request body
• Relevant headers, including content type and any required authorization you are permitted to use
• Cookie or session dependence
• Batch size and response shape
• Record identifier field
• Continuation field: offset, page number, cursor, token, or next URL
• Terminal signal: hasNextPage, absent token, empty page, or another documented value
• Active filter and sort parameters

Do not infer pagination from a URL if the actual continuation is in the request body. A common GraphQL request, for example, keeps the endpoint unchanged and sends the cursor in variables:

{
  "query": "query Products($first: Int!, $after: String) { ... }",
  "variables": {
    "first": 50,
    "after": "opaque-cursor-from-prior-response"
  }
}

Cursor values should be treated as opaque. The GraphQL Cursor Connections specification defines forward traversal through after, endCursor, and hasNextPage; it does not require cursors to be predictable or numeric. Review the cursor model.

Use the browser for observation, not arbitrary delays

Browser automation is often useful during discovery, especially when a page requires rendering before you can observe its network traffic. However, avoid logic such as “scroll, sleep five seconds, then parse.” Variable latency, throttling, and delayed rendering make this unreliable.

Instead, wait for the specific request or response associated with the list endpoint. Playwright supports page.waitForRequest() and page.waitForResponse() with URL or predicate matching. Its documentation also cautions against using fixed timeout waits and treating networkidle as a universal readiness signal. See the Page API.

A discovery-oriented example might look like this:

const responsePromise = page.waitForResponse(response =>
  response.url().includes('/api/products') && response.status() === 200
);

await page.locator('[data-results-scroll-container]').evaluate(el => {
  el.scrollTop = el.scrollHeight;
});

const response = await responsePromise;
const payload = await response.json();
console.log(payload);

The selector and endpoint are examples only. The important part is the synchronization: the script waits for evidence that a new batch arrived rather than guessing that enough time has passed.

Step 2: Model continuation state explicitly

Once you know the response shape, write down the state needed to request the next batch. Keep this state separate from presentation logic and separate from the growing output dataset.

For an offset API, state may be simple:

state = { offset: 0, limit: 100, filters: {...}, sort: "newest" }
next state = { ...state, offset: state.offset + receivedCount }

For a cursor API, state must preserve the returned cursor:

state = { after: null, first: 100, filters: {...} }
next state = { ...state, after: response.pageInfo.endCursor }

GitHub’s GraphQL pagination guide illustrates the cursor pattern: request a page, take its endCursor, pass that value as after in the next request, and consult page metadata to determine whether another page exists. Read the pagination guide.

This is why replacing a cursor with page=2 is risky. The server may encode sort position, filtering context, sharding information, or other state in an opaque token. Altering or fabricating it can cause missing ranges, repeated results, or invalid requests.

Also preserve every parameter that defines the result set. If the UI is filtered to a category, region, date range, or sort order, each follow-up request needs the same selection state. A cursor from “price low to high” is not necessarily valid for “newest first.”

Ready to make extracted content part of a repeatable workflow? Create a PagePith account to explore the product for your own documented use cases.

Step 3: Choose stopping conditions that cannot quietly truncate data

The best stop condition is a server-provided one. Prefer these in order:

1. hasNextPage === false
2. No next cursor, token, or URL in the response
3. A documented terminal response from the endpoint
4. A final batch smaller than the requested size, when that behavior is known to be reliable

An empty batch alone is not always sufficient. Transient errors, eventual consistency, or a bad request can also produce zero records. Treat it as a signal to investigate unless it agrees with the endpoint’s pagination metadata.

Use defensive guardrails as well:

• Stop and alert if the next cursor equals a cursor already seen.
• Stop and alert after a configurable number of consecutive batches with zero new records.
• Set a maximum batch count as a fail-safe, but never treat reaching it as successful completion.
• Persist the last confirmed continuation state so an interrupted run can resume deliberately.

Here is generic pseudocode for a cursor-based collector:

seen_ids = set()
seen_cursors = set()
after = None
batch_number = 0

while True:
    payload = fetch_batch(first=100, after=after, filters=filters)
    items = payload["data"]["items"]["edges"]
    page_info = payload["data"]["items"]["pageInfo"]

    batch_number += 1
    new_items = []
    for edge in items:
        item = edge["node"]
        item_id = item["id"]
        if item_id not in seen_ids:
            seen_ids.add(item_id)
            new_items.append(item)

    save_batch_audit(batch_number, after, len(items), len(new_items))
    save_items(new_items)

    if not page_info["hasNextPage"]:
        break

    next_after = page_info["endCursor"]
    if not next_after or next_after in seen_cursors:
        raise RuntimeError("Pagination cursor did not advance")

    seen_cursors.add(next_after)
    after = next_after

This loop does not assume that every response is clean. It records raw count and new unique count separately, then requires the cursor to advance before proceeding.

Step 4: Deduplicate requests and records separately

A repeated request and a repeated record are related problems, but they are not the same problem.

A job may replay an identical request after a retry, resume, or concurrency race. Separately, an API can return an overlapping item at the boundary between two valid pages. Deduplicating requests alone will not remove overlapping records; deduplicating records alone will not prevent wasted retries or pagination loops.

Use two distinct keys:

• Request identity: method, normalized URL, body, and pagination state
• Record identity: a stable source ID whenever available

Scrapy distinguishes these concerns in its architecture: scheduling has duplicate-request protection, while item pipelines are a suitable place to validate items and drop duplicate records. See Scrapy’s request/response documentation and its item pipeline examples.

Avoid using a title or position in the list as the record key. Titles change, records can move when new items appear, and identical titles may be valid. Prefer a source-provided ID. If none exists, derive a carefully normalized composite key and retain the source URL and raw record for later review.

Step 5: Prove completeness with an extraction audit

A scraper is not complete merely because it stopped. It is complete when its terminal state and its output agree.

For every batch, log:

Field	Why it matters
Run ID and timestamp	Makes results reproducible and debuggable
Batch number	Reveals unexpected early stops
Request state or cursor hash	Shows progression without exposing sensitive values in logs
Raw record count	Detects unexpectedly small batches
New unique record count	Detects overlap or stalled progression
First and last stable sort values	Helps spot ordering breaks
Terminal signal	Documents why the run ended

Then run these checks:

1. Cursor progression: No cursor or token repeats before a valid final state.
2. Record uniqueness: The count of persisted IDs equals the count of unique IDs.
3. Overlap: Measure records repeated between adjacent batches; investigate unexpected spikes.
4. Ordering: Where a stable sort field is available, confirm its sequence across batch boundaries.
5. Terminal-state validation: Confirm the final response explicitly indicates no further data, where supported.
6. Repeatability: Rerun a stable query and compare counts and identifiers, allowing for documented live-data changes.

Ordering is more than a cosmetic detail. Cursor pagination relies on a consistent sequence across pages, and the cursor specification frames cursors as boundaries in that sequence. The Relay specification is a useful reference for that model. Google’s guidance on infinite-scroll implementations similarly emphasizes coverage and avoiding overlap across component pages. See its discussion of search-friendly infinite scroll.

Reliability and access boundaries

A technically successful request is not automatic permission to automate it. Review the site’s access rules, applicable terms, authentication constraints, and the purpose of your collection before proceeding. Keep credentials out of logs and never attempt to bypass access controls.

For permitted extraction, be conservative with traffic. Use bounded concurrency, retry transient failures with backoff, and reduce pressure when latency rises. Scrapy’s AutoThrottle documentation describes adapting delays based on latency and target concurrency, which is a sound model for building polite collectors. Read about AutoThrottle.

The practical benefit is not only etiquette. A throttled, observable job is less likely to trigger rate limits that masquerade as empty pages or incomplete result sets.

A limited PagePith demonstration

The supplied PagePith proof shows a successful fetch-tier retrieval of Google’s pagination and incremental-loading documentation at the cited URL. The captured result reported the page title, a content length of 8,742, and Markdown beginning with the section “Pagination, incremental page loading, and their impact on Google Search.” That excerpt discusses presenting a subset of results and ensuring content can still be found.

This demonstrates that PagePith retrieved readable content from that specific documentation page. It does not demonstrate automatic scrolling, browser interaction, request replay, cursor traversal, or a completeness audit on an infinite-scroll target. Those capabilities should be evaluated against your own permitted source and workflow rather than inferred from this fetch result.

The operational checklist

Before you call an infinite-scroll extraction complete, confirm all of the following:

• You captured the request caused by loading one more batch.
• You identified the true continuation mechanism.
• You carry filters, sorting, and continuation state on every request.
• You wait for explicit request/response evidence, not arbitrary sleep intervals.
• You use server pagination metadata as the primary stop condition.
• You prevent repeated request states and deduplicate stable record IDs.
• You log batch-level progress and terminal evidence.
• You test for overlap, ordering problems, and rerun consistency.
• You operate within the source’s applicable access requirements and with measured request rates.

Infinite scroll is a UI pattern. For extraction, treat it as a state machine with observable transitions and an auditable end state. That mindset replaces fragile scrolling scripts with a process you can test, resume, and defend.

Want to evaluate PagePith for your content workflow? Sign up to get started.