How to Build a RAG Pipeline for Poorly Structured Web Pages

A retrieval pipeline for web pages usually fails before vector search begins. The failure is not that embeddings are incapable of representing prose. It is that the input contains the wrong prose: global navigation, cookie notices, product cards, repeated sidebars, footer links, and fragments of the actual article stripped of their hierarchy.

Help centers, policy pages, blogs, and operational guides make this problem especially visible. A question such as “How long does a refund take?” may match a dozen navigation labels and a single policy paragraph. If the pipeline flattens the page into text, chunks it at arbitrary character counts, and embeds everything equally, retrieval can return a plausible but unsupported answer.

The better model is document engineering first, retrieval engineering second. Build a chain that preserves what a page says, where it says it, and why a chunk should be considered relevant.

Start with a page representation, not a text blob

A useful ingestion record should move through explicit stages:

URL
  -> fetch result
  -> cleaned DOM
  -> typed elements
  -> structure-aware chunks
  -> indexed retrieval records
  -> ranked evidence
  -> grounded answer

Each arrow is an opportunity to lose meaning. Make each stage inspectable and persist enough data to reproduce a bad retrieval result.

A practical source record might include:

{
  "url": "https://docs.example.com/refunds",
  "canonicalUrl": "https://docs.example.com/refunds",
  "fetchedAt": "2026-09-04T12:00:00Z",
  "fetchMode": "http",
  "status": 200,
  "contentHash": "...",
  "rawHtmlLocation": "...",
  "extractionVersion": "2026-09-04.1"
}

The chunk record should add chunkId, heading path, element IDs, ordinal position, cleaned text, links, and retrieval fields such as lexical terms and embeddings. Keeping raw HTML separate from derived records lets you improve extraction or chunking later without rediscovering the original page.

Fetch responsibly and choose the right rendering path

Begin with a direct HTTP fetch for pages whose server response contains the article text. It is usually simpler to cache, faster to process, and easier to debug than browser rendering. Inspect the returned HTML rather than assuming a successful 200 means the document is usable. A successful response can still contain only an application shell, a consent overlay, or a login prompt.

Use a browser-rendered fallback when the server HTML lacks the content but the rendered page contains it. Do not use a generic “wait until the network is idle” rule as your definition of readiness. Modern pages can maintain analytics, polling, and other background traffic indefinitely. Instead, define an assertion appropriate to the target: a known article selector, a heading, or a content API response.

Crawling policy belongs in this stage. Read and apply robots.txt before scheduling fetches, retain the policy decision with the crawl record, and treat authorization separately. The Robots Exclusion Protocol describes crawler rules but is not an access-control mechanism; a page that is technically reachable is not automatically appropriate to crawl or index. See RFC 9309.

Also set operational boundaries early:

• Normalize URLs and remove tracking parameters where appropriate.
• Respect host-level concurrency and back off after errors.
• Deduplicate canonical URLs and content hashes.
• Keep authenticated content in a separately authorized ingestion path.
• Capture fetch failures as first-class records rather than silently dropping them.

Extract dominant content and remove boilerplate

The first extraction question is simple: what was the author trying to communicate on this page?

Semantic HTML offers a useful baseline. The main element represents dominant document content and should exclude repeated navigation, sidebars, logos, search forms, and copyright material. Real pages do not always honor that convention, so treat it as a strong signal rather than a guarantee.

A layered extraction strategy is more resilient than one selector:

1. Prefer a unique main, article, or site-specific content container.
2. Remove known repeated regions such as nav, footer, headers, consent dialogs, and related-content modules.
3. Score remaining DOM blocks using text density, link density, repetition across pages, semantic tags, and visual or DOM position.
4. Retain uncertain blocks for inspection instead of assuming every heuristic is correct.

This is not cosmetic cleanup. Research on DOM-aware boilerplate removal identifies main-content extraction as important for downstream NLP and information retrieval, and reports improved retrieval after boilerplate removal in its evaluation. See Web2Text: Deep Structured Boilerplate Removal.

For example, consider a refund policy page with a top navigation entry called “Refunds,” a 250-word policy section, and a footer that repeats “Refund policy” across every page. A naive pipeline can produce many high-scoring footer chunks because they share keywords with the query. A cleaned content representation makes the policy section the retrievable evidence instead.

Preserve typed elements before creating chunks

Avoid immediately converting cleaned HTML into one long string. First partition it into typed elements such as title, narrative paragraph, list item, table, code block, link, header, and footer. This creates a stable intermediate representation for multiple downstream uses: chunking, indexing, rendering citations, quality checks, and reprocessing.

This approach aligns with Unstructured’s partitioning model, which normalizes heterogeneous documents into document elements. You do not need a particular library to adopt the design. The important decision is that structural distinctions survive extraction.

An internal element schema can be modest:

{
  "elementId": "el_018",
  "type": "narrative_text",
  "text": "Refunds are issued to the original payment method...",
  "headingPath": ["Refund policy", "Processing time"],
  "sourceUrl": "https://docs.example.com/refunds",
  "domPath": "article > section:nth-of-type(2) > p:nth-of-type(1)",
  "ordinal": 18
}

The headingPath is especially valuable. The sentence “It may take up to 10 business days” is ambiguous alone. Attached to Refund policy > Processing time, it becomes understandable to both retrievers and users.

Tables deserve special handling. Flattening a pricing table, compatibility matrix, or policy exception into prose can scramble its rows and columns. Keep a table as an atomic unit when possible; if it must be split, include headers and a clear continuation marker in every fragment.

Chunk around meaning, not only token limits

Chunks should be small enough to retrieve precisely and large enough to establish a claim. The right unit is commonly a section: its heading plus adjacent paragraphs, bullets, or a table. Token ceilings still matter, but they should be a fallback for unusually long elements rather than the primary boundary detector.

A simple algorithm looks like this:

for each element in page order:
  if element begins a new heading section:
    close the current chunk
  if element is a table or code block:
    emit it as its own chunk
  otherwise:
    append it while the target size is respected
  if one element exceeds the maximum size:
    split it conservatively and mark continuation

This design is consistent with structure-aware chunking guidance: use detected elements, preserve title boundaries where useful, isolate tables, and split oversized material only when necessary.

Every emitted chunk needs a reconstruction path. At minimum, store:

• sourceUrl and canonical URL
• document title and heading path
• ordered source element IDs
• character or token offsets when available
• extraction and chunking version
• neighboring chunk IDs
• content hash

That data makes it possible to show the source section to a user, merge adjacent chunks for answer context, detect stale indexes, and answer the most important debugging question: “What page evidence produced this response?”

Build the pipeline around evidence, not opaque text. If you want to test a URL-to-markdown workflow while designing your retrieval records, start with PagePith.

Use hybrid retrieval as the practical baseline

Dense retrieval is useful for paraphrases and conceptual matches. Lexical retrieval remains useful for exact policy terms, error codes, product names, URLs, and rare identifiers. Treating either method as universally sufficient creates avoidable blind spots.

A solid first design retrieves candidates from both systems:

1. Run a full-text query over chunk text, title, and heading path.
2. Run a vector query over the same cleaned chunk representation.
3. Combine the ranked lists with reciprocal-rank fusion (RRF).
4. Apply filters for site, locale, document type, recency, or access scope.
5. Rerank the top combined candidates with a more expensive relevance model.
6. Send only the best, nonredundant evidence to the language model.

The BEIR benchmark describes BM25 as a robust baseline and finds stronger approaches can incur additional computation. Elastic’s hybrid-search guidance similarly presents lexical-plus-vector retrieval as a practical pattern.

RRF is an attractive early fusion method because it works on rank positions rather than pretending keyword and vector scores share a scale. For each document, sum a contribution like 1 / (k + rank) for each retrieval list where it appears. This rewards chunks that rank well in either system and often rewards agreement between systems.

Do not jump straight from retrieval to generation. Retrieve broadly, then rerank narrowly. The initial stage optimizes recall; the reranker and context builder optimize precision and ordering. This distinction matters when a long page contains several nearby sections that all mention the query but only one actually answers it.

Keep context focused and answers attributable

Giving the model an entire page is rarely a reliable substitute for retrieval. It can dilute useful evidence with related but conflicting sections, and it weakens citation precision. Research on long-context behavior has found that models can perform worse when relevant material sits in the middle of an input, rather than near its beginning or end. See Lost in the Middle.

Build a context assembly step that:

• removes near-duplicate chunks,
• retains heading paths and source URLs,
• merges immediate neighbors only when they complete a claim,
• favors direct answers over topical mentions,
• orders evidence predictably, and
• requires each factual answer claim to map to one or more retrieved chunks.

If the retrieved context does not support an answer, the application should say so or request a narrower question. A fluent response without a source span is not a successful retrieval result.

Evaluate retrieval separately from generation

Create a small labeled set before changing embedding models or prompts. For each query, record the relevant page or chunk IDs and, where useful, the exact supporting passage. Start with representative failures:

• a query whose answer is in a table,
• a query with an exact error code,
• a paraphrased support question,
• a question where the topically similar page is wrong,
• a question whose evidence is buried late in a long article.

Measure retrieval using precision at k, recall at k, and a ranking-sensitive metric such as average precision. Precision asks what share of retrieved records are relevant; recall asks how much of the relevant material was found. NIST’s TREC materials describe these measures and document-level relevance judgments in retrieval evaluation. See the TREC overview.

Then evaluate generated answers independently. Did the selected context contain the answer? Did the response stay faithful to that context? Did it directly address the question? RAGAS separates context quality, faithfulness, and answer relevance, which is a useful mental model even when your evaluation uses human review.

This separation prevents a common mistake: rewriting a prompt to hide a retrieval failure. If recall is low, inspect crawling, extraction, chunk boundaries, and candidate retrieval. If retrieved context is correct but the answer is wrong, inspect context formatting, citation rules, and generation behavior.

A small, honest PagePith demonstration

The supplied PagePith proof shows a fetch-tier request for the Web2Text paper at https://arxiv.org/abs/1801.02607. It reports a content length of 7,464 and returns Markdown containing the information-retrieval category, the arXiv identifier, submission history, the paper title, authors, and links to the PDF and experimental HTML.

That result illustrates the first valuable artifact in a page RAG pipeline: a captured, inspectable text representation associated with the requested URL. It does not by itself demonstrate boilerplate classification, typed element partitioning, chunk creation, embeddings, hybrid search, reranking, or answer generation. Those remain pipeline stages your application must implement and evaluate.

The practical lesson is to make every stage as inspectable as that fetch result. Store the fetch mode, cleaned representation, element and chunk IDs, ranked candidates, and cited spans. When retrieval fails, you can then determine whether the URL was fetched incorrectly, content was removed, a boundary was poor, or ranking selected the wrong evidence.

Build for diagnosis from day one

Reliable web-page RAG is not a single model choice. It is a chain of decisions about what counts as content, how structure is retained, which records can be retrieved, and whether an answer remains tied to evidence.

Start with a narrow corpus and a labeled query set. Make extraction visible. Preserve provenance through chunks. Add lexical retrieval beside vectors. Rerank candidates, limit context, and measure retrieval before celebrating generated prose. This produces a system developers can improve deliberately rather than a black box that occasionally sounds correct.

Ready to prototype the ingestion side with an inspectable URL result? Sign up for PagePith.