Build a Searchable Knowledge Base from PDFs

A team can build a searchable knowledge base from PDFs and web pages with familiar components: a parser, crawler, object store, embedding model, and vector index. The difficult part is making those components behave as one reliable system.

A PDF and a public documentation page may arrive through different connectors, but they should converge on the same normalized document model before embedding. That model needs clean content, structural boundaries, provenance, version information, and permission metadata. Without it, retrieval may surface a stale duplicate, lose the page that supports an answer, or mix restricted internal guidance with public material.

This article presents a durable implementation pattern: acquire responsibly, extract structure, normalize metadata, deduplicate, chunk by meaning, index for retrieval, and refresh only what changed.

Treat ingestion as a source-normalization pipeline

Avoid designing separate downstream systems for PDFs and web pages. Instead, use source-specific acquisition and parsing adapters that emit a common representation.

A useful pipeline looks like this:

PDF files ─────┐
               ├─> acquire -> parse -> normalize -> deduplicate
Web pages ─────┘                                  -> chunk -> embed -> index
                                                        │
                                                        └-> provenance + access metadata

The source adapters differ at the beginning:

• PDFs need a strategy for digital text, layout-heavy documents, scanned pages, and tables.
• Web pages need canonical URLs, crawl policies, HTML-to-content extraction, and change detection.

From normalization onward, both should have the same contract. Every document should have a stable identity, an extracted body, ordered structural elements, and metadata that lets a retrieval system explain where a result came from.

This prevents a common failure mode: treating a PDF as an opaque blob and a web page as an unversioned string. Both approaches discard information you will need for filtering, citations, debugging, and refreshes.

Acquire PDFs and web pages responsibly

Choose the PDF extraction strategy per file

There is no universal PDF parser. A simple digitally generated policy document may work with fast text extraction, while a scanned report or a document with multi-column layouts can require OCR and layout analysis. Unstructured’s strategy documentation describes fast, layout-aware, OCR-oriented, and automatic selection approaches for PDF processing.

Route files according to observable characteristics and maintain a fallback path. For example:

pdf_routing:
  digital_text_with_simple_layout: fast_text
  multi_column_or_table_heavy: layout_aware
  scanned_or_image_only: ocr
  unknown: automatic_then_review_on_low_quality

The extraction result should preserve more than paragraph text. A structured document representation can retain reading order and treat tables as first-class content, rather than hoping a later chunker reconstructs the original meaning. Docling’s document model illustrates why a document tree is useful for preserving hierarchy, table items, and reading order.

Tables deserve special handling. Flattening rows and columns into a run-on sentence can destroy relationships such as product-to-price, threshold-to-action, or date-to-status. Where your parser supports it, retain a textual rendition for retrieval and a structured form for rendering or downstream interpretation. Unstructured’s PDF implementation, for example, documents table structure inference that can preserve rows and cells as HTML metadata alongside text. See its PDF partitioning source documentation.

Crawl public pages under an explicit policy

For public web content, define an allowlist of domains and paths before the first fetch. Keep the original URL, the final resolved URL, the canonical URL when available, fetch timestamp, and HTTP response metadata.

Crawling policy is not an optional afterthought. The Robots Exclusion Protocol defines robots.txt as the mechanism through which site operators communicate automated-client access rules for URI paths. Review and implement those rules as part of your acquisition contract, as described in RFC 9309.

For each permitted page, extract the primary content rather than blindly embedding navigation, cookie notices, sidebars, and footer links. Preserve headings, code blocks, lists, and tables as ordered elements. Those boundaries become important in chunking.

Define a shared metadata and provenance schema

Metadata is what turns retrieved text into usable evidence. It supports source tracking and query-time filtering, as noted in LlamaIndex’s metadata guidance.

Use a schema that works across every source type. Here is a practical normalized record:

{
  "document_id": "policy:travel-expenses",
  "source_type": "pdf",
  "source_uri": "s3://internal/policies/travel-expenses.pdf",
  "canonical_url": null,
  "title": "Travel Expenses Policy",
  "location": { "page_start": 4, "page_end": 4, "section": "Meals" },
  "content_hash": "sha256:...",
  "source_version": "2026-08-12",
  "fetched_at": "2026-08-24T00:00:00Z",
  "published_at": null,
  "access_scope": ["employees"],
  "department": "finance",
  "language": "en"
}

For a web document, source_uri and canonical_url can both be URLs, while location may contain an anchor, heading path, or DOM-derived section identifier instead of page numbers.

Keep document-level metadata separate from chunk-level metadata. Every chunk should inherit the fields needed for retrieval filtering and provenance, then add its own chunk_id, sequence number, character offsets where meaningful, and precise location. This lets an answer show a PDF page or a web section without pretending all sources have the same kind of location.

Access metadata must be applied before retrieval candidates are allowed into an answer context. Do not rely on a prompt instruction to prevent sensitive chunks from being retrieved.

Ready to establish a repeatable ingestion boundary? Use a clear fetch-and-normalize step before your embedding pipeline, then iterate on parsing and chunking without changing your retrieval contract. Start with PagePith.

Deduplicate by identity and content

Duplicate handling needs two related checks.

First, define a stable document identity. A policy file and its revised upload should share an ID if they represent the same logical document. A web page’s canonical URL is often a reasonable starting point, provided you normalize tracking parameters and redirects.

Second, calculate a content hash from a normalized representation. It catches exact repeated content across locations and tells the pipeline whether a known logical document has changed.

The lifecycle is straightforward:

1. Look up document_id.
2. Compare the current normalized content hash with the stored hash.
3. Skip indexing if the ID and hash are unchanged.
4. Reprocess and upsert chunks if the ID matches but the hash changed.
5. Mark previous chunks for deletion or retirement as part of the same update.

This is consistent with the document-ID-to-hash approach described in the LlamaIndex ingestion pipeline documentation, where unchanged duplicates can be skipped and changed documents reprocessed.

Be deliberate about near duplicates. Two PDFs with slightly different cover pages may contain the same policy body. Use exact hashes for safe automated skipping, and flag high-similarity documents for review rather than automatically merging them. Similar wording does not always mean equivalent authority or permission scope.

Chunk from structure, not only token counts

Fixed-size token splitting is a useful fallback, not a complete chunking policy. Chunk boundaries influence retrieval quality: chunks that are too broad include distracting material, while chunks that are too small can lose the context needed to answer a question. Research on document segmentation for RAG identifies this tradeoff and shows why segmentation deserves separate attention.

Start with semantic structure:

• Split at heading transitions and retain the heading path.
• Keep a short section together when it answers one question coherently.
• Treat a table plus its caption, header, and nearby explanatory sentence as a unit when feasible.
• Preserve code blocks without splitting them mid-example.
• Apply a token ceiling only after structural grouping.
• If a large section must split, repeat the relevant heading path in each child chunk.

For example, a web page section might become:

Title: API Authentication
Heading path: Guides > Authentication > Rotating API keys
Chunk body: [two explanatory paragraphs]
Attached context: Key rotation interval table

A chunk for a PDF should additionally retain page provenance:

Title: Travel Expenses Policy
Heading path: Reimbursements > Meals
Pages: 4-4
Chunk body: [policy paragraph plus threshold table]

Use modest overlap only when a split breaks a necessary dependency. Large universal overlap can multiply near-identical vectors, make debugging harder, and favor repeated boilerplate in retrieval results.

Refresh efficiently and preserve history

Refreshing every web page and every PDF on every run is wasteful and increases the chance of accidental churn. For web sources, store HTTP validators such as ETag and Last-Modified. HTTP conditional requests can use those validators, and an unchanged representation can return 304 Not Modified without transferring the body, as specified in RFC 9110.

A refresh worker can follow this sequence:

1. Check whether the source is due for refresh.
2. Request it conditionally when validators are available.
3. On 304, record a successful check and keep current chunks.
4. On changed content, parse and normalize it again.
5. Compare the new normalized hash with the stored value.
6. Replace or version affected chunks atomically.

For PDFs, use storage events, file timestamps, or a scheduled manifest scan. Keep the original artifact when policy permits, plus the parser version and extraction strategy used. If a parser upgrade changes output, you can then distinguish source change from pipeline change.

An honest PagePith demonstration

The supplied PagePith proof shows a fetch-tier request for the Unstructured partitioning-strategies URL. It returned the title “Partitioning strategies - Unstructured”, a reported content length of 2585, and a Markdown excerpt discussing PDF preprocessing strategies.

That is a useful demonstration of the first stage of this architecture: acquiring a web source as content that can enter a normalization pipeline. The proof does not establish PDF parsing, crawling breadth, embeddings, vector indexing, permissions enforcement, or automated refresh behavior. Those remain responsibilities to verify and implement in the surrounding knowledge-base system.

In practice, a fetch result should be converted into your normalized web-document record, assigned a canonical identity and content hash, and then sent through the same chunking and indexing stages used for approved PDFs.

Evaluate ingestion, retrieval, and answers separately

Do not measure the system solely by whether a chat response sounds plausible. Test three layers:

1. Ingestion quality: Did extraction retain the required headings, tables, page numbers, and source locations?
2. Retrieval quality: For a test question, did the correct permitted chunks appear in the top results?
3. Answer quality: Does the answer stay supported by the retrieved material and cite the appropriate PDF page or web section?

This separation mirrors RAG evaluation dimensions such as context relevance, answer faithfulness, and answer relevance discussed by ARES. It makes failures actionable: a poor result may originate in a broken table parse, a weak chunk boundary, an incorrect filter, or an answer-generation step—not necessarily the embedding model.

Build a small evaluation set from real internal questions and expected source locations. Include difficult cases: scanned PDFs, tables, revised documents, pages with near-duplicate content, and queries that must not cross permission boundaries. Re-run it after changes to parsers, chunking rules, or indexing behavior.

Build for explainable updates

A searchable knowledge base is not finished when the first vectors are stored. It is durable when each result can be traced to a current, permitted source; when tables and sections retain their meaning; and when a refresh processes only what changed.

The core design decision is simple: normalize PDFs and web pages into one structured, versioned contract before retrieval. That gives your team a stable place to improve parsers, tune chunking, add source types, and debug answer evidence without rebuilding the entire system.

Start building your ingestion workflow with PagePith.