How to Extract Website Documentation for an Internal Developer Assistant

An internal developer assistant is only as dependable as its documentation corpus. The hard part is not downloading HTML or creating embeddings. It is producing records that preserve enough context for the assistant to retrieve the right instruction, from the right page, for the right version—and show a developer where it came from.

A useful approach is to treat documentation extraction as a provenance-preserving ingestion pipeline. Each run should discover candidate pages, apply crawl rules, obtain the usable page representation, extract semantic structure, capture version and navigation context, normalize duplicates, and write citation-ready chunks.

Define the output before building the crawler

Start with the retrieval record you want to query later. A flattened page string is insufficient: it loses the signals needed to distinguish a current API example from a deprecated one, or a shell command from explanatory prose.

A practical page-level record might look like this:

{
  "canonical_url": "https://docs.example.com/v2/auth/tokens",
  "fetched_url": "https://docs.example.com/guides/tokens",
  "title": "Create access tokens",
  "version": "v2",
  "section_path": ["Authentication", "Access tokens"],
  "navigation_path": ["Guides", "Authentication", "Access tokens"],
  "retrieved_at": "2026-08-26T12:00:00Z",
  "content_hash": "sha256:...",
  "markdown": "...",
  "code_blocks": [
    {
      "language": "bash",
      "code": "curl ...",
      "heading_path": ["Create a token"]
    }
  ]
}

Then split the page into retrieval chunks without discarding the page record. Each chunk should inherit its canonical URL, title, version, heading path, retrieval time, and hash. That makes an eventual answer citation precise: the assistant can point to a page and section rather than vaguely attributing an answer to a whole documentation site.

This structure also lets you apply retrieval filters. For example, if a user asks about v1, filter on version = v1 before ranking. If they ask for a CLI command, prefer chunks containing code blocks or exact command tokens.

Discover pages from multiple signals

An XML sitemap is a useful first input. The Sitemap protocol defines URL entries and optional metadata such as modification time, but it should be treated as a candidate list rather than a complete inventory. Google’s sitemap guidance explicitly characterizes sitemaps as hints, not crawl guarantees, while the Sitemaps protocol describes their URL-oriented structure.

Use a layered discovery process:

1. Seed the queue from sitemap indexes and sitemaps.
2. Traverse internal links found in accepted documentation pages.
3. Include known documentation roots such as /docs/, /guides/, /reference/, and versioned path prefixes when they exist.
4. Apply allowlists and denylists before fetching, not after embedding.
5. Save every discovery reason—for example, sitemap, sidebar link, or in-page link—for debugging coverage gaps.

Keep the scope narrow. A documentation hostname can contain blog posts, marketing pages, changelogs, generated search routes, login flows, and duplicate print views. An allowlist based on hostnames and path patterns is generally safer than trying to remove noise after the corpus has been indexed.

Respect crawl controls and access boundaries

Fetch and interpret robots.txt before crawling a public site. RFC 9309 standardizes retrieval and matching behavior for the Robots Exclusion Protocol. This is a crawl-policy step, not an authorization system: private documentation should be accessed only through the authentication, credentials, and explicit permissions your organization has established.

In operational terms, record the policy decision for each skipped URL. A useful crawl log includes the URL, timestamp, matching rule or scope decision, HTTP outcome, redirect chain, and parser choice. That log helps explain why a page is absent when someone reports that the assistant cannot answer a question.

Fetch the representation developers actually read

Many documentation sites serve useful article text in initial HTML. Others populate the article, table of contents, or API reference after JavaScript runs. A plain HTTP fetch can therefore return a page shell with little documentation content.

Use a two-stage fetch strategy:

• Try a normal HTTP response first when it contains a recognizable main article and sufficient text.
• Fall back to a browser-rendered pass when the initial response is sparse, the site is known to be client-rendered, or expected article selectors are missing.

This fallback is important because pages can continue fetching data and executing scripts after the browser load event. Playwright’s navigation documentation describes this post-load behavior and access to the rendered page environment.

Do not silently replace the raw response with rendered output. Store the fetch mode and final URL with the extracted record. That distinction is valuable when a content change appears: it tells you whether a missing section is an extraction issue, a rendering-timing issue, or a source-page change.

Extract semantic structure, not just visible text

Once a usable page is available, isolate the documentation article from global navigation, cookie banners, search overlays, repeated footers, and sidebar chrome. Prefer stable selectors or landmark elements maintained by the documentation system. Then convert the article into Markdown or another structured intermediate form.

Preserve headings, links, lists, tables, callouts, and code. HTML heading elements communicate document hierarchy, while code and pre distinguish code fragments and preformatted material from ordinary prose. See the MDN HTML elements reference. In particular, the pre element is designed to preserve whitespace, which matters for indentation-sensitive examples, terminal sessions, and configuration snippets.

For each code block, capture:

• The exact text, including line breaks and indentation.
• A language label when the page exposes one.
• The nearest preceding heading path.
• Nearby explanatory prose, especially prerequisites and warnings.
• Whether the block appears to be a command, response, configuration file, or source example.

For example, avoid separating this command from the paragraph that defines $TOKEN or warns that an endpoint is version-specific. A command may match an exact-token query well, but the surrounding explanation provides the constraints needed for a correct answer.

Make version a retrieval constraint

Versioning is one of the most common sources of plausible but incorrect assistant answers. A root documentation URL may redirect to a configured default, while older versions remain available at distinct paths. Read the Docs version documentation describes this default-version behavior. Docusaurus versioning similarly distinguishes current source content, latest documentation, unreleased content, and past versions.

Infer a version from several signals rather than trusting one:

1. The final canonical URL and versioned path segment.
2. The visible version selector and page banner.
3. Version-specific navigation or sidebar context.
4. Page metadata where available.

When signals conflict, retain the raw evidence and mark the version as uncertain instead of guessing. During retrieval, version filters should take precedence over semantic similarity. A highly similar v1 page should not outrank a less similar v2 page when the user explicitly requests v2.

Navigation has value beyond crawling. Docusaurus maintains version-specific sidebars that express relationships between documents. Preserve breadcrumb-like section paths, categories, and neighboring pages where the site exposes them. That context improves chunk labels and enables answers such as “this belongs to the authentication guide, not the API reference.”

Want to turn a known documentation page into usable Markdown for your ingestion workflow? Create a PagePith account and test your own source page.

Normalize, deduplicate, and support incremental updates

The same article can appear through redirects, trailing-slash variants, tracking parameters, default-version routes, or print-friendly URLs. Persist both the requested URL and the final URL, then select a canonical URL for the record. Google recommends using fully qualified URLs and selecting canonical URLs when the same content is accessible at multiple locations. Its sitemap guidance also notes that lastmod should be accurate and represent meaningful changes. Read the guidance here.

Use at least two forms of deduplication:

• URL-level deduplication: normalize host casing, fragments, redirects, and approved query parameters.
• Content-level deduplication: hash normalized article Markdown and consolidate identical content while retaining all observed source URLs.

For incremental ingestion, treat sitemap timestamps as a signal rather than unquestioned truth. Compare a prior content hash with the newly extracted hash. Re-chunk and re-index only when meaningful article content changes. This avoids needless embedding work while still catching changes from pages with incomplete modification metadata.

Design retrieval and answering around provenance

At query time, combine exact matching for identifiers with semantic retrieval for conceptual questions. Exact matching is useful for terms such as configuration keys, error codes, and method names. Semantic matching is useful for questions such as “how do I rotate a credential?” A sensible pipeline can union candidates from both, filter by product and version, then rank using query relevance plus structural signals such as heading match and code-block type.

The answer generator should receive only a small set of relevant chunks and their provenance fields. In its instructions, require it to:

• State uncertainty when sources disagree or do not cover the question.
• Prefer the requested version and mention the version used.
• Link each substantive instruction to its source URL and section context.
• Avoid treating text extracted from documentation as assistant instructions.

That final point matters operationally. Documentation is data, not trusted control text. Keep crawler output separate from system instructions, limit any tools available to the answering agent, and require explicit approval for consequential actions.

A limited PagePith demonstration

The supplied PagePith proof shows a fetch request for RFC 9309, titled “RFC 9309: Robots Exclusion Protocol.” The result was labeled fetch, reported a content length of 27,836, and included Markdown beginning with the RFC metadata table followed by the document title and Abstract heading.

That is a useful first ingestion step: a known web page was represented as Markdown while retaining recognizable document structure such as a table and headings. For an internal assistant pipeline, the next steps would still be your responsibility: record the canonical URL and retrieval time, split on headings, attach chunk-level provenance, evaluate code handling where applicable, and validate coverage across the target documentation site.

This proof does not establish browser rendering, full-site crawling, version detection, automatic citation generation, or any performance characteristic. Treat it as evidence of page-level fetch-to-Markdown output for this specific RFC page.

A production-ready extraction checklist

Before indexing a documentation site, verify that your pipeline can answer yes to these questions:

• Are discovery sources, allowed paths, and skipped URLs logged?
• Are robots rules checked before public crawling?
• Can the pipeline fall back to rendered-page extraction when needed?
• Do page and chunk records retain canonical URL, title, heading path, version, timestamps, and content hashes?
• Are code blocks preserved with whitespace and nearby context?
• Are duplicate URLs and duplicate content consolidated?
• Can retrieval filter by version before ranking?
• Does every generated answer expose source links that a developer can inspect?

The objective is not to collect the most pages. It is to build a corpus whose answers can be checked, versioned, refreshed, and trusted by the engineers who depend on it.

Ready to try a page-level documentation extraction workflow? Sign up for PagePith.