How to Extract Glossary Terms from Technical Documentation

Glossary extraction is a relationship problem

To extract glossary terms from documentation, do not begin with a keyword list. Begin with a relationship: a technical term is useful only when it is connected to the definition intended by the documentation author.

That sounds straightforward until the source is a real documentation site. A definition might appear in a dedicated glossary, but it may also be embedded in a callout, paired in a description list, introduced below a heading, placed in a table, or explained across consecutive sentences. The visible text alone often loses the clues that make the relationship reliable.

A useful terminology pipeline therefore has two separate responsibilities:

1. Find candidate domain terms.
2. Find, normalize, and link the definition that belongs to each term.

This separation follows the shape of prior glossary and terminology-extraction work: terminology identification and definition extraction are related, but distinct tasks. IBM Research’s glossary work describes selecting terms and extracting their definitions as separate parts of glossary construction. The DEFT benchmark similarly separates identifying definitional content, labeling spans, and linking terms to definitions.

For a RAG application, the output should not be an untraceable array such as:

["embedding", "chunk", "reranker"]

It should be a collection of evidence-backed records that can answer questions such as:

• What does this term mean in this product’s documentation?
• Where was this definition stated?
• Is the same label used differently in another section or version?
• Is this an alias, acronym, API identifier, or canonical term?

Preserve structure before flattening text

The most common extraction mistake is converting HTML or rendered documentation directly into one plain-text blob. That is sometimes necessary for downstream models, but it should not be the first irreversible transformation.

HTML structure carries semantic signals. Description lists explicitly associate terms and descriptions through dl, dt, and dd; MDN notes that this pattern is commonly used to build glossaries. See the MDN description-list reference. Headings label document sections, while aside represents content tangential to the main flow, such as a sidebar, according to the HTML Standard.

Treat those structures as extraction features, not formatting noise.

Build a normalized document tree

Before extracting terms, normalize each source into ordered elements. A minimal internal model might include:

Document
├── Heading(level=2, text="Authentication")
├── Paragraph(text="A bearer token is ...")
├── Callout(kind="note", text="Tokens expire after ...")
├── DescriptionList
│   ├── Item(term="Access token", definition="...")
│   └── Item(term="Refresh token", definition="...")
├── Table
│   └── Row(cells=["Term", "Meaning"])
└── Aside(text="Related concepts: ...")

For every element, retain at least:

• document URL or stable source identifier;
• element type, such as paragraph, heading, table, or callout;
• source order;
• section path, for example Authentication > Token types;
• original text and normalized text;
• a stable element ID;
• parent or neighboring element IDs when available.

Document-processing systems can expose structure and metadata such as titles, list items, tables, parent-child relationships, and headers or footers. Unstructured’s document-element documentation is one example of why preserving this information is practical rather than theoretical.

A flattened document can still be generated later for embeddings or LLM prompts. The key is to retain a reference back to the structured source.

Extract high-confidence pairs from explicit layouts first

Not all candidate records deserve the same treatment. Start with layouts that already encode a term-definition relationship.

1. Description lists

A dt followed by one or more dd elements is a strong signal. Extract the dt as the candidate term and join the adjacent dd values as its definition.

for group in description_lists:
    for item in group.items:
        emit_pair(
            term=clean_term(item.dt.text),
            definition=clean_definition(item.dd.text),
            method="description_list",
            confidence=0.98,
            evidence_ids=[item.dt.id, item.dd.id]
        )

The confidence above is an application-defined ranking value, not a universal benchmark. Its purpose is to make explicit structural pairs rank above ambiguous inference-based pairs.

2. Definition tables

Many API references use tables with columns such as Term, Description, Parameter, Type, or Meaning. Detect header rows, classify column roles, then extract records only when a row provides a plausible label-to-description mapping.

Be careful with tables that describe configuration options. timeout may be a parameter name, not a glossary concept. It can still be valuable to index, but store a type such as parameter or field rather than labeling everything as a general technical term.

3. Headings and local explanatory text

A heading frequently acts as the term label, with the next paragraph serving as a definition or explanation. This is weaker than a description list because a section can begin with history, examples, or instructions instead of a definition.

Use a small local window:

1. Take the heading text as a candidate label.
2. Inspect the first one to three content elements below it.
3. Score sentences that explain what the label is, rather than how to configure or use it.
4. Stop at the next peer or higher-level heading.

The output should retain the heading and the selected sentence or span as separate evidence. That lets an evaluator see whether the model linked a definition to the right section.

4. Callouts, sidebars, and expandable content

Definitions are often hidden where a reader is expected to need context: notes, warnings, tooltips, expandable panels, and related-information sidebars. Do not discard these as boilerplate by default.

Instead, preserve the element type and assign a source-specific prior. For example, a note callout can be a definition candidate, but may receive a lower initial score than a dt/dd pair. An aside can provide useful alias or scope information even when its text should not become the primary definition.

Generate terms beyond explicit glossaries

Explicit markup will not cover the whole corpus. The next stage finds candidate terms in headings, paragraphs, code-adjacent prose, and table cells.

A robust candidate generator combines multiple signals instead of trusting frequency alone. Terminology-extraction research covers linguistic patterns, statistical evidence, multiword cohesion, and knowledge-base or ontology signals; see the Semantic Web survey.

Useful signals include:

Signal	Example use
Part-of-speech pattern	Prefer noun phrases such as retrieval pipeline or context window.
Multiword cohesion	Keep vector database together rather than indexing only vector.
Document frequency	Down-rank generic language that appears everywhere.
Section position	Boost terms introduced in conceptual sections and glossaries.
Typography and code form	Preserve identifiers such as HTTP 429, client_id, and Retry-After.
Existing vocabulary	Boost known product terms, aliases, and approved abbreviations.

Domain context matters. A word such as index has very different meanings in a database guide, a search platform, and a programming-language reference. Seed dictionaries and controlled vocabularies can improve ranking and disambiguation. This aligns with IBM’s domain-focused glossary findings and research on improving terminology ranking with ontology relationships, such as this ontology-assisted terminology-extraction study.

Do not force every candidate into the glossary. Keep a review queue for candidates with strong domain signals but no reliable definition.

Build terminology records with source evidence from the beginning. If you are evaluating a documentation ingestion workflow, create a PagePith account and test it against a representative source before committing to an index design.

Extract definitions with a tiered strategy

Definitions in free text are not reliably limited to the pattern X is Y. The DEFT task was specifically designed around definitions that may have no explicit marker, can cross sentence boundaries, and may require relation extraction rather than simple pattern matching. Read the DEFT task description.

Use a tiered strategy so that simple, auditable rules handle easy cases and more expensive inference is reserved for ambiguity.

Tier A: structural extraction

Extract pairs from description lists, glossary tables, labeled callouts, and clearly paired UI metadata. These records usually have the best provenance and need the least interpretation.

Tier B: definitional patterns

Search local windows for patterns such as:

• X is ...
• X refers to ...
• X represents ...
• X, also called Y, ...
• A X is a ...
• X: ...

Pattern matching is useful for proposing candidates, but not for deciding truth by itself. A sentence like “A cache is enabled by default” is grammatical but does not adequately define a cache.

Tier C: span and relation inference

For difficult prose, ask a classifier or model three constrained questions:

1. Does this sentence window contain a definition?
2. Which token span names the term?
3. Which span states the definition, and does it describe that term?

This mirrors the decomposition used by DEFT: definition detection, span labeling, and term-definition relation classification. Constrain the model to a section-sized context and require it to return source offsets or element IDs. Avoid accepting a rewritten definition when exact extraction is the goal; a paraphrase may be useful as a secondary field, but it is not the same as source evidence.

Store a provenance-aware glossary record

A string pair cannot explain why it exists or resolve conflict between two documentation pages. Store enough context to filter, audit, deduplicate, and update records.

{
  "canonical_term": "refresh token",
  "aliases": ["refresh-token"],
  "definition": "A credential used to obtain a new access token.",
  "record_type": "concept",
  "scope": "Authentication",
  "section_path": ["Authentication", "Token types"],
  "source_url": "https://docs.example.test/auth/tokens",
  "source_element_ids": ["h2-14", "p-15"],
  "source_span": {"start": 0, "end": 49},
  "extraction_method": "heading_local_definition",
  "confidence": 0.84,
  "captured_at": "2026-09-06T12:00:00Z"
}

The confidence field should describe confidence in the extracted link, not the importance of the concept. Keep those dimensions separate. A rare but critical error code may be highly important even if its prose definition is ambiguous.

For duplicate handling, normalize casing, punctuation, singular/plural variants, and obvious aliases. Then compare both the label and definition context. Do not blindly merge equal strings from different scopes: session can legitimately have multiple definitions within a large platform.

Index for exact lookup and conceptual questions

Glossary retrieval has two modes:

• Lexical lookup: “What is Retry-After?”
• Conceptual lookup: “Which response header tells my client when to retry?”

The first needs exact matching on identifiers, acronyms, error messages, and code-like tokens. The second benefits from semantic similarity between a user’s wording and the definition. Pinecone’s hybrid-search guidance explains why lexical and semantic signals address different failure modes, and its indexing overview calls out code, error messages, named entities, and similar tokens as important exact-match cases.

A practical index design is:

• lexical fields: canonical_term, aliases, identifiers, abbreviations;
• semantic text: definition plus a short scoped context;
• filters: document, product, version, section, record type, and confidence band;
• display fields: the original definition, URL, section path, and extraction method.

For example, embed this semantic text:

Term: refresh token
Scope: Authentication
Definition: A credential used to obtain a new access token.

But retain the standalone canonical term in a lexical index. When a user types the exact token, exact matching should be able to dominate. When they ask an indirect question, semantic retrieval can surface the definition. A reranker can then prefer records whose term, scope, and source context jointly fit the query.

Evaluate the pipeline as three separate problems

Do not report one vague “glossary accuracy” number. Evaluate each stage independently:

1. Definition detection: Of the locations marked as definitional, how many are correct? Measure precision and recall.
2. Span extraction: Does the extracted term and definition text match the expected spans? Use span-level precision, recall, and F1.
3. Term-definition linkage: Did the system pair the right term with the right definition? Measure pair-level accuracy or F1.

This breakdown is directly motivated by the task structure in DEFT. It also makes debugging far easier. A missed definition in a sidebar is a structural-coverage problem; an incorrect pair in a dense paragraph is a relation problem; a truncated explanation is a span-boundary problem.

Create a small gold set that deliberately includes description lists, tables, headings, callouts, aliases, multi-sentence definitions, and repeated terms with different scopes. Review false positives for generic nouns, navigation labels, and examples mistaken for definitions. Review false negatives for layout types your parser omitted.

A limited PagePith demonstration

The supplied PagePith proof shows a fetch request for the DEFT corpus task page that returned the page title, a reported content length of 12,943, and a Markdown excerpt beginning with the linked paper title and author information.

That is useful evidence for a narrow ingestion check: the fetched result preserved a document title and produced Markdown content for this source. The excerpt also contains surrounding page material, including metadata-correction text. In other words, this proof does not establish that PagePith automatically identifies every definition, reconstructs all term-definition pairs, or perfectly removes page chrome. Those steps still need the structural parsing, extraction, and evaluation workflow described above.

Start with one documentation section, inspect the captured structure and evidence fields, then expand only after you can explain each extracted pair.

Turn documentation into an auditable terminology layer

Reliable glossary extraction is not keyword mining. It is a provenance-preserving pipeline that respects layout, distinguishes candidate generation from definition linkage, ranks domain-specific evidence, and indexes records for both exact and semantic retrieval.

When every result retains its source location and extraction method, your RAG system can answer terminology questions with evidence rather than guesses—and your team can improve the pipeline using concrete failure cases.

Ready to test a documentation ingestion workflow on a real source? Sign up for PagePith.