How to Extract Figures, Captions, and Nearby Explanations from Technical PDFs

A figure is a document region, not just an image

To extract figures and captions from technical PDFs reliably, start by rejecting a tempting assumption: a figure is not always an embedded bitmap.

A PDF page may combine selectable text, raster images, vector paths, fonts, and other content streams. As a result, a diagram can be a collection of vector drawing commands, a chart can combine text and lines, and a scanned figure may be part of one full-page image. PyMuPDF specifically notes that vector graphics cannot be extracted directly as images; extracting their paths or rendering a clipped page region is the appropriate alternative. PyMuPDF FAQ

That distinction changes the goal. Instead of producing a directory of image files, produce a linked record:

• a stable document identifier
• page index
• figure bounding box
• rendered figure crop path
• caption text and caption bounding box
• figure label, such as Figure 3, when available
• nearby explanatory text in reading order
• extraction mode and validation flags

This preserves the evidence a researcher, engineer, or reviewer needs to answer basic questions: Where did this visual come from? Which caption belongs to it? What does the surrounding document say it means?

Model the task as detection, association, and context collection

A robust pipeline has three separate jobs. Treating them separately makes failures easier to inspect and improve.

1. Detect candidate figure regions

Candidate regions can come from several sources:

• image blocks reported by a PDF parser
• clusters of vector drawing paths
• rendered page regions proposed near a caption
• whole-page or partial-page visual detection for scanned documents

PyMuPDF can return text and image blocks with bounding boxes and image content, and can expose vector drawing paths. Image extraction documentation Drawing and graphics documentation

Do not equate Page.get_images() with “all figures.” An embedded-image list can omit inline-image occurrences and says nothing about vector diagrams. A block-oriented view is better for a first pass, while rendering a proposed rectangle is the safer output path for mixed or vector-heavy visuals.

2. Find and bind captions

Captions usually provide the strongest anchor because they often contain a recognizable label: Figure 2, Fig. 2, or FIGURE 2. But locating the first matching line is not enough. A caption may span multiple blocks, wrap across columns, or sit above rather than below its figure.

PDFFigures2 illustrates a useful association pattern for scholarly documents: find caption locations, identify non-textual elements, construct full captions, classify text regions, generate nearby figure proposals, and score the caption-to-region link. Its output keeps page, figure box, caption box, caption text, figure name, and related fields together. PDFFigures2

For a general-purpose pipeline, use similar signals:

1. Caption syntax: Does the text begin with a figure label?
2. Distance: Is the candidate region adjacent to the caption?
3. Direction: Is the figure immediately above or below, consistent with the document’s local layout?
4. Intervening content: Are there unrelated paragraphs or another caption between the two?
5. Size and shape: Is the candidate region substantial enough to plausibly be a figure?
6. Uniqueness: Does one candidate clearly score better than alternatives?

Store the score and reasons, not only the winning pair. An explicit needs_review flag is more useful than quietly attaching the wrong chart to a caption.

3. Collect nearby explanation as context, not caption text

The paragraph that explains a figure is often separate from its caption. It may appear before the figure (“Figure 4 shows…”), after it, or in a neighboring column. Mixing that prose into the caption loses an important semantic boundary.

Instead, collect context as coordinate-aware text blocks. Keep the caption as one field, then preserve a small set of blocks immediately before and after the figure/caption pair in reading order. This lets downstream systems display a compact evidence bundle without asserting that every nearby sentence is part of the formal caption.

A transparent PyMuPDF baseline

A lightweight baseline is valuable even if you later adopt a more specialized parser. It gives you inspectable geometry, clear failure modes, and a way to create labeled examples for evaluation.

The following simplified example does four things:

1. extracts text and image blocks from a page,
2. identifies likely caption starts,
3. pairs each caption with the nearest image block above it,
4. writes a structured record including nearby text.

import json
import re
from pathlib import Path

import fitz  # PyMuPDF

CAPTION_RE = re.compile(r"^\s*(fig(?:ure)?\.?\s*\d+[A-Za-z]?)\b", re.I)


def block_text(block):
    """Join spans from a PyMuPDF text block."""
    return " ".join(
        span["text"]
        for line in block.get("lines", [])
        for span in line.get("spans", [])
    ).strip()


def rect_distance_above(figure_bbox, caption_bbox):
    """Prefer a figure whose bottom is close to the caption's top."""
    fx0, fy0, fx1, fy1 = figure_bbox
    cx0, cy0, cx1, cy1 = caption_bbox
    vertical_gap = max(0, cy0 - fy1)
    horizontal_miss = max(0, max(fx0 - cx1, cx0 - fx1))
    return vertical_gap + 0.25 * horizontal_miss


def nearby_text(text_blocks, anchor_y, window=180):
    selected = []
    for block in text_blocks:
        x0, y0, x1, y1 = block["bbox"]
        if abs(((y0 + y1) / 2) - anchor_y) <= window:
            selected.append({"bbox": block["bbox"], "text": block_text(block)})
    return selected


def extract(pdf_path, out_dir):
    pdf_path = Path(pdf_path)
    out_dir = Path(out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    doc = fitz.open(pdf_path)
    records = []

    for page_index, page in enumerate(doc):
        data = page.get_text("dict")
        text_blocks = [b for b in data["blocks"] if b["type"] == 0]
        image_blocks = [b for b in data["blocks"] if b["type"] == 1]

        captions = [
            block for block in text_blocks
            if CAPTION_RE.match(block_text(block))
        ]

        for caption_index, caption in enumerate(captions):
            caption_bbox = caption["bbox"]
            candidates = [
                image for image in image_blocks
                if image["bbox"][3] <= caption_bbox[1]
            ]

            if not candidates:
                continue

            figure = min(
                candidates,
                key=lambda item: rect_distance_above(item["bbox"], caption_bbox),
            )
            figure_rect = fitz.Rect(figure["bbox"])
            crop_name = f"page-{page_index}-figure-{caption_index}.png"
            crop_path = out_dir / crop_name

            # Rendering preserves a complete page region, including vector content
            # that may be visually part of an otherwise image-based figure.
            pix = page.get_pixmap(matrix=fitz.Matrix(2, 2), clip=figure_rect)
            pix.save(crop_path)

            anchor_y = (figure_rect.y0 + figure_rect.y1) / 2
            records.append({
                "document_id": pdf_path.stem,
                "page_index": page_index,
                "figure_crop_path": str(crop_path),
                "figure_bbox": list(figure["bbox"]),
                "caption_bbox": list(caption_bbox),
                "caption_text": block_text(caption),
                "nearby_text_blocks": nearby_text(text_blocks, anchor_y),
                "extraction_mode": "image-block-nearest-caption-baseline",
                "validation_flags": ["heuristic_pairing", "review_recommended"],
            })

    return records


records = extract("paper.pdf", "figure-crops")
Path("figures.json").write_text(json.dumps(records, indent=2), encoding="utf-8")

This is intentionally a baseline, not a claim of complete figure extraction. It will miss figures that contain no raster image block, may pair incorrectly in multi-column layouts, and does not reconstruct multi-block captions. Its value is that every decision is visible in figures.json and every crop can be reviewed against the source page.

Working with a collection of technical PDFs? Use the same structured record from the beginning so page coordinates, captions, and explanatory text remain connected as you iterate. Explore PagePith.

Handle vector, composite, and scanned PDFs differently

One extractor mode is rarely enough. Route documents or regions through a small cascade instead.

Born-digital PDFs with raster figures

Start with native text blocks and image blocks. The baseline above can identify straightforward cases, then render the selected page rectangle rather than depending exclusively on raw embedded image bytes. Rendering is useful when labels, borders, or annotations are separate PDF objects around a bitmap.

Vector and composite diagrams

For line drawings, plots, CAD-style diagrams, and text-heavy composites, an embedded image may not exist. PyMuPDF’s drawing APIs can reveal vector paths, but the most faithful visual artifact is commonly a rendered crop of the associated page region. Drawing and graphics documentation PyMuPDF FAQ

This is why caption-first region proposal is practical: the caption supplies an anchor; nearby non-textual content supplies a crop candidate; page rendering preserves the combination.

Scanned and hybrid PDFs

A scan may have no usable native text layer, so caption detection must depend on OCR. OCRmyPDF is designed to add a searchable OCR text layer to scanned PDFs. OCRmyPDF documentation

Keep OCR provenance explicit. For example, set text_source to native, ocr, or mixed, and retain the original page index and bounding boxes. OCR errors in a figure number can change an association decision, so lower-confidence OCR captions deserve additional review.

Research on mixed raster and vector PDFs also treats scanned pages as a distinct challenge, rather than assuming a single text-extraction approach covers every page type. Figure and Figure Caption Extraction for Mixed Raster and Vector PDFs

When to use PDFFigures2 or Docling

A custom PyMuPDF workflow is a strong foundation when you need predictable output, domain-specific rules, and direct control over validation. But higher-level tools can reduce implementation work in the right document class.

PDFFigures2 is a worthwhile baseline for scholarly PDFs, particularly when its figure/table/caption model aligns with your inputs. It explicitly returns figure and caption geometry and can export rasterized figures. However, its own documentation identifies difficult cases including unusual layouts, text-heavy figures, nearby figures, rotated text, and documents outside its primary focus. Treat it as an evaluated component, not a universal parser. PDFFigures2

Docling is useful when the output needs richer document structure. Its document model includes picture items with captions, provenance, and image access, and its examples show exporting figure and table images alongside document output. Docling document reference Docling figure-export example

In either case, preserve your own acceptance criteria. A tool-generated association should still be auditable through a page number, figure box, caption box, and a rendered crop.

Evaluate the links, not only detection

An image crop can look correct while belonging to the wrong caption. Evaluate the pipeline at multiple levels and across at least three sets: born-digital raster PDFs, vector/composite PDFs, and scanned or hybrid PDFs.

Track these separately:

• Caption detection: Was a caption found, and was its full text recovered?
• Figure-region overlap: Does the predicted rectangle cover the intended visual without excessive neighboring material?
• Association accuracy: Is the detected caption paired with the correct figure?
• Context precision: Are the retained nearby blocks relevant explanation rather than unrelated column text?
• Provenance completeness: Does every result include page, figure box, caption box, source mode, and a review status?

For region detection, intersection-over-union is a useful geometric measure, but it cannot measure semantic correctness of a caption link. Pair-level review is essential for high-stakes engineering, research, and compliance workflows.

An honest PagePith demonstration

The supplied PagePith proof shows a fetch-tier retrieval of the PDFFigures2 repository URL. It returned the page title, “GitHub - allenai/pdffigures2: Given a scholarly PDF, extract figures, tables, captions, and section titles,” and a Markdown excerpt beginning with the project’s input/output description. That is useful evidence that PagePith can retrieve readable page content from this repository page for research or documentation workflows.

It does not demonstrate that PagePith executed PDFFigures2, parsed a PDF, detected a figure, or validated a caption association. Those capabilities are not established by the supplied proof. The extraction workflow in this article should therefore be implemented and evaluated with the PDF-processing tools described above.

Build for reviewable evidence

The reliable output is not an isolated PNG or a single OCR string. It is a compact evidence package: rendered figure region, caption, nearby explanatory blocks, coordinate provenance, extraction mode, and uncertainty markers.

That design accommodates the real diversity of technical PDFs. It also makes every automated decision inspectable, whether you start with PyMuPDF, test a scholarly parser such as PDFFigures2, or use a structured-document approach such as Docling.

Start exploring PagePith