Extracting Text and Metadata from PDFs with Python

Pulling the text out of a PDF looks like a one-liner, and for a single-column letter it is. The trouble starts at volume: a search index fed from ten thousand reports, a classifier trained on contracts, a compliance job that needs each file's author and creation date. Suddenly two-column layouts come out interleaved line by line, headers and page numbers repeat inside every paragraph, hyphenated words split in half, some files return (cid:72)(cid:101) instead of letters, and the "creation date" is a string like D:20260112093000+01'00' that no date parser accepts.

A PDF has no concept of paragraphs or reading order. It stores positioned glyph runs, and every extraction library reconstructs words, lines and blocks from coordinates using its own heuristics. Metadata lives in two different places that can disagree, and the navigation structure — bookmarks and page labels — is a third store again. This guide treats extraction as a pipeline: inspect the file, pick the extraction mode that matches the layout, pull metadata and outline in a normalised form, and measure text quality before anything downstream consumes it. Table extraction is a separate problem, covered in Extracting Tables from PDFs.

Prerequisites

python -m venv .venv && source .venv/bin/activate
pip install pymupdf pypdf pdfplumber
mkdir -p in out
# optional, for comparing against poppler's extractor
sudo apt-get install -y poppler-utils

PyMuPDF is the workhorse here: fast, with word, block and dictionary views of each page. pypdf is pure Python and useful where compiled wheels are not allowed, and for cross-checking. pdfplumber gives character-level control when you need to filter by font or position. Keep a small test corpus that includes at least one two-column document, one scan, one file with a custom-encoded font and one produced by Word — those four cover most extraction surprises.

Diagnostic: Inspect Before Extracting

Profile each file: page count, whether text exists, which fonts are embedded and whether they carry Unicode maps, and whether the layout has columns. Those facts pick the code path.

# pip install pymupdf
from pathlib import Path
import pymupdf

SOURCE = Path("in/annual-report.pdf")

def profile(pdf_path: Path, sample_pages: int = 3) -> dict:
    try:
        doc = pymupdf.open(pdf_path)
    except (pymupdf.FileDataError, RuntimeError) as exc:
        raise SystemExit(f"cannot open {pdf_path}: {exc}")
    with doc:
        info = {"pages": doc.page_count, "encrypted": doc.needs_pass,
                "has_toc": bool(doc.get_toc()), "fonts": set(), "columns": [], "chars": 0}
        for page in doc.pages(0, min(sample_pages, doc.page_count)):
            info["chars"] += len(page.get_text())
            for xref, ext, ftype, basefont, *_ in page.get_fonts(full=True):
                info["fonts"].add((basefont, ftype, ext))
            # count distinct left edges of text blocks as a rough column signal
            lefts = sorted({round(b[0] / 20) * 20 for b in page.get_text("blocks") if b[6] == 0})
            info["columns"].append(len([x for x in lefts if x < page.rect.width * 0.6]))
        info["replacement_ratio"] = 0.0
        sample = doc[0].get_text() if doc.page_count else ""
        if sample:
            bad = sample.count("\ufffd") + sample.count("(cid:")
            info["replacement_ratio"] = bad / len(sample)
    return info

if __name__ == "__main__":
    for key, value in profile(SOURCE).items():
        print(f"{key:>18}: {value}")

Read the profile like this. chars near zero on a document with pages means a scan — route it to Scanning and OCR Processing. A non-trivial replacement_ratio means fonts without a usable Unicode map, the subject of fix CID garbled characters in PDF text. Font type Type3 often signals glyphs drawn as pictures, which no text extractor can decode. Column counts above one on most sampled pages means plain get_text() will interleave columns.

The \ufffd escape in that snippet is the Unicode replacement character that extractors emit for glyphs they cannot map.

Core Implementation

Step 1: Extract Text in Reading Order

page.get_text("text") emits text in content-stream order — the order the generating program wrote it, which is often not reading order. sort=True sorts blocks top-to-bottom then left-to-right, which fixes most single-column documents. The TEXT_DEHYPHENATE flag joins words broken at line ends.

# pip install pymupdf
from pathlib import Path
import pymupdf

FLAGS = pymupdf.TEXT_DEHYPHENATE | pymupdf.TEXT_PRESERVE_WHITESPACE | pymupdf.TEXT_MEDIABOX_CLIP

def page_texts(pdf_path: Path) -> list[str]:
    """Return one reading-order string per page."""
    try:
        with pymupdf.open(pdf_path) as doc:
            return [page.get_text("text", flags=FLAGS, sort=True) for page in doc]
    except (pymupdf.FileDataError, RuntimeError) as exc:
        raise RuntimeError(f"extraction failed for {pdf_path}: {exc}") from exc

if __name__ == "__main__":
    pages = page_texts(Path("in/annual-report.pdf"))
    print(f"{len(pages)} pages, first 300 chars:\n{pages[0][:300]}")

TEXT_MEDIABOX_CLIP drops text positioned outside the visible page — printer marks and hidden notes some generators leave off-canvas. Leave it on unless you specifically need that text.

Step 2: Handle Multi-Column Layouts with Blocks

Sorting by y-coordinate alone reads straight across both columns. Assign each text block to a column by its left edge, then read each column top-to-bottom.

# pip install pymupdf
import pymupdf

def column_text(page: pymupdf.Page, gutter_ratio: float = 0.5) -> str:
    """Two-column reading order: full-width blocks in place, then left column, then right."""
    width = page.rect.width
    split = width * gutter_ratio
    blocks = [b for b in page.get_text("blocks", flags=pymupdf.TEXT_DEHYPHENATE) if b[6] == 0]
    full, left, right = [], [], []
    for x0, y0, x1, y1, text, *_ in blocks:
        if x0 < split and x1 > split + 20:          # spans the gutter: heading or wide figure caption
            full.append((y0, text))
        elif x0 < split:
            left.append((y0, text))
        else:
            right.append((y0, text))
    ordered = sorted(full) + sorted(left) + sorted(right)
    return "\n".join(text.strip() for _, text in ordered)

This is deliberately simple and works for the common report layout. Full-width blocks are emitted first, which is wrong when a full-width figure sits between column sections; for those layouts, split the page into horizontal bands at each full-width block and apply the column logic within each band. Three-column newsletters need a third bucket — derive boundaries from a histogram of block left edges rather than a fixed ratio.

Step 3: Strip Running Headers and Footers

Headers, footers and page numbers repeat on every page and pollute search indexes and embeddings. Detect lines that recur in the top or bottom band across most pages and remove them.

# pip install pymupdf
import re
from collections import Counter
import pymupdf

def running_lines(doc: pymupdf.Document, band: float = 0.08, min_share: float = 0.6) -> set[str]:
    """Normalised lines that appear in the header/footer band on most pages."""
    seen = Counter()
    for page in doc:
        h = page.rect.height
        lines = set()
        for x0, y0, x1, y1, text, *_ in page.get_text("blocks"):
            if y1 < h * band or y0 > h * (1 - band):
                norm = re.sub(r"\d+", "#", text.strip())     # 'Page 3 of 40' -> 'Page # of #'
                lines.add(norm)
        seen.update(lines)
    threshold = max(2, int(doc.page_count * min_share))
    return {line for line, count in seen.items() if count >= threshold}

def body_blocks(page: pymupdf.Page, running: set[str]) -> list[str]:
    return [
        text for *_, text, _, btype in [(*b[:5], b[5], b[6]) for b in page.get_text("blocks", sort=True)]
        if btype == 0 and re.sub(r"\d+", "#", text.strip()) not in running
    ]

Replacing digits with # before counting is what lets Page 3 of 40 and Page 4 of 40 count as the same line. The band and share thresholds work for typical reports; lower min_share for documents with different first-page layouts.

Step 4: Read Metadata from Both Stores

A PDF carries an Info dictionary (/Title, /Author, /CreationDate …) and, often, an XMP packet with the same facts in XML. Editors update one and not the other, so read both and record which you used.

# pip install pymupdf
import re
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pymupdf

PDF_DATE = re.compile(r"D:(\d{4})(\d{2})?(\d{2})?(\d{2})?(\d{2})?(\d{2})?([+\-Z])?(\d{2})?'?(\d{2})?")

def parse_pdf_date(value: str | None) -> datetime | None:
    """Parse 'D:20260112093000+01'00'' into an aware datetime; tolerate truncated forms."""
    if not value:
        return None
    m = PDF_DATE.match(value.strip())
    if not m:
        return None
    year, month, day, hour, minute, second, sign, tzh, tzm = m.groups()
    dt = datetime(int(year), int(month or 1), int(day or 1),
                  int(hour or 0), int(minute or 0), int(second or 0))
    if sign in ("+", "-"):
        offset = timedelta(hours=int(tzh or 0), minutes=int(tzm or 0))
        return dt.replace(tzinfo=timezone(offset if sign == "+" else -offset))
    if sign == "Z":
        return dt.replace(tzinfo=timezone.utc)
    return dt                                              # no zone given: leave naive

def read_metadata(pdf_path: Path) -> dict:
    with pymupdf.open(pdf_path) as doc:
        info = doc.metadata or {}
        xmp = doc.get_xml_metadata() or ""
    xmp_title = re.search(r"<dc:title>.*?<rdf:li[^>]*>(.*?)</rdf:li>", xmp, re.S)
    return {
        "title": (info.get("title") or (xmp_title.group(1) if xmp_title else "")).strip(),
        "author": (info.get("author") or "").strip(),
        "producer": info.get("producer") or "",
        "created": parse_pdf_date(info.get("creationDate")),
        "modified": parse_pdf_date(info.get("modDate")),
        "has_xmp": bool(xmp),
    }

Treat metadata as a hint rather than truth. Title is frequently a template name (Microsoft Word - Document1), and Author is whoever owned the machine the template was built on. The dedicated walkthrough in extract PDF metadata and bookmarks with Python covers XMP namespaces and cleaning rules in depth.

Step 5: Pull Bookmarks and Page Labels

Bookmarks give a document its section structure, which is valuable for chunking text by section. Page labels map physical page indices to printed numbers (iv, A-3), which matters when a user reports "the table on page 12".

# pip install pymupdf
from pathlib import Path
import pymupdf

def outline(pdf_path: Path) -> list[dict]:
    with pymupdf.open(pdf_path) as doc:
        rows = []
        for level, title, page_no in doc.get_toc(simple=True):
            label = doc[page_no - 1].get_label() if 0 < page_no <= doc.page_count else ""
            rows.append({"level": level, "title": title.strip(),
                         "page_index": page_no - 1, "page_label": label or str(page_no)})
        return rows

get_toc returns one-based page numbers and -1 for bookmarks whose target could not be resolved; the bounds check guards against both.

Step 6: Emit One Normalised Record per Document

# pip install pymupdf
import json
from pathlib import Path

def write_record(pdf_path: Path, pages: list[str], meta: dict, toc: list[dict], out: Path) -> None:
    record = {
        "file": pdf_path.name,
        "meta": {k: (v.isoformat() if hasattr(v, "isoformat") else v) for k, v in meta.items()},
        "toc": toc,
        "pages": [{"index": i, "text": t} for i, t in enumerate(pages)],
    }
    out.parent.mkdir(parents=True, exist_ok=True)
    with out.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(record, ensure_ascii=False) + "\n")

JSON Lines keeps one document per line, streams well, and loads straight into pandas with pd.read_json(path, lines=True) for the kind of downstream work in Extracting PDF Data into pandas.

Edge Cases and Variants

Pure-Python Environments with pypdf

Where compiled wheels are banned, pypdf covers text, metadata and outline. Its layout mode approximates the visual arrangement with spaces, which keeps simple tables and columns aligned:

# pip install pypdf
from pathlib import Path
from pypdf import PdfReader
from pypdf.errors import PdfReadError

def pypdf_extract(pdf_path: Path) -> dict:
    try:
        reader = PdfReader(pdf_path)
    except PdfReadError as exc:
        raise RuntimeError(f"{pdf_path}: {exc}") from exc
    meta = reader.metadata
    pages = [page.extract_text(extraction_mode="layout") for page in reader.pages]

    def walk(items, level=1):
        for item in items:
            if isinstance(item, list):
                yield from walk(item, level + 1)
            else:
                yield level, item.title, reader.get_destination_page_number(item)

    return {
        "title": meta.title if meta else None,
        "created": meta.creation_date if meta else None,   # already a datetime
        "toc": list(walk(reader.outline)),
        "pages": pages,
    }

pypdf is several times slower than PyMuPDF on large files. The trade-offs are laid out in PyMuPDF vs pypdf for text extraction.

Words Run Together or Split Apart

Some generators position every word individually without space characters; extractors infer spaces from gaps, and tight kerning defeats the inference, producing Totalrevenuefortheyear. Others letter-space headings, producing A N N U A L. Both are tuning problems of the word-gap threshold, covered in fix PDF text extraction missing spaces.

Rotated and Vertical Text

Table headers rotated 90 degrees and landscape pages come out as one character per line or in odd order. PyMuPDF's dict output includes a dir vector per line; filter or reorder lines whose direction is not (1, 0):

# pip install pymupdf
import pymupdf

def horizontal_text(page: pymupdf.Page) -> str:
    out = []
    for block in page.get_text("dict", sort=True)["blocks"]:
        for line in block.get("lines", []):
            dx, dy = line["dir"]
            if abs(dy) < 0.01:                     # keep horizontal lines only
                out.append("".join(span["text"] for span in line["spans"]))
    return "\n".join(out)

Validation

Measure text quality instead of eyeballing a sample. Four cheap metrics per page catch nearly every extraction failure: characters extracted, share of replacement or cid tokens, share of alphabetic characters, and the ratio of dictionary words for the document's language.

# pip install pymupdf
import re
from pathlib import Path
import pymupdf

COMMON = {"the", "and", "of", "to", "in", "for", "is", "on", "with", "by", "total", "date"}

def quality(pdf_path: Path) -> list[dict]:
    rows = []
    with pymupdf.open(pdf_path) as doc:
        for page in doc:
            text = page.get_text(sort=True)
            n = len(text) or 1
            words = re.findall(r"[A-Za-z]{2,}", text.lower())
            rows.append({
                "page": page.number + 1,
                "chars": len(text),
                "bad_ratio": (text.count("\ufffd") + 5 * text.count("(cid:")) / n,
                "alpha_ratio": sum(c.isalpha() for c in text) / n,
                "common_ratio": sum(w in COMMON for w in words) / (len(words) or 1),
            })
    return rows

def assert_quality(rows: list[dict]) -> None:
    bad = [r for r in rows if r["chars"] > 200 and (r["bad_ratio"] > 0.01 or r["common_ratio"] < 0.02)]
    assert not bad, f"{len(bad)} page(s) look garbled, e.g. {bad[0]}"

if __name__ == "__main__":
    rows = quality(Path("in/annual-report.pdf"))
    assert_quality(rows)
    print(f"{len(rows)} page(s) passed text quality checks")

The common-word ratio is the most useful single signal: garbled text from broken font maps still consists of letters, so alpha_ratio looks healthy, but none of those letters form English function words. Tune the word list to your documents' language.

For a second opinion, extract the same pages with pdfplumber and compare normalised word sets; a Jaccard similarity below about 0.9 between two libraries on a digital page is worth investigating.

Performance and Scale Notes

PyMuPDF extracts plain text at hundreds of pages per second on one core; the dict and rawdict modes are several times slower because they build span and character objects. Use the cheapest mode that answers the question. Open, extract and close each document inside a worker process — ProcessPoolExecutor with one task per file — and never pass Document objects between processes. Memory scales with the largest page, not the file, so long documents are safe if you do not accumulate all page strings; write each page to the output as you go for multi-thousand-page files. The header/footer detector needs a full pass before extraction, so cache its result per file rather than recomputing it per page.

Troubleshooting

Complete Working Script

#!/usr/bin/env python3
# pip install pymupdf
"""Extract reading-order text, metadata and outline from every PDF in a folder to JSONL."""
import argparse
import json
import logging
import re
import sys
from collections import Counter
from pathlib import Path

import pymupdf

FLAGS = pymupdf.TEXT_DEHYPHENATE | pymupdf.TEXT_MEDIABOX_CLIP
log = logging.getLogger("pdftext")


def running_lines(doc, band=0.08, share=0.6):
    seen = Counter()
    for page in doc:
        h = page.rect.height
        seen.update({re.sub(r"\d+", "#", b[4].strip())
                     for b in page.get_text("blocks") if b[3] < h * band or b[1] > h * (1 - band)})
    need = max(2, int(doc.page_count * share))
    return {k for k, v in seen.items() if v >= need}


def page_text(page, running):
    blocks = [b for b in page.get_text("blocks", flags=FLAGS, sort=True) if b[6] == 0]
    return "\n".join(b[4].strip() for b in blocks
                     if re.sub(r"\d+", "#", b[4].strip()) not in running)


def extract(pdf_path: Path) -> dict:
    with pymupdf.open(pdf_path) as doc:
        if doc.needs_pass:
            raise PermissionError("encrypted")
        running = running_lines(doc) if doc.page_count > 2 else set()
        pages = [page_text(p, running) for p in doc]
        meta = doc.metadata or {}
        toc = [{"level": lvl, "title": t.strip(), "page": pg} for lvl, t, pg in doc.get_toc()]
    chars = sum(len(p) for p in pages)
    return {
        "file": pdf_path.name,
        "title": (meta.get("title") or "").strip() or pdf_path.stem,
        "author": (meta.get("author") or "").strip(),
        "created_raw": meta.get("creationDate") or "",
        "toc": toc,
        "chars": chars,
        "likely_scanned": chars < 50 * max(1, len(pages)),
        "pages": pages,
    }


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("src", type=Path)
    ap.add_argument("out", type=Path, help="output .jsonl file")
    args = ap.parse_args()
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")

    args.out.parent.mkdir(parents=True, exist_ok=True)
    failed = 0
    with args.out.open("w", encoding="utf-8") as fh:
        for pdf in sorted(args.src.rglob("*.pdf")):
            try:
                record = extract(pdf)
            except Exception as exc:
                failed += 1
                log.error("%s: %s", pdf, exc)
                continue
            if record["likely_scanned"]:
                log.warning("%s: little text, probably needs OCR", pdf.name)
            fh.write(json.dumps(record, ensure_ascii=False) + "\n")
            log.info("%s: %d pages, %d chars", pdf.name, len(record["pages"]), record["chars"])
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Frequently Asked Questions

Which library gives the best reading order? For ordinary documents PyMuPDF with sort=True and pdfplumber are close; pypdf's layout mode is best at preserving visual alignment. None of them understands complex magazine layouts — that needs block-level logic like Step 2 or a layout model.

Why does the same file extract differently in two libraries? Each reconstructs words and lines from glyph positions with different gap thresholds and ordering rules. Differences in spacing and line breaks are normal; differences in the characters themselves point at font-map problems.

Can I get font size and bold information for headings? Yes. page.get_text("dict") returns spans with size, font and flags; bit 16 of flags marks bold. Classifying spans larger than the body size as headings is a reliable way to chunk text by section when the outline is missing.

How do I extract text from only part of a page? Pass clip=pymupdf.Rect(...) to get_text. The coordinate techniques in extract a PDF table by bounding box coordinates apply equally to text.

Related

• Extract PDF Metadata and Bookmarks with Python — Info vs XMP, dates and outline trees
• Fix PDF Text Extraction Missing Spaces — words glued together or letter-spaced apart
• Fix CID Garbled Characters in PDF Text — fonts without Unicode maps
• Comparing PDF Table Extraction Libraries — when the text you need is tabular
• Redacting Sensitive Data in PDFs with Python — the same text model, used to remove values

Part of Automating PDF Extraction & Generation.