Python Doc & Data Automation

Redact PDF Text by Regex Pattern

The symptom: a regex finds every account number when you run it over page.get_text(), yet the redacted PDF still shows some of them — usually the ones that wrap onto a second line, contain a non-breaking space, or were split into separately positioned glyph runs by the generator. page.search_for() only locates literal strings, so the obvious bridge — find matches with re, then search for each matched string — silently loses anything the text extractor and the search engine see differently.

Root Cause

PyMuPDF offers two views of the same page. get_text("text") returns a reading-order string with line breaks inserted wherever the layout wraps. search_for(needle) scans the page's character positions for the needle and returns rectangles, joining characters across spans on the same line but not across lines. A regex that matches GB29 NWBK 6016 1331 9268 19 in a whitespace-normalised string produces a needle that never exists on any single line of the page. Separately, characters such as U+00A0 (non-breaking space) or U+2011 (non-breaking hyphen) match \s or - in your normalised text but differ from the space you pass to search_for. The regex was right; the mapping from match back to page geometry was lossy.

Minimal Diagnostic

Compare what the regex matches in normalised text against what search_for can locate. Every match with zero rectangles is a value that would survive redaction.

# pip install pymupdf
import re
from pathlib import Path
import pymupdf

SOURCE = Path("in/statement.pdf")
IBAN = re.compile(r"\b[A-Z]{2}\d{2}(?:\s?[A-Z0-9]{4}){3,7}(?:\s?[A-Z0-9]{1,3})?\b")

def unlocatable_matches(pdf_path: Path, rx: re.Pattern) -> list[tuple[int, str]]:
    missing = []
    try:
        doc = pymupdf.open(pdf_path)
    except Exception as exc:
        raise RuntimeError(f"cannot open {pdf_path}: {exc}") from exc
    with doc:
        for page in doc:
            flat = " ".join(page.get_text("text").split())   # normalise all whitespace
            for m in rx.finditer(flat):
                if not page.search_for(m.group(0)):
                    missing.append((page.number + 1, m.group(0)))
    return missing

if __name__ == "__main__":
    for page_no, value in unlocatable_matches(SOURCE, IBAN):
        print(f"page {page_no}: regex matched {value!r} but search_for found nothing")

page 1: regex matched 'GB29 NWBK 6016 1331 9268 19' but search_for found nothing
page 3: regex matched 'DE89 3704 0044 0532 0130 00' but search_for found nothing

If that list is non-empty, a match-then-search pipeline leaks those values.

Why a regex match cannot always be searched back The left panel shows the whitespace-normalised extracted text, in which the regex matches the full IBAN as one string. The right panel shows the page as search_for sees it, with the IBAN split across two lines, so searching for the full string returns no rectangles and the value survives redaction. Extracted text (regex view) Pay to: J Example IBAN GB29 NWBK 6016 1331 9268 19 Ref: INV-2231 regex match: 1 hit Page geometry (search_for view) line 4: Pay to: J Example line 5: IBAN GB29 NWBK 6016 line 6: 1331 9268 19 search_for(full value): 0 rects The fix maps regex matches onto word rectangles instead of searching for the matched string

Fix: Match Over Words, Redact Word Rectangles

Build the searchable string from get_text("words") so every character offset maps to a word with a known rectangle. A match then selects the words it overlaps, regardless of how many lines they span.

# pip install pymupdf
import re
from pathlib import Path
import pymupdf

SOURCE = Path("in/statement.pdf")
DEST = Path("out/statement-redacted.pdf")

PATTERNS = {
    "iban": re.compile(r"\b[A-Z]{2}\d{2}(?: [A-Z0-9]{4}){3,7}(?: [A-Z0-9]{1,3})?\b"),
    "email": re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
}

NBSP = {"\u00a0": " ", "\u2011": "-", "\u2013": "-"}      # normalise look-alike characters

def word_index(page: pymupdf.Page):
    """Join words with single spaces and remember each word's span and rectangle."""
    words = page.get_text("words", sort=True)               # sort=True gives reading order
    text, spans = "", []
    for x0, y0, x1, y1, word, *_ in words:
        clean = "".join(NBSP.get(ch, ch) for ch in word)     # changed: map look-alikes per char
        start = len(text)
        text += clean + " "                                  # changed: exactly one space between words
        spans.append((start, start + len(clean), pymupdf.Rect(x0, y0, x1, y1)))
    return text, spans

def regex_rects(page: pymupdf.Page) -> list[tuple[str, pymupdf.Rect]]:
    text, spans = word_index(page)
    hits = []
    for label, rx in PATTERNS.items():
        for m in rx.finditer(text):
            for start, end, rect in spans:
                if start < m.end() and end > m.start():      # changed: overlap, not equality
                    hits.append((label, rect))
    return hits

def redact(src: Path, dest: Path) -> int:
    count = 0
    with pymupdf.open(src) as doc:
        for page in doc:
            hits = regex_rects(page)
            for label, rect in hits:
                page.add_redact_annot(rect, text=f"[{label}]", fontsize=6, fill=(0, 0, 0))
            if hits:
                page.apply_redactions(images=pymupdf.PDF_REDACT_IMAGE_PIXELS)  # once per page
            count += len(hits)
        dest.parent.mkdir(parents=True, exist_ok=True)
        doc.save(dest, garbage=4, deflate=True, clean=True)  # drop orphaned original text
    return count

if __name__ == "__main__":
    try:
        print(f"{redact(SOURCE, DEST)} word box(es) redacted")
    except Exception as exc:
        raise SystemExit(f"redaction failed: {exc}")

The patterns now use a literal space between groups rather than \s?, because the word index guarantees single spaces. Values printed without spaces (GB29NWBK...) are a single word and still match the pattern with the space group made optional; add ? after the space if your documents mix both styles.

A word that is only partly inside a match — IBAN:GB29 glued together by the generator — is redacted whole. That over-redacts a label, which is the safe direction. If precision matters more, use page.get_text("rawdict") to get per-character boxes and union only the characters inside the match.

Variant Fix 1: Card Numbers Matching Everything

A pattern like (?:\d[ -]?){13,16} redacts invoice numbers, phone numbers and timestamps. Filter candidates with the Luhn checksum that real card numbers satisfy:

# pip install pymupdf
import re

CARD = re.compile(r"(?<!\d)(?:\d[ -]?){12,18}\d(?!\d)")

def luhn_ok(candidate: str) -> bool:
    digits = [int(c) for c in candidate if c.isdigit()]
    if not 13 <= len(digits) <= 19:
        return False
    checksum = 0
    for i, d in enumerate(reversed(digits)):
        if i % 2:                       # double every second digit from the right
            d = d * 2 - 9 if d > 4 else d * 2
        checksum += d
    return checksum % 10 == 0

def card_matches(text: str) -> list[re.Match]:
    return [m for m in CARD.finditer(text) if luhn_ok(m.group(0))]

Replace rx.finditer(text) with card_matches(text) for the card label in regex_rects. Roughly one random digit string in ten passes Luhn, so this cuts false positives by about ninety percent without ever skipping a real card number.

Effect of the Luhn filter on card-number candidates In a sample of 400 statement pages the loose digit pattern produced 1,240 candidates. After the Luhn checksum filter 131 remained. 118 of those were genuine card numbers, so the filter removed about ninety percent of false positives while keeping every real card number. 400 statement pages, loose 13 to 19 digit pattern Digit-run candidates 1240 Pass Luhn checksum 131 Genuine card numbers 118 Luhn never rejects a valid card number, so the filter only removes false positives

Variant Fix 2: Case and Accent Differences in Names

Names typed in a CRM (Zoë Ó Briain) rarely match the PDF byte-for-byte: the generator may have decomposed accents, used a different apostrophe, or upper-cased the header. Normalise both sides before comparing, while keeping offsets aligned to the original words:

# pip install pymupdf
import re
import unicodedata

def fold(s: str) -> str:
    """Casefold and strip combining accents without changing string length per character."""
    return "".join(
        unicodedata.normalize("NFKD", ch)[0].casefold() if ch.strip() else " "
        for ch in s
    )

def name_pattern(full_name: str) -> re.Pattern:
    parts = [re.escape(fold(p)) for p in full_name.split()]
    return re.compile(r"\b" + r"\W{0,2}".join(parts) + r"\b")

Apply fold to each word inside word_index and match with name_pattern. Taking only the first code point of each NFKD decomposition keeps one output character per input character, so the offsets still map to the same words. \W{0,2} between name parts tolerates O'Briain, O’Briain and O Briain.

Variant Fix 3: Letter-Spaced Values Become Separate Words

Some generators — payroll systems and form-filling tools in particular — position every character of a field individually, often inside printed boxes. The extractor sees 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 as sixteen one-character words, and a pattern expecting 4111 1111 groups never matches. Detect runs of single-character words on the same line and match against a collapsed copy, while keeping a map from each collapsed character back to its word:

# pip install pymupdf
import re
import pymupdf

def collapsed_index(page: pymupdf.Page):
    """Collapse runs of one-character words on a line into a single token for matching."""
    words = page.get_text("words", sort=True)
    text, spans = "", []
    prev_line, prev_single = None, False
    for x0, y0, x1, y1, word, block, line, _ in words:
        key = (block, line)
        single = len(word) == 1 and word.isalnum()
        if not (single and prev_single and key == prev_line):
            text += " "                                  # new token unless continuing a run
        start = len(text)
        text += word
        spans.append((start, len(text), pymupdf.Rect(x0, y0, x1, y1)))
        prev_line, prev_single = key, single
    return text, spans

CARD_COMPACT = re.compile(r"(?<!\d)\d{13,19}(?!\d)")

Run CARD_COMPACT (and a compact IBAN pattern without spaces) over this index in addition to the normal one, and union the rectangles. Running both indexes costs one extra pass over a list the page already produced, so the overhead is negligible; skipping it leaves exactly the fields most likely to hold personal data — boxed form entries — unredacted.

The same issue appears in reverse on the extraction side, where letter-spaced headings come out as S T A T E M E N T; fix PDF text extraction missing spaces explains how the extractor decides where one word ends, which is the setting that produces both symptoms.

Keeping Patterns Maintainable

Once a job carries more than three or four patterns, keep them in a small table that a reviewer can read without parsing Python, and load it at start-up. Each entry needs a label for the replacement text, the expression, and an optional validator name:

# pip install pyyaml
from pathlib import Path
import re
import yaml

from card_checks import luhn_ok          # the Luhn helper from Variant Fix 1

PATTERN_FILE = Path("redaction_patterns.yaml")
VALIDATORS = {"luhn": lambda s: luhn_ok(s), None: lambda s: True}

def load_patterns(path: Path) -> list[tuple[str, re.Pattern, callable]]:
    try:
        entries = yaml.safe_load(path.read_text(encoding="utf-8"))
    except (OSError, yaml.YAMLError) as exc:
        raise SystemExit(f"cannot load {path}: {exc}")
    return [(e["label"], re.compile(e["regex"]), VALIDATORS[e.get("validator")])
            for e in entries]

Version the pattern file alongside the job and log its hash with every run. When someone asks months later whether a given document was screened for national insurance numbers, the log answers it.

From regex match to redaction rectangles Five steps. get_text words returns each word with its rectangle. The words are joined with single spaces while recording character spans. The regex runs over that joined string. Each match selects every word whose span overlaps it, even across line breaks. Those rectangles become redaction annotations that are applied once per page and saved with garbage collection. words get_text words Index char span per word Match regex over joined text Overlap words touching match Apply redact and save Overlap selection is what lets a match span two lines of the page

Verification

Verification reuses the same patterns against an independent extraction, and also checks that the word count dropped only by roughly the number of redacted words — a large unexplained drop means the redaction boxes were too big and removed surrounding text.

# pip install pdfplumber pymupdf
from pathlib import Path
import re
import pdfplumber
import pymupdf

PATTERNS = {
    "iban": re.compile(r"\b[A-Z]{2}\d{2}(?: ?[A-Z0-9]{4}){3,7}(?: ?[A-Z0-9]{1,3})?\b"),
    "email": re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
}

def word_count(pdf: Path) -> int:
    with pymupdf.open(pdf) as doc:
        return sum(len(page.get_text("words")) for page in doc)

def verify(original: Path, redacted: Path, expected_removed: int) -> None:
    with pdfplumber.open(redacted) as pdf:
        flat = " ".join(" ".join((p.extract_text() or "") for p in pdf.pages).split())
    leaks = {k: rx.findall(flat) for k, rx in PATTERNS.items() if rx.search(flat)}
    assert not leaks, f"values still extractable: {leaks}"

    before, after = word_count(original), word_count(redacted)
    removed = before - after
    # labels like [iban] add one word per redaction box, so allow a small margin
    assert removed <= expected_removed + 5, (
        f"{removed} words disappeared but only {expected_removed} were targeted"
    )
    print(f"ok: {removed} word(s) removed, no pattern matches remain")

if __name__ == "__main__":
    verify(Path("in/statement.pdf"), Path("out/statement-redacted.pdf"), expected_removed=24)

Run the diagnostic from the top of the page against the redacted output as well: it should now report nothing, because there is nothing left for the regex to match.

FAQ

Why not use page.search_for with a regex directly? It does not accept patterns — it takes a literal needle. The word-index approach is the supported way to combine regular expressions with page geometry.

Does sort=True matter for get_text("words")? Yes. Without it, words come in content-stream order, which for multi-column layouts can interleave columns and split a value that reads contiguously on the page. Sorting by position restores reading order for ordinary layouts.

How do I handle values split by hyphenation at a line end? Add pymupdf.TEXT_DEHYPHENATE to the flags when extracting: page.get_text("words", flags=pymupdf.TEXT_DEHYPHENATE, sort=True). The joined word keeps the rectangle of its first half, so also redact the next word when a match ends at a dehyphenated word.

Can I preview matches before removing anything? Add the redaction annotations, save to a review copy without calling apply_redactions, and open it in a viewer — the marked areas show as outlined boxes. Apply only on the copy that gets distributed.

Part of Redacting Sensitive Data in PDFs with Python.

/html>