Redacting Sensitive Data in PDFs with Python

Redaction fails quietly. A black rectangle drawn over a customer's account number looks finished on screen, prints finished, and passes a visual review — while the digits remain in the content stream, one copy-paste or pdftotext call away. The failure is not in the drawing tool; it is in treating redaction as a presentation problem when it is a content removal problem. A PDF page is a program that paints glyphs, vector paths and images in order, and an extra paint operation on top removes nothing underneath.

Generic "find and replace" advice does not transfer either. PDF text is positioned glyph runs, not an editable string, so there is nothing to replace in place. Sensitive values also hide outside the visible page: in document metadata, XMP packets, form field values, annotations, bookmarks, embedded files and, for scanned pages, inside image pixels where no text search will ever find them. This guide builds a redaction workflow with PyMuPDF that actually deletes the content, cleans the places text leaks into, and proves the result with an independent re-extraction.

Prerequisites

python -m venv .venv && source .venv/bin/activate
pip install pymupdf pdfplumber
# only for scanned pages (see "Scanned pages" below)
sudo apt-get install -y tesseract-ocr
pip install pytesseract pillow
mkdir -p in out

PyMuPDF does the searching, annotation and removal. pdfplumber is used only in the verification step, deliberately: a second, independent parser catches cases where one library's text model disagrees with another's. Test against a real document — a statement or payslip with the same fonts your production files use — because encoding quirks are the main reason search misses a value.

Diagnostic: Classify Every Page Before Redacting

Redaction strategy depends on how the sensitive value is stored. A digital page stores text as glyphs you can search; a scanned page stores a picture of text; a mixed page (a scan with a typed stamp, or a digital page with a pasted screenshot) stores both. Classify first, because a text-only redaction on a scanned page reports zero matches and looks like success.

# pip install pymupdf
from pathlib import Path
import pymupdf

SOURCE = Path("in/statement.pdf")

def classify_pages(pdf_path: Path) -> list[dict]:
    """Return per-page text length, image count and a storage class."""
    if not pdf_path.exists():
        raise FileNotFoundError(pdf_path)
    try:
        doc = pymupdf.open(pdf_path)
    except pymupdf.FileDataError as exc:
        raise RuntimeError(f"Not a readable PDF: {pdf_path}") from exc
    report = []
    with doc:
        if doc.needs_pass:
            raise PermissionError(f"{pdf_path} is encrypted — decrypt before redacting")
        for page in doc:
            chars = len(page.get_text("text").strip())
            images = len(page.get_images(full=True))
            area = page.rect.width * page.rect.height
            covered = sum(
                r.width * r.height
                for img in page.get_images(full=True)
                for r in page.get_image_rects(img[0])
            )
            if chars < 20 and covered > 0.5 * area:
                kind = "scanned"
            elif images and covered > 0.2 * area:
                kind = "mixed"
            else:
                kind = "digital"
            report.append({"page": page.number + 1, "chars": chars,
                           "images": images, "kind": kind})
    return report

if __name__ == "__main__":
    for row in classify_pages(SOURCE):
        print(row)

The thresholds are pragmatic, not magic: fewer than twenty extractable characters with an image covering most of the page is a scan; a digital page carrying a large image is mixed. Print the report for a sample of production files and adjust the cut-offs to what your scanners and generators actually produce.

Core Implementation

Redaction in PyMuPDF is deliberately two-phase. You first mark areas with redaction annotations, which are reviewable and still reversible, and then call apply_redactions(), which rewrites the page content: glyphs intersecting a marked area are removed from the content stream, image pixels under it are blanked, and the annotation is replaced by a filled rectangle. Only the second phase removes anything.

Step 1: Define What Counts as Sensitive

Keep patterns in one place, named, so a reviewer can see exactly what the job looks for and so a verification step can reuse the same list.

# pip install pymupdf
import re

PATTERNS: dict[str, re.Pattern] = {
    "iban": re.compile(r"\b[A-Z]{2}\d{2}(?: ?[A-Z0-9]{4}){3,7}(?: ?[A-Z0-9]{1,3})?\b"),
    "card": re.compile(r"\b(?:\d[ -]?){13,16}\b"),
    "email": re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
    "uk_nino": re.compile(r"\b[A-CEGHJ-PR-TW-Z]{2} ?\d{2} ?\d{2} ?\d{2} ?[A-D]\b"),
    "phone": re.compile(r"(?<!\d)(?:\+44 ?|0)7\d{3} ?\d{6}(?!\d)"),
}

# Exact strings known in advance, e.g. the customer's name from the CRM record
LITERALS: list[str] = ["Jane Q. Example", "ACC-004417"]

Patterns over-match on purpose. A redaction job that removes a harmless sixteen-digit reference number costs a support question; one that misses a card number costs a breach notification. The tighter per-pattern guide, redact PDF text by regex pattern, covers validation such as Luhn checks when false positives become a real cost.

Step 2: Find Match Rectangles on Each Page

page.search_for() returns rectangles for a literal string. Regular expressions run over the extracted page text, and each distinct match is then located with search_for, which handles words split across spans and ligatures better than hand-mapping character offsets.

# pip install pymupdf
from pathlib import Path
import pymupdf

def find_hits(page: pymupdf.Page, patterns: dict, literals: list[str]) -> list[tuple[str, pymupdf.Quad]]:
    """Return (label, quad) for every sensitive value on the page."""
    hits: list[tuple[str, pymupdf.Quad]] = []
    text = page.get_text("text")
    needles: set[tuple[str, str]] = {("literal", s) for s in literals}
    for label, rx in patterns.items():
        for m in rx.finditer(text):
            needles.add((label, m.group(0).strip()))
    for label, needle in needles:
        # quads=True keeps rotated or skewed text tightly covered
        for quad in page.search_for(needle, quads=True):
            hits.append((label, quad))
    return hits

Two behaviours matter here. search_for is case-insensitive, so a literal also removes differently cased copies — usually what you want in redaction. And a match whose extracted text contains a line break will not be found as one string; step 4 handles values wrapped across lines.

Step 3: Mark, Then Apply

# pip install pymupdf
import pymupdf

def redact_page(page: pymupdf.Page, hits, label_text: bool = False) -> int:
    """Add redaction annotations for hits and apply them. Returns hit count."""
    for label, quad in hits:
        page.add_redact_annot(
            quad,
            text=f"[{label}]" if label_text else None,   # optional replacement label
            fontsize=6,
            fill=(0, 0, 0),                               # black box after removal
        )
    if hits:
        page.apply_redactions(
            images=pymupdf.PDF_REDACT_IMAGE_PIXELS,       # blank pixels under boxes, keep the rest
            graphics=pymupdf.PDF_REDACT_LINE_ART_REMOVE_IF_TOUCHED,
        )
    return len(hits)

images=PDF_REDACT_IMAGE_PIXELS blanks only the covered pixels of an overlapping image; PDF_REDACT_IMAGE_REMOVE would delete the whole image, which destroys a scanned page. The graphics option removes vector drawings that touch a redaction, which matters when a value was drawn as outlined paths rather than text — common in PDFs exported from design tools. If your PyMuPDF build predates that parameter, drop it; text and image handling are unchanged.

Step 4: Catch Values Wrapped Across Lines

A long IBAN or address wraps, and the extracted text contains GB29 NWBK 6016\n1331 9268 19. Normalising whitespace before matching, then redacting each word that belongs to a match, covers it. The word list from get_text("words") carries a rectangle per word:

# pip install pymupdf
import re
import pymupdf

def wrapped_hits(page: pymupdf.Page, rx: re.Pattern, label: str) -> list[tuple[str, pymupdf.Rect]]:
    """Match a pattern over the page's words joined by spaces; return word rects inside matches."""
    words = page.get_text("words", sort=True)          # (x0, y0, x1, y1, word, block, line, n)
    joined, spans = "", []
    for w in words:
        start = len(joined)
        joined += w[4] + " "
        spans.append((start, start + len(w[4]), pymupdf.Rect(w[:4])))
    out = []
    for m in rx.finditer(joined):
        for start, end, rect in spans:
            if start < m.end() and end > m.start():     # word overlaps the match
                out.append((label, rect))
    return out

Joining with single spaces means patterns must allow optional spaces between groups — the IBAN pattern in step 1 already does. Word rectangles are axis-aligned, which is fine for normal text; keep search_for(..., quads=True) for rotated labels.

Step 5: Scrub Everything Outside the Page Content

# pip install pymupdf
import pymupdf

def scrub_document(doc: pymupdf.Document) -> None:
    """Remove metadata, XMP, annotations, links, embedded files, JavaScript and hidden text."""
    doc.scrub(
        attached_files=True, clean_pages=True, embedded_files=True,
        hidden_text=True, javascript=True, metadata=True,
        redactions=True, remove_links=True, reset_fields=True,
        reset_responses=True, thumbnails=True, xml_metadata=True,
    )
    doc.set_toc([])                                     # bookmark titles often carry names

hidden_text=True removes text rendered invisibly — the OCR layer of a scanned-and-searchable PDF is exactly that, and it will otherwise contain every value you just blanked out of the image. Bookmarks are not covered by scrub, hence the explicit set_toc([]); drop that line if the outline is known to be clean and readers need it.

Step 6: Save a New File with Garbage Collection

# pip install pymupdf
from pathlib import Path
import pymupdf

def save_clean(doc: pymupdf.Document, dest: Path) -> Path:
    dest.parent.mkdir(parents=True, exist_ok=True)
    doc.save(dest, garbage=4, deflate=True, clean=True)   # never incremental=True here
    return dest

garbage=4 removes unreferenced objects and merges duplicates. Without it, the original content stream can survive as an orphaned object — invisible in any viewer, fully readable by anyone who opens the file in a text editor. The failure mode is covered in detail in fix redacted text still searchable in PDF.

Edge Cases and Variants

Scanned Pages

A scan has no glyphs, so the text pass finds nothing. Run OCR to get word boxes in image pixel coordinates, convert them to page points, and add redaction annotations over those rectangles; apply_redactions(images=PDF_REDACT_IMAGE_PIXELS) then blanks the pixels.

# pip install pymupdf pytesseract pillow
import io
import re
import pymupdf
import pytesseract
from PIL import Image

def ocr_redact_page(page: pymupdf.Page, rx: re.Pattern, dpi: int = 300) -> int:
    pix = page.get_pixmap(dpi=dpi)
    image = Image.open(io.BytesIO(pix.tobytes("png")))
    data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
    scale = 72 / dpi                                  # image pixels -> PDF points
    count = 0
    for i, word in enumerate(data["text"]):
        if word.strip() and rx.search(word):
            x, y, w, h = (data[k][i] * scale for k in ("left", "top", "width", "height"))
            page.add_redact_annot(pymupdf.Rect(x, y, x + w, y + h), fill=(0, 0, 0))
            count += 1
    if count:
        page.apply_redactions(images=pymupdf.PDF_REDACT_IMAGE_PIXELS)
    return count

The scale factor assumes an unrotated page whose origin is top-left, which is how PyMuPDF reports coordinates. Rotated pages need page.derotation_matrix applied to the rectangle. OCR per word misses values split across words, so the full procedure — line grouping, padding and confidence thresholds — lives in redact scanned PDF regions with OCR. Accuracy matters more here than anywhere else in OCR work; the preprocessing in improve OCR accuracy with image preprocessing directly reduces misses.

Form Fields

Filled AcroForm values live in widget annotations, not in the page content. scrub(reset_fields=True) clears them, but if the form must stay filled apart from one field, redact the value explicitly:

# pip install pymupdf
import pymupdf

def blank_field(doc: pymupdf.Document, field_name: str) -> int:
    changed = 0
    for page in doc:
        for widget in page.widgets():
            if widget.field_name == field_name:
                widget.field_value = ""
                widget.update()
                changed += 1
    return changed

A form that was flattened before redaction has its values burned into the content stream and needs the text pass instead; see flatten PDF form fields with Python for how that flattening changes where the value lives.

Area Redaction by Template

Some documents put sensitive data in a fixed place — a signature block, a photo, the address window of a letter. Redact the region by coordinates regardless of content, which also covers handwriting that neither text search nor OCR will match:

# pip install pymupdf
import pymupdf

ADDRESS_WINDOW = pymupdf.Rect(56, 120, 300, 210)   # points, measured once from a sample

def redact_region(page: pymupdf.Page, rect: pymupdf.Rect) -> None:
    page.add_redact_annot(rect & page.rect, fill=(1, 1, 1))  # white fill for a clean look
    page.apply_redactions(images=pymupdf.PDF_REDACT_IMAGE_PIXELS)

Measure the rectangle on a real page with the coordinate techniques from extract a PDF table by bounding box coordinates; the same page geometry applies whether you are extracting or removing.

Validation

Never trust the redaction step to report its own success. Re-open the saved file with a different parser, extract all text, and assert that no pattern matches. Add checks for the leak points that text extraction does not cover.

# pip install pdfplumber pymupdf
from pathlib import Path
import pdfplumber
import pymupdf

def verify_redacted(pdf_path: Path, patterns: dict, literals: list[str]) -> None:
    """Raise AssertionError if any sensitive value is recoverable."""
    with pdfplumber.open(pdf_path) as pdf:
        text = "\n".join((p.extract_text() or "") for p in pdf.pages)
    raw = pdf_path.read_bytes()
    problems = []
    for label, rx in patterns.items():
        if rx.search(text) or rx.search(" ".join(text.split())):
            problems.append(f"pattern {label} still in extracted text")
    for lit in literals:
        if lit.lower() in text.lower():
            problems.append(f"literal {lit!r} still in extracted text")
        if lit.encode() in raw:
            problems.append(f"literal {lit!r} present in raw file bytes")
    with pymupdf.open(pdf_path) as doc:
        leftover = {k: v for k, v in doc.metadata.items() if v and k not in ("format", "encryption")}
        if leftover:
            problems.append(f"metadata not empty: {leftover}")
        if doc.get_xml_metadata():
            problems.append("XMP metadata still present")
        if any(True for page in doc for _ in page.annots()):
            problems.append("annotations remain")
    assert not problems, "; ".join(problems)
    print(f"{pdf_path.name}: no recoverable sensitive values")

The raw-bytes check is crude — compressed streams hide plain strings — but it catches uncompressed leftovers from incremental saves, which is the exact failure the garbage-collected save prevents. The format key of doc.metadata is always populated by PyMuPDF itself, which is why it is excluded. For a statistical view, count pattern matches before and after and log both numbers per file; a sudden drop to zero before redaction means extraction broke, not that the documents became clean.

Performance and Scale Notes

Redaction cost is dominated by apply_redactions and by OCR. The text pass is cheap — a 200-page statement redacts in a couple of seconds — but each call to apply_redactions rewrites the page content stream, so call it once per page after adding every annotation, never once per hit. OCR at 300 DPI takes roughly a second per page per core; process files in a ProcessPoolExecutor sized to physical cores, one document per worker, because PyMuPDF documents must not be shared across processes.

Memory stays flat if you open, redact, save and close each file inside the worker. The garbage=4 save is the slowest part for large files because it walks every object; it is also the step you cannot skip. For very large archives, write outputs to a staging directory and move them into place only after verification passes, so a half-processed run never leaves unverified files beside verified ones — the pattern from move processed files to archive folders.

Troubleshooting

Complete Working Script

#!/usr/bin/env python3
# pip install pymupdf pdfplumber
"""Redact sensitive values from every PDF in a folder and verify the result."""
import argparse
import logging
import re
import sys
from pathlib import Path

import pdfplumber
import pymupdf

PATTERNS = {
    "iban": re.compile(r"\b[A-Z]{2}\d{2}(?: ?[A-Z0-9]{4}){3,7}(?: ?[A-Z0-9]{1,3})?\b"),
    "card": re.compile(r"\b(?:\d[ -]?){13,16}\b"),
    "email": re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"),
}

log = logging.getLogger("redact")


def page_hits(page: pymupdf.Page, literals: list[str]) -> list:
    text = page.get_text("text")
    needles = {s for s in literals}
    for rx in PATTERNS.values():
        needles.update(m.group(0).strip() for m in rx.finditer(text))
    quads = []
    for needle in needles:
        quads.extend(page.search_for(needle, quads=True))
    return quads


def redact_file(src: Path, dest: Path, literals: list[str]) -> int:
    total = 0
    with pymupdf.open(src) as doc:
        if doc.needs_pass:
            raise PermissionError(f"{src} is encrypted")
        for page in doc:
            quads = page_hits(page, literals)
            for quad in quads:
                page.add_redact_annot(quad, fill=(0, 0, 0))
            if quads:
                page.apply_redactions(images=pymupdf.PDF_REDACT_IMAGE_PIXELS)
            total += len(quads)
        doc.scrub(metadata=True, xml_metadata=True, hidden_text=True,
                  embedded_files=True, attached_files=True, javascript=True,
                  remove_links=True, thumbnails=True, redactions=True)
        doc.set_toc([])
        dest.parent.mkdir(parents=True, exist_ok=True)
        doc.save(dest, garbage=4, deflate=True, clean=True)
    return total


def verify(dest: Path, literals: list[str]) -> None:
    with pdfplumber.open(dest) as pdf:
        text = " ".join(" ".join((p.extract_text() or "") for p in pdf.pages).split())
    leaks = [k for k, rx in PATTERNS.items() if rx.search(text)]
    leaks += [s for s in literals if s.lower() in text.lower()]
    if leaks:
        raise AssertionError(f"{dest.name}: still recoverable: {leaks}")


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("src_dir", type=Path)
    ap.add_argument("out_dir", type=Path)
    ap.add_argument("--literal", action="append", default=[], help="exact string to remove")
    args = ap.parse_args()
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")

    failures = 0
    for src in sorted(args.src_dir.glob("*.pdf")):
        dest = args.out_dir / src.name
        try:
            hits = redact_file(src, dest, args.literal)
            verify(dest, args.literal)
            log.info("%s: %d area(s) redacted, verified", src.name, hits)
        except Exception as exc:  # keep the batch going, report at the end
            failures += 1
            log.error("%s: %s", src.name, exc)
            dest.unlink(missing_ok=True)          # never leave an unverified output behind
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Run it as python redact.py in out --literal "Jane Q. Example". The script deletes any output that fails verification, so the output folder only ever contains files that passed — the property an auditor will ask about.

Frequently Asked Questions

Is drawing a black box with ReportLab or pypdf ever enough? No. Any overlay adds paint on top and leaves the content underneath. Use a tool that rewrites the content stream — PyMuPDF's apply_redactions — and verify with an independent parser.

Can redaction be undone after apply_redactions and a garbage-collected save? Not from the file. The glyphs and pixels are gone from the content stream and the orphaned objects are dropped. The only recovery path is an original copy, so keep originals under separate access control.

Should I replace redacted values with a label like [email]? It helps reviewers and downstream readers understand what was removed. Pass text= to add_redact_annot. Avoid labels that leak information, such as the length of the removed value.

Does redaction break accessibility tags or text flow? Removed glyphs leave gaps in the structure tree, and screen readers skip them. Adding a replacement label keeps the sentence readable. Complex tagged PDFs should be spot-checked in an accessibility checker after redaction.

How do I redact the same data from the Word original too? Redact the source, not only the export. Text replacement in .docx has its own traps, covered in find and replace text in Word documents.

Related

• Redact PDF Text by Regex Pattern — pattern design, Luhn checks and wrapped values
• Fix Redacted Text Still Searchable in PDF — the overlay and incremental-save traps
• Redact Scanned PDF Regions with OCR — pixel redaction driven by word boxes
• Watermarking and Securing PDFs — encryption and permissions for documents that keep their data
• Extracting Text and Metadata from PDFs — how PDF text is stored, and why search misses values

Part of Automating PDF Extraction & Generation.