Python Doc & Data Automation

Fix Redacted Text Still Searchable in PDF

The redacted PDF looks right — solid black bars over every account number — but pressing Ctrl+F in the viewer highlights text under the bars, selecting the area and pasting reveals the digits, or pdftotext out.pdf - prints them in plain text. The redaction covered the value visually and removed nothing.

$ pdftotext out/statement-redacted.pdf - | grep -c "GB29 NWBK"
2

Any non-zero count is a data leak, however the page looks.

Root Cause

A PDF page is a sequence of drawing operations: set font, show glyphs at a position, fill a path, paint an image. Viewers render them in order and text extraction reads the glyph operations directly, ignoring what is painted on top. Four distinct mistakes leave the glyph operations in place. The code drew a filled rectangle (page.draw_rect, a ReportLab overlay merged with pypdf, or a stamp) — paint on top, content untouched. Or it added redaction annotations but never called apply_redactions(), so the file carries proposed redactions. Or it applied them and then saved incrementally, which appends the new page content while the old content stream remains in the file as an unreferenced object. Or the page was a scan with an invisible OCR text layer, and the redaction blanked the image pixels while the OCR text — rendered with invisible text mode — still sat above them.

Minimal Diagnostic

Classify which of the four failures you have. The script checks for unapplied redaction annotations, extractable matches under filled areas, invisible text, and raw bytes of the old content in the file.

# pip install pymupdf
import re
import zlib
from pathlib import Path
import pymupdf

TARGET = Path("out/statement-redacted.pdf")
SECRET = "GB29 NWBK"

def diagnose(pdf_path: Path, secret: str) -> None:
    try:
        doc = pymupdf.open(pdf_path)
    except Exception as exc:
        raise SystemExit(f"cannot open {pdf_path}: {exc}")
    with doc:
        pending = sum(1 for p in doc for a in p.annots() if a.type[0] == pymupdf.PDF_ANNOT_REDACT)
        print(f"unapplied redaction annotations: {pending}")
        for page in doc:
            hits = page.search_for(secret)
            if not hits:
                continue
            # A full-page image with extractable text on top is almost always a scan + OCR layer
            page_area = page.rect.width * page.rect.height
            image_area = sum(
                r.width * r.height
                for img in page.get_images(full=True)
                for r in page.get_image_rects(img[0])
            )
            ocr_layer = image_area > 0.8 * page_area
            print(f"page {page.number + 1}: {len(hits)} extractable hit(s)"
                  + (" over a full-page image (likely OCR layer)" if ocr_layer else ""))
        print(f"xref count: {doc.xref_length()}  (compare with a garbage-collected save)")

    raw = pdf_path.read_bytes()
    streams = re.findall(rb"stream\r?\n(.*?)\r?\nendstream", raw, re.S)
    leaked = 0
    for body in streams:
        try:
            data = zlib.decompress(body)
        except zlib.error:
            data = body
        leaked += secret.encode("latin-1") in data
    print(f"streams containing the secret as plain bytes: {leaked}")

if __name__ == "__main__":
    diagnose(TARGET, SECRET)

Read the output this way. Unapplied annotations greater than zero: the apply step never ran. Extractable hits with nothing invisible: an overlay was drawn. Hits over a full-page image: the OCR layer of a scan. No extractable hits but streams still contain the bytes: an incremental save kept the old content. Glyphs are often encoded as font-specific codes rather than ASCII, so a zero in the last line is not proof of safety — the extraction checks are the authoritative ones.

Four ways a redaction leaves text behind Four cards. Overlay only, where a rectangle is painted over glyphs and search still finds the text. Annotation never applied, where redaction annotations are present but the content stream is unchanged. Incremental save, where the page looks clean but the old content stream remains as an orphaned object in the file bytes. Hidden OCR layer, where scan pixels were blanked but invisible text above them is still searchable. Overlay only draw_rect or a merged stamp paints on top. Signal: search hits under the bar. Annotation never applied add_redact_annot without apply_redactions. Signal: redact annots remain. Incremental save Old content stream kept as an orphan object. Signal: bytes still in the file. Hidden OCR layer Pixels blanked, invisible text left above them. Signal: invisible hits.

Fix: Remove Content, Then Save a Clean File

Replace the overlay or incomplete redaction with the full sequence. Every changed line carries a comment.

# pip install pymupdf
from pathlib import Path
import pymupdf

SOURCE = Path("in/statement.pdf")          # changed: always start from the ORIGINAL, not the bad output
DEST = Path("out/statement-redacted.pdf")
SECRETS = ["GB29 NWBK 6016 1331 9268 19", "Jane Q. Example"]

def redact(src: Path, dest: Path, secrets: list[str]) -> None:
    if src.resolve() == dest.resolve():
        raise ValueError("write to a new file; saving over the source invites incremental saves")
    with pymupdf.open(src) as doc:
        for page in doc:
            found = False
            for secret in secrets:
                for quad in page.search_for(secret, quads=True):
                    page.add_redact_annot(quad, fill=(0, 0, 0))   # changed: redaction annot, not draw_rect
                    found = True
            if found:
                page.apply_redactions(                            # changed: actually remove the glyphs
                    images=pymupdf.PDF_REDACT_IMAGE_PIXELS,       # changed: blank pixels, keep the scan
                )
        doc.scrub(hidden_text=True, metadata=True, xml_metadata=True)  # changed: drop invisible OCR text
        dest.parent.mkdir(parents=True, exist_ok=True)
        doc.save(dest, garbage=4, deflate=True, clean=True)       # changed: full save, orphans removed

if __name__ == "__main__":
    try:
        redact(SOURCE, DEST, SECRETS)
    except Exception as exc:
        raise SystemExit(f"redaction failed: {exc}")

Starting from the original matters. If you re-open the leaky output and redact again, an overlay rectangle from the previous attempt is now part of the page content; apply_redactions will remove the text under it correctly, but a failed earlier run may have written an incremental update on top of an already incremental file. Starting clean removes the question.

Why an Incremental Save Keeps the Old Text

The incremental-save case deserves its own explanation because it defeats careful code. PDF was designed so that editors could append changes to the end of a file without rewriting it — useful for signatures, which must not disturb signed bytes, and for speed on large files. An incremental update appends new versions of changed objects plus a new cross-reference table that points at them. Viewers follow the newest table and render the redacted page. The previous page content stream is still physically present earlier in the file; nothing references it any more, but nothing removed it either.

You can see the effect by comparing the two save modes on the same redacted document:

# pip install pymupdf
from pathlib import Path
import pymupdf

SOURCE = Path("in/statement.pdf")
SECRET = "GB29 NWBK 6016 1331 9268 19"

def compare_save_modes(src: Path) -> None:
    work = Path("out/work.pdf")
    work.parent.mkdir(parents=True, exist_ok=True)
    work.write_bytes(src.read_bytes())
    with pymupdf.open(work) as doc:
        page = doc[0]
        for quad in page.search_for(SECRET, quads=True):
            page.add_redact_annot(quad)
        page.apply_redactions()
        doc.saveIncr()                         # appends; old objects stay in the file
    raw = work.read_bytes()
    print("incremental:", len(raw), "bytes,", raw.count(b"endstream"), "stream objects in the file")
    with pymupdf.open(work) as doc:
        doc.save("out/full.pdf", garbage=4, deflate=True)
    full = Path("out/full.pdf").read_bytes()
    print("full save:  ", len(full), "bytes,", full.count(b"endstream"), "stream objects in the file")

if __name__ == "__main__":
    compare_save_modes(SOURCE)

The incremental file is larger than the original even though content was removed, and it carries more stream objects, because it contains both page versions — the diagnostic's decompression loop will find the old one. The garbage-collected file is smaller than either. File size moving the wrong way after redaction is a cheap, useful alarm to put in a batch job's log.

What an incremental save leaves in the file Stacked regions of the file from start to end. The original header and body contain the page content stream with the account number. The original cross-reference table follows. The appended update contains the new redacted content stream and a new cross-reference table that viewers follow. Viewers render the update, but the original stream is still in the file bytes until a full save with garbage collection rewrites the file. Header and original body content stream WITH the IBAN still readable with any stream parser Original xref table still points at old objects ignored by viewers but not removed Appended update redacted content stream rendered on screen New xref and trailer what viewers follow full save with garbage=4 keeps only this chain

Variant Fix 1: The OCR Layer Survives Pixel Redaction

A scanned statement that was made searchable (by a scanner, Acrobat or OCRmyPDF) carries a text layer drawn in render mode 3 — invisible, but extractable. search_for finds the words in that layer, so a search-driven redaction removes both the invisible glyphs and the pixels beneath, if the OCR text lines up with the image. When it does not — OCR run at a different resolution, or on a deskewed copy — the boxes land beside the visible value. Two options:

# pip install pymupdf
import pymupdf

def strip_invisible_text(doc: pymupdf.Document) -> None:
    """Drop the whole OCR layer; re-run OCR on the redacted output if searchability is needed."""
    doc.scrub(hidden_text=True)

def redact_where_ocr_says(page: pymupdf.Page, secret: str, pad: float = 2.0) -> int:
    """Use the OCR layer only to locate values, padding boxes to absorb misalignment."""
    quads = page.search_for(secret, quads=True)
    for quad in quads:
        rect = quad.rect + (-pad, -pad, pad, pad)     # widen by a couple of points each side
        page.add_redact_annot(rect, fill=(0, 0, 0))
    if quads:
        page.apply_redactions(images=pymupdf.PDF_REDACT_IMAGE_PIXELS)
    return len(quads)

Removing the layer and re-running OCR after redaction is the safer default: the new layer is generated from the blanked image, so it cannot contain the removed value.

Variant Fix 2: The Redaction Was Done in Another Tool

Files arrive "already redacted" by a person using a PDF editor's rectangle tool. Do not trust them. Detect filled black rectangles that cover extractable text and fail the intake:

# pip install pymupdf
from pathlib import Path
import pymupdf

def overlay_leaks(pdf_path: Path) -> list[tuple[int, str]]:
    leaks = []
    with pymupdf.open(pdf_path) as doc:
        for page in doc:
            dark = [
                d["rect"] for d in page.get_drawings()
                if d.get("fill") and max(d["fill"]) < 0.15 and d["rect"].width > 20
            ]
            for x0, y0, x1, y1, word, *_ in page.get_text("words"):
                word_rect = pymupdf.Rect(x0, y0, x1, y1)
                if any(r.contains(word_rect) for r in dark):
                    leaks.append((page.number + 1, word))
    return leaks

Any result means someone drew bars instead of redacting. Route such files back to the sender or run them through the fix above using the bar rectangles as redaction areas.

Overlay versus real redaction in code The left panel shows the leaking version, which draws a filled rectangle over the value and saves incrementally into the same file. The right panel shows the fixed version, which adds a redaction annotation, applies it with pixel handling, scrubs hidden text and saves a new file with garbage collection level four. Leaks the value r = page.search_for(s)[0] page.draw_rect(r, fill=(0,0,0)) doc.saveIncr() text under bar: extractable Removes the value q = page.search_for(s, quads=True) page.add_redact_annot(q[0]) page.apply_redactions() doc.scrub(hidden_text=True) doc.save(new, garbage=4)

Verification

Check the output three independent ways: a different extractor, the command-line tool a recipient might use, and the raw object count.

# pip install pdfplumber pymupdf
import subprocess
from pathlib import Path
import pdfplumber
import pymupdf

def verify(pdf_path: Path, secrets: list[str]) -> None:
    with pdfplumber.open(pdf_path) as pdf:
        text = " ".join(" ".join((p.extract_text() or "") for p in pdf.pages).split())
    assert not [s for s in secrets if s in text], "pdfplumber still extracts a secret"

    try:
        out = subprocess.run(["pdftotext", str(pdf_path), "-"], capture_output=True,
                             text=True, check=True).stdout
        flat = " ".join(out.split())
        assert not [s for s in secrets if s in flat], "pdftotext still extracts a secret"
    except FileNotFoundError:
        print("pdftotext not installed; skipped poppler check")

    with pymupdf.open(pdf_path) as doc:
        assert not any(a.type[0] == pymupdf.PDF_ANNOT_REDACT for p in doc for a in p.annots()), \
            "unapplied redaction annotations remain"
        for xref in range(1, doc.xref_length()):
            if doc.xref_is_stream(xref):
                data = doc.xref_stream(xref) or b""
                assert not [s for s in secrets if s.encode("latin-1", "ignore") in data], \
                    f"secret bytes in stream xref {xref}"
    print(f"{pdf_path.name}: verified clean")

if __name__ == "__main__":
    verify(Path("out/statement-redacted.pdf"), ["GB29 NWBK 6016 1331 9268 19", "Jane Q. Example"])

xref_stream returns decompressed stream content, so this check sees inside compressed objects that the grep in the diagnostic could not. After a garbage-collected save, orphaned streams are gone and the loop only visits live objects; before the fix, it would have found the original content stream still sitting in the file.

A useful regression test is to keep one deliberately bad file — produced with draw_rect and saveIncr — in the test corpus and assert that verify raises on it. A verifier that has never been seen failing has not been tested.

FAQ

Does "Print to PDF" remove the hidden text? Sometimes, and not reliably. Some print drivers rasterise, others preserve text operators. Treat it as unverified and run the verification script on its output.

Why does Acrobat's own redaction tool pass but my script's output fails? Acrobat applies and then performs a full save by default. The usual script difference is saveIncr() or save(..., incremental=True); switch to a full save with garbage=4.

Will garbage=4 change how the document looks? No. It removes unreferenced objects and merges identical ones. Rendering is unaffected; the file usually gets smaller.

Is flattening the PDF to images a valid alternative? It removes all text, so it is safe if the output never needs to be searchable or accessible. It also inflates file size and destroys text quality for screen readers, which is why content-level redaction is preferred.

Part of Redacting Sensitive Data in PDFs with Python.

/html>