Python Doc & Data Automation

Fix CID Garbled Characters in PDF Text

Text extraction returns (cid:39)(cid:68)(cid:87)(cid:72) instead of Date, or a string of plausible-looking but meaningless letters such as 'DWH — while the page renders perfectly in every viewer. pdfplumber and pdfminer print the (cid:NN) form; PyMuPDF and pypdf tend to print replacement characters or shifted letters for the same file.

>>> import pdfplumber
>>> pdfplumber.open("in/supplier-invoice.pdf").pages[0].extract_text()[:48]
'(cid:44)(cid:81)(cid:89)(cid:82)(cid:76)(cid:70)(cid:72)'

Root Cause

Inside a PDF, text is a sequence of character codes, and each font maps codes to glyph shapes for drawing. Rendering only needs that code-to-glyph mapping, which every embedded font has. Extraction needs a different mapping — code to Unicode — which the PDF supplies through a /ToUnicode CMap on the font or through a standard encoding the extractor recognises. Subsetting tools, some PDF printers and many report generators embed fonts with custom encodings and no ToUnicode map, often numbering codes by glyph index. The viewer draws the right shapes; the extractor has codes with no meaning, and prints them as cid numbers or guesses a standard encoding and produces shifted letters. The characters were never stored as text in a recoverable form — so the fix is to find which fonts are affected, and then either recover the mapping or read the glyphs visually.

Minimal Diagnostic

List each font on the page, whether it carries a ToUnicode map, and how much of its text extracts as garbage. The font without the map is the culprit.

# pip install pymupdf
from pathlib import Path
import pymupdf

SOURCE = Path("in/supplier-invoice.pdf")
BAD = "\N{REPLACEMENT CHARACTER}"

def font_report(pdf_path: Path, page_index: int = 0) -> None:
    try:
        doc = pymupdf.open(pdf_path)
    except (pymupdf.FileDataError, RuntimeError) as exc:
        raise SystemExit(f"cannot open {pdf_path}: {exc}")
    with doc:
        page = doc[page_index]
        spans_by_font: dict[str, list[str]] = {}
        for block in page.get_text("dict")["blocks"]:
            for line in block.get("lines", []):
                for span in line["spans"]:
                    spans_by_font.setdefault(span["font"], []).append(span["text"])
        for xref, ext, ftype, basefont, name, encoding, *_ in page.get_fonts(full=True):
            key_type, key_val = doc.xref_get_key(xref, "ToUnicode")
            short = basefont.split("+")[-1]              # drop subset prefix 'ABCDEF+'
            sample = "".join(spans_by_font.get(short, spans_by_font.get(basefont, [])))[:40]
            bad = sample.count(BAD) + sample.count("(cid:")
            print(f"{basefont:<32} type={ftype:<9} enc={encoding or '-':<16} "
                  f"ToUnicode={'yes' if key_type != 'null' else 'NO '}  sample={sample!r} bad={bad}")

if __name__ == "__main__":
    font_report(SOURCE)

BCDEEE+Calibri                   type=TrueType  enc=WinAnsiEncoding  ToUnicode=yes  sample='Invoice number 22817' bad=0
BCDFEE+ArialNarrow-Bold          type=Type0     enc=Identity-H       ToUnicode=NO   sample="'DWH" bad=0

Identity-H encoding with no ToUnicode is the classic signature: codes are glyph IDs, and nothing in the file says which glyph is which letter. A Type3 font type is a harder variant — glyphs are drawn as small pictures, and no mapping exists to recover at all.

Two mappings, only one present A character code stored in the content stream goes through the embedded font to a glyph shape, which is why the page renders correctly. Extraction needs a second path from the code through a ToUnicode CMap to a Unicode character. When the font has no ToUnicode map and uses Identity-H encoding, that path is broken and the extractor outputs cid numbers or shifted letters. The page renders because only the upper path is needed for drawing Char code 0x27 in the content stream Embedded font code to glyph ID Glyph shape drawn on screen ToUnicode CMap missing: no code to letter map Viewer shows D text looks correct Extractors see only the code, so they print (cid:39) or guess a standard encoding

Fix 1: Try an Extractor with Better Encoding Recovery

Libraries recover differently. PyMuPDF (built on MuPDF) reads more embedded CFF and TrueType cmap tables than pdfminer does, so it often recovers text that pdfplumber prints as cid codes. Try it before anything heavier:

# pip install pymupdf pdfplumber
from pathlib import Path
import pdfplumber
import pymupdf

SOURCE = Path("in/supplier-invoice.pdf")
BAD = "\N{REPLACEMENT CHARACTER}"

def garbage_share(text: str) -> float:
    if not text:
        return 1.0
    return (text.count("(cid:") * 6 + text.count(BAD)) / len(text)

def best_extraction(pdf_path: Path, page_index: int) -> tuple[str, str]:
    candidates = {}
    try:
        with pdfplumber.open(pdf_path) as pdf:
            candidates["pdfplumber"] = pdf.pages[page_index].extract_text() or ""
    except Exception as exc:
        print(f"pdfplumber failed: {exc}")
    try:
        with pymupdf.open(pdf_path) as doc:
            candidates["pymupdf"] = doc[page_index].get_text(sort=True)      # changed: second engine
    except Exception as exc:
        print(f"pymupdf failed: {exc}")
    if not candidates:
        raise RuntimeError("no extractor could open the file")
    name = min(candidates, key=lambda k: garbage_share(candidates[k]))      # changed: pick the cleanest
    return name, candidates[name]

if __name__ == "__main__":
    engine, text = best_extraction(SOURCE, 0)
    print(engine, repr(text[:80]))

Garbage share only measures cid tokens and replacement characters. Shifted-letter output ('DWH for Date) looks clean to it, so combine it with the common-word ratio from Extracting Text and Metadata from PDFs before trusting the winner.

Fix 2: Remap a Consistent Code Offset

Many subset fonts number glyphs in alphabetical order with a constant offset from ASCII, so every letter is shifted by the same amount: 'DWH is Date shifted down by 29. When one font on one generator produces this pattern, a translation table fixes it exactly. Derive the offset from a known word rather than guessing:

# pip install pymupdf
from pathlib import Path
import pymupdf

KNOWN_WORD = "Invoice"            # a word you know appears in the broken font on this page

def find_offset(garbled: str, known: str, max_shift: int = 64) -> int | None:
    for shift in range(-max_shift, max_shift + 1):
        try:
            candidate = "".join(chr(ord(c) + shift) for c in garbled)
        except ValueError:
            continue
        if known in candidate:
            return shift
    return None

def fix_font_spans(page: pymupdf.Page, font_name: str, shift: int) -> str:
    out = []
    for block in page.get_text("dict", sort=True)["blocks"]:
        for line in block.get("lines", []):
            parts = []
            for span in line["spans"]:
                text = span["text"]
                if span["font"].endswith(font_name):
                    text = "".join(chr(ord(c) + shift) if 32 <= ord(c) + shift < 0x250 else c
                                   for c in text)          # changed: shift only the broken font
                parts.append(text)
            out.append("".join(parts))
    return "\n".join(out)

if __name__ == "__main__":
    with pymupdf.open(Path("in/supplier-invoice.pdf")) as doc:
        page = doc[0]
        broken = "".join(s["text"] for b in page.get_text("dict")["blocks"]
                         for l in b.get("lines", []) for s in l["spans"]
                         if s["font"].endswith("ArialNarrow-Bold"))
        shift = find_offset(broken, KNOWN_WORD)
        if shift is None:
            raise SystemExit("no constant offset — use OCR (Fix 3)")
        print(f"offset {shift}")
        print(fix_font_spans(page, "ArialNarrow-Bold", shift))

Apply a remap only per font and only after confirming the offset on a known word — other fonts on the same page are usually fine, and shifting them corrupts good text. The offset is a property of how one generator subsets one font; record it keyed by the producer string from the metadata and the font's base name, and re-verify when either changes. Offsets are not universal: fonts subsetted in glyph order rather than alphabet order produce scrambles no single shift can fix.

A constant glyph offset, before and after remapping The left panel shows extracted text in which every letter is shifted, such as apostrophe D W H for Date and a string of symbols for Invoice total. The right panel shows the same spans after adding an offset of 29 to each character code, producing Date, Invoice total and Due, while text in other fonts on the page is left unchanged. Extracted (offset -29) 'DWH 14 Sep 2026 ,QYRLFH WRWDO 1,240.50 'XH 30 days numbers: other font, fine After remap (+29) Date 14 Sep 2026 Invoice total 1,240.50 Due 30 days numbers untouched

Fix 3: OCR Only the Broken Pages

When the mapping cannot be recovered — scrambled glyph order, Type3 fonts, outlines — read the glyphs visually. OCR is slower and introduces its own errors, so restrict it to pages whose extracted text fails quality checks, and keep native extraction for the rest.

# pip install pymupdf pytesseract pillow
import io
from pathlib import Path
import pymupdf
import pytesseract
from PIL import Image

BAD = "\N{REPLACEMENT CHARACTER}"
COMMON = {"the", "and", "of", "to", "for", "date", "total", "invoice", "number", "due"}

def looks_broken(text: str) -> bool:
    words = [w.lower().strip(".,:;") for w in text.split()]
    common = sum(w in COMMON for w in words) / (len(words) or 1)
    return "(cid:" in text or BAD in text or (len(words) > 30 and common < 0.02)

def page_text_with_fallback(pdf_path: Path, dpi: int = 300) -> list[tuple[str, str]]:
    results = []
    with pymupdf.open(pdf_path) as doc:
        for page in doc:
            native = page.get_text(sort=True)
            if not looks_broken(native):
                results.append(("native", native))
                continue
            pix = page.get_pixmap(dpi=dpi)                     # changed: render broken page only
            image = Image.open(io.BytesIO(pix.tobytes("png")))
            try:
                results.append(("ocr", pytesseract.image_to_string(image)))
            except pytesseract.TesseractNotFoundError:
                raise SystemExit("tesseract is not installed")
    return results

if __name__ == "__main__":
    for i, (source, text) in enumerate(page_text_with_fallback(Path("in/supplier-invoice.pdf")), 1):
        print(f"page {i}: {source}, {len(text)} chars")

For whole-file conversion, ocrmypdf --force-ocr rasterises and re-OCRs every page, replacing the broken text layer — see make scanned PDFs searchable with OCRmyPDF. The per-page approach above is better inside extraction pipelines because it keeps exact native text wherever it exists.

Variant: Ligatures and Missing Letters

A milder form of the same problem drops or replaces specific letter pairs: finance extracts as nance, offer as o er. The font maps the fi and ff ligature glyphs to nothing or to a single private-use code. Normalise known ligatures after extraction and treat private-use characters as a quality signal:

# pip install pymupdf
import unicodedata

def normalise_ligatures(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)        # 'fi' -> 'fi', 'ff' -> 'ff'
    private = sum(0xE000 <= ord(c) <= 0xF8FF for c in text)
    if private:
        print(f"warning: {private} private-use character(s) remain")
    return text

NFKC also folds full-width digits and some symbols; if exact characters matter — for instance in product codes — apply it only to prose fields. PyMuPDF preserves ligatures as single characters unless TEXT_PRESERVE_LIGATURES is turned off in the flags; the NFKC pass handles both cases.

Choosing a fix for undecodable text The root asks what the garbled output looks like. cid numbers lead to trying PyMuPDF first and falling back to OCR. Shifted but consistent letters lead to finding a constant offset from a known word. Missing letter pairs lead to NFKC ligature normalisation. Scrambled output or Type3 fonts lead directly to OCR of the affected pages. What does the garbage look like? sample spans from the broken font (cid:NN) Switch engine PyMuPDF often recovers shifted letters Constant offset derive from a known word fi or ff missing Ligatures NFKC normalise scrambled / Type3 No mapping exists glyphs only Still cid OCR the page Remap per font record offset OCR broken pages keep native elsewhere

Verification

Score each page's final text and fail the document if any page still looks broken. Store which method produced each page, so later quality issues can be traced to OCR or remapping rather than to the source document.

# pip install pymupdf pytesseract pillow
from pathlib import Path

def verify_pages(results: list[tuple[str, str]], min_chars: int = 40) -> None:
    problems = []
    for i, (source, text) in enumerate(results, 1):
        if len(text.strip()) < min_chars:
            problems.append(f"page {i} ({source}): almost no text")
        elif "(cid:" in text or "\N{REPLACEMENT CHARACTER}" in text:
            problems.append(f"page {i} ({source}): undecoded characters remain")
    assert not problems, "; ".join(problems)
    counts = {s: sum(1 for src, _ in results if src == s) for s in {src for src, _ in results}}
    print(f"all {len(results)} pages decoded: {counts}")

if __name__ == "__main__":
    from fix_cid import page_text_with_fallback          # the Fix 3 function saved as fix_cid.py
    verify_pages(page_text_with_fallback(Path("in/supplier-invoice.pdf")))

Spot-check a remapped or OCR'd page against the rendered page for numbers specifically. A shift remap that is off by one produces confident-looking wrong digits, and OCR confuses 1, 7 and l in narrow fonts — exactly the characters that matter on an invoice.

FAQ

Why do some viewers let me copy the text correctly when Python cannot? A few viewers ship heuristics or glyph-name lookups for common subset fonts. When copy-paste works in one viewer, try PyMuPDF first; it shares more of that recovery logic than pdfminer-based tools.

Can I fix the PDF itself so every tool extracts it? In principle, by adding a ToUnicode CMap to the font object. In practice it is fragile and specific to one font subset. Re-exporting from the source application with "embed full fonts" or "tagged PDF" enabled is far more reliable if you can reach the producer.

Does this affect table extraction too? Yes. Camelot, tabula and pdfplumber all read the same codes, so broken cells show cid tokens. Fix the text layer (usually with OCR) before table extraction, as in how to extract tables from scanned PDFs.

Is redaction safe on these files? Search-based redaction cannot find values it cannot decode. Treat pages with broken fonts like scans — see redact scanned PDF regions with OCR.

Part of Extracting Text and Metadata from PDFs with Python.