Python Doc & Data Automation

Fix PDF Text Extraction Missing Spaces

Text extracted from a PDF reads Totalrevenuefortheyearended31December — words glued together — while the same page looks perfectly spaced in a viewer. The mirror-image symptom shows up in the same documents: headings come out as A N N U A L R E P O R T, and numbers like 1 2 4 0 . 5 0 are split into single characters. Search, regex matching and any downstream tokeniser break on both.

>>> import pdfplumber
>>> pdfplumber.open("in/report.pdf").pages[2].extract_text()[:60]
'Totalrevenuefortheyearended31December2025increasedby8%'

Root Cause

Most PDF generators do not write space characters between words. They position each word — or each glyph — at explicit coordinates, or use the TJ operator with numeric kerning adjustments to shift the pen right. The visual gap is geometry, not a character. Every extractor therefore infers spaces: when the horizontal distance between the end of one character and the start of the next exceeds a tolerance, it inserts a space. Tight typesetting, justified text, condensed fonts and small font sizes produce inter-word gaps below that tolerance, so words merge. Letter-spaced headings and tabular numbers with generous tracking produce inter-character gaps above it, so letters split. The tolerance is a library default tuned for average documents; your document is not average.

Measured character gaps on one page On the sample page the median gap between characters inside a word is 0.2 points. The median gap between words in justified body text is 2.1 points, below pdfplumber's default x tolerance of 3 points, so words merge. The median gap between letters in a letter-spaced heading is 4.6 points, above the tolerance, so letters split. A single fixed tolerance cannot separate both correctly. Median horizontal gap in points on a justified report page Within a word 0.2 pt Between words (body) 2.1 pt Default x_tolerance 3.0 pt Within letter-spaced heading 4.6 pt Body word gaps fall below the tolerance and merge; heading letter gaps rise above it and split

Minimal Diagnostic

Measure the actual gaps on the problem page. pdfplumber exposes every character with its coordinates, so a histogram of gaps between consecutive characters on the same line shows where the natural break between "inside a word" and "between words" sits.

# pip install pdfplumber
from collections import Counter
from pathlib import Path
import pdfplumber

SOURCE = Path("in/report.pdf")
PAGE = 2

def gap_histogram(pdf_path: Path, page_index: int) -> Counter:
    try:
        pdf = pdfplumber.open(pdf_path)
    except Exception as exc:
        raise SystemExit(f"cannot open {pdf_path}: {exc}")
    with pdf:
        chars = sorted(pdf.pages[page_index].chars, key=lambda c: (round(c["top"]), c["x0"]))
    gaps = Counter()
    for a, b in zip(chars, chars[1:]):
        if round(a["top"]) != round(b["top"]):          # different line
            continue
        if a["text"] == " " or b["text"] == " ":        # real space characters
            continue
        gap = b["x0"] - a["x1"]
        gaps[round(gap, 1)] += 1
    return gaps

if __name__ == "__main__":
    hist = gap_histogram(SOURCE, PAGE)
    for gap, count in sorted(hist.items()):
        if -1 <= gap <= 8:
            print(f"{gap:5.1f} pt  {'#' * min(count // 5, 60)}")
    spaces = sum(1 for c in pdfplumber.open(SOURCE).pages[PAGE].chars if c["text"] == " ")
    print(f"space characters on page: {spaces}")

  0.0 pt  ############################################################
  0.2 pt  #####################################
  1.9 pt  ##########
  2.1 pt  ##################
  2.4 pt  ######
space characters on page: 0

Zero real space characters and a second cluster of gaps around two points confirms the cause: word boundaries exist only as geometry, and they sit below the default three-point tolerance. The valley between the clusters — here roughly 0.5 to 1.8 points — is where the tolerance belongs.

Fix: Set the Tolerance from the Measured Gaps

In pdfplumber, x_tolerance is the maximum gap still treated as part of the same word. Set it inside the valley.

# pip install pdfplumber
from pathlib import Path
import pdfplumber

SOURCE = Path("in/report.pdf")

def extract_with_tolerance(pdf_path: Path, x_tol: float = 1.0) -> list[str]:
    try:
        with pdfplumber.open(pdf_path) as pdf:
            return [
                page.extract_text(
                    x_tolerance=x_tol,          # changed: below the word-gap cluster (was default 3)
                    y_tolerance=3,
                    keep_blank_chars=False,
                ) or ""
                for page in pdf.pages
            ]
    except Exception as exc:
        raise RuntimeError(f"extraction failed: {exc}") from exc

if __name__ == "__main__":
    print(extract_with_tolerance(SOURCE)[2][:80])
    # Total revenue for the year ended 31 December 2025 increased by 8%

A fixed tolerance in points breaks when font size varies across the document: a gap of one point separates words in 7-point footnotes but is inter-letter spacing in a 24-point title. Recent pdfplumber versions accept x_tolerance_ratio, which scales the tolerance by each character's size:

# pip install "pdfplumber>=0.11"
import pdfplumber

with pdfplumber.open("in/report.pdf") as pdf:
    text = pdf.pages[2].extract_text(x_tolerance_ratio=0.15)   # 15% of the font size

Measure first with the histogram, express the valley as a fraction of the dominant font size, and use that ratio.

Choosing the Tolerance Automatically

Batches mix generators, and a tolerance that suits one supplier's invoices merges words in another's. Rather than hard-coding a value, estimate it per document from the gap distribution: collect gaps relative to font size, then place the threshold in the widest empty band between the "inside a word" cluster near zero and the "between words" cluster.

# pip install pdfplumber
from pathlib import Path
import pdfplumber

def estimate_ratio(pdf_path: Path, max_pages: int = 5, default: float = 0.15) -> float:
    """Estimate a word-gap ratio (gap / font size) from the widest valley in the gap distribution."""
    ratios = []
    try:
        with pdfplumber.open(pdf_path) as pdf:
            for page in pdf.pages[:max_pages]:
                chars = sorted(page.chars, key=lambda c: (round(c["top"]), c["x0"]))
                for a, b in zip(chars, chars[1:]):
                    if round(a["top"]) != round(b["top"]) or " " in (a["text"], b["text"]):
                        continue
                    gap = b["x0"] - a["x1"]
                    if 0 <= gap < a["size"]:                # ignore column gutters
                        ratios.append(round(gap / a["size"], 2))
    except Exception as exc:
        raise RuntimeError(f"cannot sample {pdf_path}: {exc}") from exc
    if len(ratios) < 200:
        return default                                      # too little text to trust
    present = sorted(set(ratios))
    best_gap, best_mid = 0.0, default
    for lo, hi in zip(present, present[1:]):
        if lo < 0.02:                                        # still inside the within-word cluster
            continue
        if hi - lo > best_gap:
            best_gap, best_mid = hi - lo, (lo + hi) / 2
    return best_mid if best_gap >= 0.04 else default

Log the estimated ratio with every processed file. When extraction quality drops for one supplier, a sudden change in that number — usually after they upgrade their invoicing software — points straight at the cause. Keep the default as a floor for short documents: a two-line delivery note does not have enough character pairs to show a clear valley, and a wrong estimate from a handful of gaps does more harm than the library default.

Variant Fix 1: PyMuPDF Words Run Together

PyMuPDF inserts spaces based on the font's own space width and glyph advances, and does not expose a tolerance parameter. When it merges words, rebuild them from characters in rawdict output with your own threshold:

# pip install pymupdf
from pathlib import Path
import pymupdf

def rebuild_lines(page: pymupdf.Page, ratio: float = 0.12) -> list[str]:
    """Rebuild each line's text, inserting a space where the gap exceeds ratio x font size."""
    out = []
    raw = page.get_text("rawdict", flags=pymupdf.TEXT_MEDIABOX_CLIP)
    for block in raw["blocks"]:
        for line in block.get("lines", []):
            pieces, prev_x1 = [], None
            for span in line["spans"]:
                threshold = span["size"] * ratio            # changed: size-relative word gap
                for ch in span["chars"]:
                    x0, _, x1, _ = ch["bbox"]
                    if prev_x1 is not None and x0 - prev_x1 > threshold and ch["c"] != " ":
                        pieces.append(" ")                   # changed: insert inferred space
                    pieces.append(ch["c"])
                    prev_x1 = x1
            out.append(" ".join("".join(pieces).split()))    # collapse doubled spaces
    return out

if __name__ == "__main__":
    with pymupdf.open(Path("in/report.pdf")) as doc:
        print("\n".join(rebuild_lines(doc[2])[:5]))

rawdict is slower than plain text extraction — reserve this path for documents your quality check flags. The detection metric is simple: a page whose average token length exceeds about twelve characters is almost certainly gluing words.

Rebuilding words from characters Four steps. Read characters with bounding boxes from rawdict. For each pair of neighbours on a line, compute the gap between the right edge of one and the left edge of the next. Compare it with the font size multiplied by a ratio measured from the gap histogram. Insert a space when the gap is larger, then collapse repeated spaces. Read chars with bbox page.get_text('rawdict') gives each character with x0 and x1 inside its span Compute neighbour gap gap = next.x0 - prev.x1, measured on the same line only Compare to size x ratio a ratio of 0.1 to 0.15 of the font size separates words in most body text Insert space and collapse add a space above the threshold, then normalise runs of whitespace

Variant Fix 2: Letter-Spaced Text Splits into Characters

The opposite problem needs the opposite adjustment, but lowering the tolerance globally re-breaks body text. Detect runs of single-character tokens and join them afterwards instead:

# pip install pdfplumber
import re
import pdfplumber

SPACED = re.compile(r"\b(?:[A-Za-z0-9] ){3,}[A-Za-z0-9]\b")    # four or more single chars

def join_letter_spaced(text: str) -> str:
    return SPACED.sub(lambda m: m.group(0).replace(" ", ""), text)

with pdfplumber.open("in/report.pdf") as pdf:
    raw = pdf.pages[0].extract_text(x_tolerance=1.0)
    print(join_letter_spaced(raw))       # 'S T A T E M E N T' -> 'STATEMENT'

Requiring four or more single characters avoids joining legitimate sequences such as A B testing or initials. For two-word spaced headings (A N N U A L R E P O R T), the double space between the words survives because the pattern only consumes single spaces; check that your extractor preserves the wider gap as two spaces, or split on the wider gap before joining.

Variant Fix 3: pdfminer-Based Pipelines

Tools built directly on pdfminer.six — including many older scripts — control word breaks through LAParams. word_margin is the gap, relative to character width, above which a space is inserted; char_margin groups characters into lines.

# pip install pdfminer.six
from pathlib import Path
from pdfminer.high_level import extract_text
from pdfminer.layout import LAParams

params = LAParams(
    word_margin=0.05,        # smaller than the 0.1 default: split words on tighter gaps
    char_margin=2.0,
    line_margin=0.5,
)
try:
    text = extract_text(Path("in/report.pdf"), page_numbers=[2], laparams=params)
except Exception as exc:
    raise SystemExit(f"pdfminer failed: {exc}")
print(text[:200])
Word-gap controls by library pdfplumber uses x_tolerance in points with a default of 3, or x_tolerance_ratio relative to font size. PyMuPDF has no tolerance parameter and needs rebuilding from rawdict characters. pdfminer uses word_margin relative to character width with a default of 0.1. pypdf layout mode infers spacing from positions and exposes no word gap parameter. Library Parameter Unit Default pdfplumber x_tolerance points 3 pdfplumber 0.11+ x_tolerance_ratio x font size off pdfminer.six word_margin x char width 0.1 PyMuPDF none exposed rebuild from rawdict

Verification

Check the fix across the whole document with a token-length statistic and a dictionary hit rate, both before and after. Glued words push the average token length up and the dictionary rate down; split letters push the share of one-character tokens up.

# pip install pdfplumber
import re
from pathlib import Path
import pdfplumber

COMMON = {"the", "and", "of", "to", "in", "for", "by", "on", "with", "year", "total", "revenue"}

def spacing_stats(pages: list[str]) -> dict:
    tokens = [t for p in pages for t in re.findall(r"[A-Za-z]+", p)]
    n = len(tokens) or 1
    return {
        "avg_token_len": sum(map(len, tokens)) / n,
        "single_char_share": sum(len(t) == 1 for t in tokens) / n,
        "common_word_share": sum(t.lower() in COMMON for t in tokens) / n,
    }

def compare(pdf_path: Path, x_tol: float) -> None:
    with pdfplumber.open(pdf_path) as pdf:
        before = spacing_stats([p.extract_text() or "" for p in pdf.pages])
        after = spacing_stats([p.extract_text(x_tolerance=x_tol) or "" for p in pdf.pages])
    for key in before:
        print(f"{key:>18}: {before[key]:.3f} -> {after[key]:.3f}")
    assert after["avg_token_len"] < 9, "tokens still look glued"
    assert after["single_char_share"] < 0.08, "too many single-character tokens"

if __name__ == "__main__":
    compare(Path("in/report.pdf"), x_tol=1.0)

     avg_token_len: 14.820 -> 5.310
 single_char_share: 0.004 -> 0.031
 common_word_share: 0.011 -> 0.142

An average English token length between four and six characters is healthy. If the single-character share climbs sharply after lowering the tolerance, you have overshot and are now splitting words; move the tolerance back toward the upper edge of the valley.

FAQ

Why does copy-paste from the viewer have correct spacing when extraction does not? Viewers run their own heuristics, often tuned per font, and some use the document's structure tree when it is tagged. The underlying file has no space characters either way.

Would OCR avoid the problem? OCR infers spaces from pixels and usually gets them right, but it introduces character errors and costs far more time. Tune the tolerance first; OCR only pages that stay broken.

Is this related to (cid:…) output? No. Missing spaces is a geometry problem with correctly decoded characters. cid codes mean the characters themselves could not be decoded; see fix CID garbled characters in PDF text.

Does the same tolerance affect table extraction? Yes. pdfplumber's table finder builds cell text with the same word logic, so glued cell values have the same fix — see fix PDF text extraction alignment issues.

Part of Extracting Text and Metadata from PDFs with Python.