Fix PDF Size Grows After Merging
Fifty customer statements, each about 200 KB, are merged into one monthly batch file — and the result is 31 MB instead of roughly 10. Worse, every rerun of the job on the same output makes it bigger: the batch file that was 31 MB on Monday is 46 MB after Tuesday's corrections were merged in.
>>> sum(p.stat().st_size for p in Path("in/statements").glob("*.pdf")) / 1e6
10.2
>>> Path("out/statements-2026-09.pdf").stat().st_size / 1e6
31.4
Root Cause
Every input PDF is self-contained: it embeds its own copies of the fonts it uses, its own logo image, its own colour profile. A naive merge copies each page together with everything that page references, so fifty statements produce fifty copies of the same bold font, fifty copies of the same letterhead logo and fifty copies of the same ICC profile. Nothing in the merge step notices that those objects are byte-identical. Separately, two save habits add weight on top: saving over an existing output incrementally appends new objects without removing the old ones, and merging with the "clone the whole document" approach carries along objects that no copied page references — form field definitions, unused images, outlines from pages that were not included.
The file is not larger because content was added; it is larger because identical content was stored many times, and orphans were kept.
Minimal Diagnostic
Hash every stream in the merged file and count duplicates. A handful of hashes appearing fifty times each is the signature of this problem.
# pip install pymupdf
import hashlib
from collections import defaultdict
from pathlib import Path
import pymupdf
TARGET = Path("out/statements-2026-09.pdf")
def duplicate_streams(pdf_path: Path, top: int = 8) -> None:
groups: dict[str, list[int]] = defaultdict(list)
sizes: dict[str, int] = {}
kinds: dict[str, str] = {}
try:
doc = pymupdf.open(pdf_path)
except (pymupdf.FileDataError, RuntimeError) as exc:
raise SystemExit(f"cannot open {pdf_path}: {exc}")
with doc:
for xref in range(1, doc.xref_length()):
try:
if not doc.xref_is_stream(xref):
continue
raw = doc.xref_stream_raw(xref) or b""
except RuntimeError:
continue
digest = hashlib.sha1(raw).hexdigest()
groups[digest].append(xref)
sizes[digest] = len(raw)
subtype = doc.xref_get_key(xref, "Subtype")[1]
kinds[digest] = "image" if subtype == "/Image" else (
"font" if doc.xref_get_key(xref, "Length1")[0] != "null" or "Font" in subtype else "other")
wasted = sum(sizes[d] * (len(x) - 1) for d, x in groups.items())
print(f"file: {pdf_path.stat().st_size / 1e6:.1f} MB, duplicated bytes: {wasted / 1e6:.1f} MB")
for digest, xrefs in sorted(groups.items(), key=lambda kv: -sizes[kv[0]] * len(kv[1]))[:top]:
if len(xrefs) > 1:
print(f" {kinds[digest]:>5} {sizes[digest] / 1e3:8.1f} KB x{len(xrefs)}")
if __name__ == "__main__":
duplicate_streams(TARGET)
file: 31.4 MB, duplicated bytes: 19.6 MB
font 212.4 KB x50
font 168.9 KB x50
image 21.7 KB x50
other 3.1 KB x50
Nearly twenty megabytes of the file is duplicate copies. If duplicated bytes are small but the file is still much larger than the sum of its inputs, the extra weight is orphaned objects — see Variant Fix 1.
Fix: Merge, Then Deduplicate and Garbage-Collect
With pypdf, merge into a fresh writer, then ask it to merge identical objects and drop orphans before writing. Every changed line is commented.
# pip install "pypdf>=5.0"
from pathlib import Path
from pypdf import PdfReader, PdfWriter
INPUTS = sorted(Path("in/statements").glob("*.pdf"))
DEST = Path("out/statements-2026-09.pdf")
def merge_compact(inputs: list[Path], dest: Path) -> int:
writer = PdfWriter() # changed: always a fresh writer
for path in inputs:
try:
writer.append(PdfReader(path), import_outline=False) # changed: skip per-file outlines
except Exception as exc:
raise RuntimeError(f"cannot append {path.name}: {exc}") from exc
writer.compress_identical_objects( # changed: merge byte-identical objects
remove_identicals=True,
remove_orphans=True, # changed: drop unreferenced objects
)
for page in writer.pages:
page.compress_content_streams() # changed: deflate page content
tmp = dest.with_suffix(".part")
dest.parent.mkdir(parents=True, exist_ok=True)
with tmp.open("wb") as fh: # changed: write new file, never append
writer.write(fh)
tmp.replace(dest) # changed: atomic swap into place
return dest.stat().st_size
if __name__ == "__main__":
print(f"{merge_compact(INPUTS, DEST) / 1e6:.1f} MB")
9.8 MB
compress_identical_objects compares objects by content, so it only merges fonts that are truly byte-identical. Statements from the same generator share identical font subsets only when the subsets contain the same glyphs — a statement with a Polish customer name may carry a slightly larger subset of the same font, which stays separate. That is correct behaviour; the saving is still large because most documents share most subsets. The outline decision is covered in fix merged PDF bookmarks lost — pass import_outline=True when readers need per-statement bookmarks, at a small cost in size.
Variant Fix 1: The File Grows on Every Rerun
A job that opens yesterday's batch file, adds corrected statements and saves in place with PyMuPDF's saveIncr() or save(..., incremental=True) appends. Each run leaves the previous version's objects in the file. Rebuild from the inputs, or at minimum do a full rewrite:
# pip install pymupdf
from pathlib import Path
import pymupdf
def rewrite_clean(path: Path) -> tuple[int, int]:
before = path.stat().st_size
tmp = path.with_suffix(".clean.pdf")
with pymupdf.open(path) as doc:
doc.save(tmp, garbage=4, deflate=True, clean=True) # full save: orphans and duplicates removed
tmp.replace(path)
return before, path.stat().st_size
garbage=4 in PyMuPDF performs the same identical-object merging as pypdf's call, so this one rewrite fixes both the orphan and the duplicate problem for files produced by any merge tool. Building each batch from its source inputs is still better practice: it makes reruns deterministic and removes any doubt about which corrections a file contains, a concern in batch merge PDFs with a Python script.
Variant Fix 2: Large Files Where pypdf Is Too Slow
compress_identical_objects compares every object with every other and can take minutes on batches of thousands of statements. pikepdf, backed by the C++ qpdf library, merges pages faster, and a PyMuPDF garbage-collected save then deduplicates quickly:
# pip install pikepdf pymupdf
from pathlib import Path
import pikepdf
import pymupdf
def fast_merge(inputs: list[Path], dest: Path) -> None:
tmp = dest.with_suffix(".merged.pdf")
with pikepdf.new() as out:
for path in inputs:
try:
with pikepdf.open(path) as src:
out.pages.extend(src.pages)
except pikepdf.PdfError as exc:
raise RuntimeError(f"{path.name}: {exc}") from exc
out.save(tmp)
with pymupdf.open(tmp) as doc:
doc.save(dest, garbage=4, deflate=True) # deduplicate identical streams
tmp.unlink(missing_ok=True)
On a batch of 2,000 statements this pattern typically finishes in well under a minute, where pypdf deduplication takes several.
Preventing Duplicates at the Source
When the inputs are generated by your own code, you can often avoid producing a batch of self-contained files in the first place. If statements are rendered with ReportLab and only ever delivered as one batch, draw every statement into a single canvas: the fonts and the logo are then embedded once, by construction, and no deduplication pass is needed.
# pip install reportlab
from pathlib import Path
from reportlab.lib.pagesizes import A4
from reportlab.lib.utils import ImageReader
from reportlab.pdfbase import pdfmetrics
from reportlab.pdfbase.ttfonts import TTFont
from reportlab.pdfgen import canvas
FONT_PATH = Path("assets/DejaVuSans.ttf")
LOGO = Path("assets/logo.png")
def render_batch(customers: list[dict], dest: Path) -> None:
try:
pdfmetrics.registerFont(TTFont("Body", str(FONT_PATH))) # registered once, embedded once
logo = ImageReader(str(LOGO)) # one image object, reused per page
except (OSError, Exception) as exc:
raise RuntimeError(f"assets missing or unreadable: {exc}") from exc
dest.parent.mkdir(parents=True, exist_ok=True)
pdf = canvas.Canvas(str(dest), pagesize=A4)
for customer in customers:
pdf.drawImage(logo, 40, 770, width=120, height=40, mask="auto")
pdf.setFont("Body", 11)
pdf.drawString(40, 720, f"Statement for {customer['name']}")
pdf.drawString(40, 700, f"Balance: {customer['balance']:,.2f}")
pdf.showPage() # new page, same resources
pdf.save()
ReportLab caches an ImageReader by content and writes the image object once however many pages draw it, and a registered TrueType font is subset once across the whole canvas. The individual per-customer files can still be produced afterwards by splitting the batch, which is cheap — split a PDF by page ranges with Python shows the pattern, and each split file only carries the resources its own pages use.
When inputs come from third parties, you cannot change how they are produced, so the deduplication fix above remains the tool. Measure the saving per supplier on the first few batches: a supplier whose files never share font subsets with each other — because their generator randomises subset tags or embeds full fonts with different metadata — will not benefit, and for them the font-subsetting step in the parent guide is the one that helps.
Verification
Check that the merged file is close to the sum of unique content, that the page count equals the inputs' total, and that duplicates are gone.
# pip install pymupdf
import hashlib
from collections import Counter
from pathlib import Path
import pymupdf
def verify_merge(inputs: list[Path], merged: Path, max_ratio: float = 1.1) -> None:
expected_pages = 0
for path in inputs:
with pymupdf.open(path) as doc:
expected_pages += doc.page_count
with pymupdf.open(merged) as doc:
assert doc.page_count == expected_pages, f"{doc.page_count} pages, expected {expected_pages}"
digests = Counter()
for xref in range(1, doc.xref_length()):
try:
if doc.xref_is_stream(xref):
raw = doc.xref_stream_raw(xref) or b""
if len(raw) > 2048: # ignore tiny streams
digests[hashlib.sha1(raw).hexdigest()] += 1
except RuntimeError:
continue
dupes = {d: n for d, n in digests.items() if n > 1}
assert not dupes, f"{len(dupes)} large stream(s) still duplicated"
total_in = sum(p.stat().st_size for p in inputs)
ratio = merged.stat().st_size / total_in
assert ratio <= max_ratio, f"merged file is {ratio:.2f}x the inputs"
print(f"ok: {expected_pages} pages, {ratio:.2f}x input size, no large duplicates")
if __name__ == "__main__":
verify_merge(sorted(Path("in/statements").glob("*.pdf")), Path("out/statements-2026-09.pdf"))
A ratio well under 1.0 is normal after deduplication. A ratio above 1.1 means something is still stored more than once or orphans remain; rerun the diagnostic on the output to see which.
FAQ
Does deduplication change what the pages look like? No. Only byte-identical objects are merged, and every page keeps references to the same content it had before.
Why are two copies of the same font not merged?
They are different subsets — each contains only the glyphs its source document used. Merging them would need font re-subsetting, which doc.subset_fonts() in PyMuPDF can do on the merged file.
Should I compress images too? Only if images, not fonts, dominate the size after deduplication. Measure with the breakdown in Compressing and Optimizing PDFs before touching image quality.
Is it safe to merge statements containing form fields? Field names collide across copies of the same form. Flatten them first, or rename fields per input, as described in flatten PDF form fields with Python.
Related
- Compressing and Optimizing PDFs with Python — size breakdown and technique selection
- Batch Merge PDFs with a Python Script — the merge job this fix plugs into
- Reduce PDF File Size with Python — when images rather than duplicates are the weight
- Fix Redacted Text Still Searchable in PDF — orphaned objects as a privacy problem, not just a size one