Python Doc & Data Automation

Batch Replace Placeholders in Many DOCX Files

A regional office sends 260 letter templates that need the new company name, address and registration number, and a second request follows: produce one engagement letter per client from a master template, filling [CLIENT], [FEE] and [START] from an Excel sheet. The one-document script works. Run over the folder, it overwrites three templates that were open in Word, crashes on ~$letter-17.docx, reports success for 40 files where a placeholder was misspelled as [CLEINT], and leaves no record of which files changed.

Root Cause

Batch replacement fails at the edges of the single-document logic, not in it. Word creates hidden lock files named ~$…docx beside every open document, and they are not valid packages. Writing outputs over inputs turns any bug into data loss and makes reruns non-idempotent. A replacement function that reports only "done" cannot distinguish "replaced 3 placeholders" from "replaced 0 because the template spells it differently". Templates drift: some use [CLIENT], others {{client}} or «Client» left from an old mail merge, and a mapping written for one convention silently skips the rest. And per-row generation from Excel inherits every data problem in the sheet — blank cells, dates formatted as serial numbers, numbers with float noise — which then appear verbatim in client-facing letters.

Minimal Diagnostic

Before replacing anything, scan the whole folder for placeholder-like tokens and compare them with the keys you plan to replace. Unknown tokens are the misspellings and legacy conventions that would otherwise ship unreplaced.

# pip install "python-docx>=1.1"
import re
from collections import Counter
from pathlib import Path
from docx import Document

SOURCE_DIR = Path("in/templates")
MAPPING_KEYS = {"[COMPANY]", "[ADDRESS]", "[REG_NO]"}
TOKEN = re.compile(r"\[[A-Z_]{3,}\]|\{\{\s*\w+\s*\}\}|«[^»]{2,30}»")

def scan(folder: Path) -> None:
    found: Counter = Counter()
    files_with: dict[str, list[str]] = {}
    bad_files = []
    for path in sorted(folder.glob("*.docx")):
        if path.name.startswith("~$"):
            continue
        try:
            doc = Document(path)
        except Exception as exc:
            bad_files.append(f"{path.name}: {exc}")
            continue
        text = "\n".join(p.text for p in doc.paragraphs)
        text += "\n".join(c.text for t in doc.tables for r in t.rows for c in r.cells)
        text += "\n".join(p.text for s in doc.sections for p in s.header.paragraphs + s.footer.paragraphs)
        for token in set(TOKEN.findall(text)):
            found[token] += 1
            files_with.setdefault(token, []).append(path.name)
    for token, n in found.most_common():
        status = "mapped" if token in MAPPING_KEYS else "UNKNOWN"
        print(f"{status:>8} {token:<20} in {n} file(s), e.g. {files_with[token][:2]}")
    for missing in sorted(MAPPING_KEYS - set(found)):
        print(f"{'UNUSED':>8} {missing}")
    for line in bad_files:
        print(f"{'ERROR':>8} {line}")

if __name__ == "__main__":
    scan(SOURCE_DIR)

  mapped [COMPANY]            in 258 file(s), e.g. ['letter-001.docx', 'letter-002.docx']
  mapped [ADDRESS]            in 241 file(s), e.g. ['letter-001.docx', 'letter-003.docx']
 UNKNOWN [ADRESS]             in 17 file(s), e.g. ['letter-002.docx', 'letter-044.docx']
 UNKNOWN «Company»            in 2 file(s), e.g. ['legacy-a.docx', 'legacy-b.docx']
  mapped [REG_NO]             in 260 file(s), e.g. ['letter-001.docx', 'letter-002.docx']

Seventeen templates misspell the address placeholder and two still carry Word mail-merge fields. The mapping needs [ADRESS] as an alias, and the legacy files need separate handling.

Placeholder tokens found across 260 templates The folder scan found COMPANY in 258 files, REG_NO in all 260, ADDRESS in 241, a misspelled ADRESS token in 17 files that the mapping would miss, and legacy mail-merge Company markers in 2 files that need separate handling. Files containing each token (260 templates) [REG_NO] 260 files [COMPANY] 258 files [ADDRESS] 241 files [ADRESS] (misspelt) 17 files «Company» (legacy) 2 files

Fix: A Batch Runner with Safe Outputs and a Manifest

The runner skips lock files, writes to a separate folder through a temporary name, records counts per placeholder per file, flags unknown tokens left after replacement, and never touches the inputs. The cross-run replacement comes from fix python-docx replace text split across runs; the paragraph iterator from Find and Replace Text in Word Documents. Changed lines carry comments.

# pip install "python-docx>=1.1" "pandas>=2.2"
import re
from pathlib import Path
import pandas as pd
from docx import Document

from wordreplace import replace_in_paragraph, iter_all_paragraphs   # shared helpers

SOURCE_DIR = Path("in/templates")
OUT_DIR = Path("out/templates-2026")
MAPPING = {
    "[COMPANY]": "Northwind Group plc",
    "[ADDRESS]": "3 Wharf Road, Leeds LS1 4AB",
    "[ADRESS]": "3 Wharf Road, Leeds LS1 4AB",                 # changed: alias for the misspelling
    "[REG_NO]": "09876543",
}
LEFTOVER = re.compile(r"\[[A-Z_]{3,}\]|\{\{\s*\w+\s*\}\}|«[^»]{2,30}»")

def process(path: Path) -> dict:
    record = {"file": path.name, "status": "ok", **{k: 0 for k in MAPPING}, "leftover": ""}
    doc = Document(path)
    keys = sorted(MAPPING, key=len, reverse=True)
    pattern = re.compile("|".join(re.escape(k) for k in keys))
    def repl(m):
        record[m.group(0)] += 1                                  # changed: count per placeholder
        return MAPPING[m.group(0)]
    texts = []
    for paragraph in iter_all_paragraphs(doc):
        replace_in_paragraph(paragraph, pattern, repl)
        texts.append(paragraph.text)
    leftovers = sorted(set(LEFTOVER.findall("\n".join(texts))))
    if leftovers:
        record["status"] = "check"                               # changed: flag, do not fail silently
        record["leftover"] = " ".join(leftovers)
    tmp = OUT_DIR / f".{path.stem}.tmp.docx"
    doc.save(tmp)                                                # changed: write to a temp name ...
    tmp.replace(OUT_DIR / path.name)                             # ... then move into place atomically
    return record

def run_batch() -> pd.DataFrame:
    if SOURCE_DIR.resolve() == OUT_DIR.resolve():
        raise SystemExit("output folder must differ from input folder")   # changed: never overwrite
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    rows = []
    for path in sorted(SOURCE_DIR.glob("*.docx")):
        if path.name.startswith("~$"):                           # changed: skip Word lock files
            continue
        try:
            rows.append(process(path))
        except Exception as exc:
            rows.append({"file": path.name, "status": "error", "leftover": str(exc)})
    manifest = pd.DataFrame(rows)
    manifest.to_csv(OUT_DIR / "manifest.csv", index=False)
    return manifest

if __name__ == "__main__":
    manifest = run_batch()
    print(manifest["status"].value_counts().to_string())

status
ok       258
check      2

The manifest is the deliverable as much as the documents are: it shows every file, how many times each placeholder was replaced, and which files still contain something placeholder-shaped. The two check rows are the legacy mail-merge templates, now listed explicitly instead of discovered by a client.

What the batch runner does per file For each file the runner skips Word lock files, opens the document, replaces placeholders across runs while counting each key, scans all paragraphs for leftover placeholder-shaped tokens, saves to a temporary name and moves it into the output folder, and appends a manifest row with status ok, check or error. Skip lock files names starting with ~$ are not documents Replace and count cross-run replacement; one counter per placeholder Scan for leftovers any [TOKEN], {{ token }} or «Field» still present → status check Atomic save temp file then rename into a separate output folder Manifest row file, status, counts per placeholder, leftover tokens

Variant Fix 1: One Document per Row from Excel

Generating one filled document per client from a master template adds data cleaning between the sheet and the replacement. Format values for people, not for Python, and refuse rows with blank required fields:

# pip install "python-docx>=1.1" "pandas>=2.2" openpyxl
import re
from pathlib import Path
import pandas as pd
from docx import Document

from wordreplace import replace_in_paragraph, iter_all_paragraphs

TEMPLATE = Path("in/engagement-letter.docx")
SHEET = Path("in/clients.xlsx")
OUT = Path("out/letters")
REQUIRED = ["client", "fee", "start"]

def display(value, column: str) -> str:
    if pd.isna(value):
        return ""
    if column == "fee":
        return f"£{float(value):,.2f}"
    if column == "start":
        return pd.Timestamp(value).strftime("%-d %B %Y")         # 1 October 2026
    return str(value).strip()

def safe_name(text: str) -> str:
    return re.sub(r"[^\w\-]+", "-", text).strip("-")[:60]

def letters_from_sheet() -> pd.DataFrame:
    rows = pd.read_excel(SHEET, dtype={"client_id": "string"})
    OUT.mkdir(parents=True, exist_ok=True)
    results = []
    for _, row in rows.iterrows():
        values = {f"[{c.upper()}]": display(row[c], c) for c in REQUIRED}
        missing = [c for c in REQUIRED if not values[f"[{c.upper()}]"]]
        if missing:
            results.append({"client_id": row["client_id"], "status": f"skipped: blank {missing}"})
            continue
        doc = Document(TEMPLATE)
        pattern = re.compile("|".join(re.escape(k) for k in values))
        for p in iter_all_paragraphs(doc):
            replace_in_paragraph(p, pattern, lambda m: values[m.group(0)])
        dest = OUT / f"{row['client_id']}-{safe_name(values['[CLIENT]'])}.docx"
        doc.save(dest)
        results.append({"client_id": row["client_id"], "status": "ok", "file": dest.name})
    return pd.DataFrame(results)
From a spreadsheet row to a finished letter Each client row is read from the Excel sheet. Values are formatted for people, with fees as pounds with thousands separators and dates as day month year. Rows with blank required fields are skipped and recorded. The template is reopened fresh for every row so values cannot leak between clients. Placeholders are replaced across runs, and the document is saved under the client id and a safe version of the client name. Read row clients.xlsx Format values £1,250.00, 1 October 2026 Check required blank = skip Fresh template reopened per row Replace and save id-name.docx Blank field recorded as skipped

Reopening the template for every row matters: python-docx edits the document in memory, so reusing one Document object would carry the first client's values into every letter. %-d removes the day's leading zero on Linux and macOS; on Windows use %#d. For richer per-row documents — tables of line items, optional clauses — move to a Jinja template as described in generate one DOCX per row from Excel.

Variant Fix 2: Large Folders in Parallel

Each document is independent, so the batch parallelises cleanly across processes. Keep results flowing back to the parent, which alone writes the manifest:

# pip install "python-docx>=1.1" "pandas>=2.2"
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
import os
import pandas as pd

def run_parallel(paths: list[Path], workers: int | None = None) -> pd.DataFrame:
    workers = workers or max(1, (os.cpu_count() or 2) - 1)
    rows = []
    with ProcessPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(process, p): p for p in paths}     # process() from the fix
        for future in as_completed(futures):
            try:
                rows.append(future.result())
            except Exception as exc:
                rows.append({"file": futures[future].name, "status": "error", "leftover": str(exc)})
    return pd.DataFrame(rows).sort_values("file")

With a few hundred files, parallelism saves little because each document takes milliseconds; it pays off at thousands of documents or when templates are large. Network shares are the usual bottleneck — copy inputs locally first.

Verification

Assert on the manifest and on a sample of outputs. Every input must appear exactly once, no file may be error, every check file must be acknowledged, and each output must contain each mapped value when its template contained the key.

# pip install "python-docx>=1.1" "pandas>=2.2"
from pathlib import Path
import pandas as pd
from docx import Document

def verify_batch(source_dir: Path, out_dir: Path, acknowledged: set[str] = frozenset()) -> None:
    manifest = pd.read_csv(out_dir / "manifest.csv")
    inputs = {p.name for p in source_dir.glob("*.docx") if not p.name.startswith("~$")}
    assert set(manifest["file"]) == inputs, "manifest does not cover every input exactly"
    assert not manifest["file"].duplicated().any(), "file processed twice"
    errors = manifest[manifest["status"] == "error"]
    assert errors.empty, f"errors: {errors['file'].tolist()}"
    unacknowledged = set(manifest.loc[manifest["status"] == "check", "file"]) - set(acknowledged)
    assert not unacknowledged, f"files with leftover placeholders: {sorted(unacknowledged)}"
    sample = manifest[manifest["status"] == "ok"].sample(min(10, len(manifest)), random_state=1)
    for name in sample["file"]:
        text = "\n".join(p.text for p in Document(out_dir / name).paragraphs)
        assert "Northwind Group plc" in text, f"{name}: company name not found in output"
    print(f"batch verified: {len(manifest)} files, {len(unacknowledged)} unacknowledged checks")

Acknowledging check files explicitly — passing their names — forces a person to decide about each one rather than letting a growing list of warnings become background noise. Sampling outputs with a fixed random seed makes the spot check reproducible between runs.

FAQ

Should I use docxtpl instead for batch jobs? For new templates, yes: Jinja tags handle loops and conditions and are designed for generation. For editing existing documents that were never designed as templates, run-aware replacement is the right tool.

How do I handle «Field» mail-merge fields? They are Word fields (MERGEFIELD), not plain text. Replace the whole field element — w:fldSimple or the w:fldChar begin/separate/end sequence — with a run containing the value, or convert the template once to plain placeholders.

Can the job run while people have templates open? Reading is fine; Word's lock file is skipped. Never write outputs into a folder where people open files, or the atomic rename can fail on Windows with a sharing violation.

How do I roll back? Because inputs are untouched and outputs go to a new folder, rolling back is deleting the output folder. Keep the manifest from each run to know what was produced.

Part of Find and Replace Text in Word Documents with Python.