Python Doc & Data Automation

Batch Convert DOCX to PDF in Parallel

Converting one file at a time is reliable and slow. The obvious speed-up is not:

with ThreadPoolExecutor(max_workers=8) as pool:
    pool.map(convert_one, docx_files)   # 200 files

Some PDFs appear, some do not, and the process exits zero either way. The log shows javaldx: Could not find a Java Runtime, occasional Error: source file could not be loaded, and — the real clue — several conversions that produced no output and no error at all.

Root Cause

A LibreOffice process keeps its state in a user profile directory, by default ~/.config/libreoffice/4/user, and takes an exclusive lock on it at startup. A second soffice invocation finding that lock does not start its own instance: it hands the document to the already-running one over a UNO connection and exits immediately — often before the first instance has finished writing the PDF. From the caller's point of view the subprocess returned zero, so nothing looks wrong; the output file simply never appears, or appears seconds after the code checked for it. The fix is not to serialise everything, but to give every worker a profile of its own so each really is a separate instance.

Why parallel conversions vanish The first soffice process starts, locks the shared user profile and begins converting. A second invocation starts, finds the lock, connects to the running instance over UNO, hands over the document and exits with status zero. The caller sees success and moves on, but the PDF has not been written yet, and under load the first instance may drop the queued document entirely. Giving each worker its own profile directory removes the shared lock. Worker 1 starts soffice locks the user profile, blue, converts normally Worker 2 starts soffice finds the profile locked, amber, does not start its own instance Worker 2 delegates over UNO hands the document over, amber, then exits with status 0 Caller sees success checks for the PDF, red, file is not there yet, or ever Per-worker profile no shared lock, green, each worker is a real instance

Minimal Diagnostic

Convert a small batch, then compare inputs to outputs instead of trusting exit codes.

# stdlib only
import subprocess, time
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

IN, OUT = Path("in"), Path("out")

def naive_convert(docx: Path) -> tuple[str, int, bool]:
    result = subprocess.run(
        ["soffice", "--headless", "--convert-to", "pdf", "--outdir", str(OUT), str(docx)],
        capture_output=True, text=True, timeout=180)
    produced = (OUT / f"{docx.stem}.pdf").exists()
    return docx.name, result.returncode, produced

if __name__ == "__main__":
    OUT.mkdir(exist_ok=True)
    files = sorted(IN.glob("*.docx"))
    started = time.monotonic()
    with ThreadPoolExecutor(max_workers=8) as pool:
        results = list(pool.map(naive_convert, files))
    missing = [name for name, _, produced in results if not produced]
    lying = [name for name, code, produced in results if code == 0 and not produced]
    print(f"{len(files)} in, {len(files) - len(missing)} out, {time.monotonic() - started:.1f}s")
    print(f"exit 0 but no PDF: {len(lying)} -> {lying[:5]}")

40 in, 27 out, 19.4s
exit 0 but no PDF: 13 -> ['inv-0007.docx', 'inv-0011.docx', 'inv-0012.docx', 'inv-0019.docx', 'inv-0023.docx']

Thirteen conversions reported success and produced nothing. Any batch pipeline that checks only the return code will ship those as completed.

Fix: One Profile per Worker, Bounded Concurrency

Give each worker a private profile directory, use processes rather than threads, and verify each output file before calling it done.

# stdlib only
import os, shutil, subprocess, tempfile
from concurrent.futures import ProcessPoolExecutor, as_completed
from dataclasses import dataclass
from pathlib import Path

IN, OUT = Path("in"), Path("out")

@dataclass(frozen=True)
class Result:
    source: str
    pdf: str | None
    error: str | None

def convert_one(docx: Path, outdir: Path, timeout: int = 180) -> Result:
    profile = Path(tempfile.mkdtemp(prefix=f"lo-{os.getpid()}-"))      # changed: private profile
    env = {**os.environ, "HOME": str(profile)}                          # changed: isolate HOME too
    try:
        completed = subprocess.run(
            ["soffice", "--headless", "--norestore", "--invisible",
             f"-env:UserInstallation=file://{profile}",                 # changed: no shared lock
             "--convert-to", "pdf", "--outdir", str(outdir), str(docx)],
            capture_output=True, text=True, timeout=timeout, env=env)
    except subprocess.TimeoutExpired:
        return Result(docx.name, None, f"timeout after {timeout}s")
    finally:
        shutil.rmtree(profile, ignore_errors=True)                      # changed: profiles are disposable
    pdf = outdir / f"{docx.stem}.pdf"
    if not pdf.exists() or pdf.stat().st_size == 0:                     # changed: trust the file, not the code
        detail = (completed.stderr or completed.stdout or "").strip().splitlines()
        return Result(docx.name, None, detail[-1] if detail else "no output produced")
    return Result(docx.name, str(pdf), None)

def convert_all(files: list[Path], outdir: Path, workers: int | None = None) -> list[Result]:
    workers = workers or max(1, min(4, (os.cpu_count() or 2) // 2))     # changed: bounded, LibreOffice is heavy
    outdir.mkdir(parents=True, exist_ok=True)
    results = []
    with ProcessPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(convert_one, f, outdir): f for f in files}
        for future in as_completed(futures):
            results.append(future.result())
    return results

if __name__ == "__main__":
    outcomes = convert_all(sorted(IN.glob("*.docx")), OUT)
    failed = [r for r in outcomes if r.error]
    print(f"{len(outcomes) - len(failed)}/{len(outcomes)} converted")
    for result in failed[:10]:
        print(f"  FAIL {result.source}: {result.error}")

Three changes carry the fix. -env:UserInstallation is what makes each soffice a genuinely separate instance; setting HOME as well catches the parts of LibreOffice that ignore that flag and write to ~/.cache. Checking the output file rather than the return code turns silent skips into reported failures. Bounding workers to about half the cores keeps the machine responsive — each instance is a full office suite, and oversubscribing produces timeouts rather than throughput.

Shared profile versus per-worker profile Eight workers sharing the default profile complete forty files in nineteen seconds but produce only twenty-seven PDFs, with thirteen silent skips reported as success. Four workers each with a private profile take thirty-one seconds and produce all forty PDFs, with failures reported as failures. The slower run is the only one that can be trusted. 8 workers, shared profile 40 files -> 27 PDFs 19.4s wall clock 13 silent skips, exit 0 failures invisible 4 workers, own profiles 40 files -> 40 PDFs 31.2s wall clock 0 silent skips failures reported

Variant Fix 1: Keep One Instance Warm Instead of Starting Many

LibreOffice takes one to two seconds to start. For large batches that startup dominates, and a persistent instance driven over UNO is faster:

# pip install unoserver
# terminal 1 — one server per port, one port per worker
#   unoserver --port 2003 --user-installation /tmp/lo-2003 &
#   unoserver --port 2004 --user-installation /tmp/lo-2004 &
import subprocess
from pathlib import Path

def convert_via_server(docx: Path, outdir: Path, port: int, timeout: int = 120) -> Path | None:
    pdf = outdir / f"{docx.stem}.pdf"
    completed = subprocess.run(
        ["unoconvert", "--port", str(port), "--convert-to", "pdf", str(docx), str(pdf)],
        capture_output=True, text=True, timeout=timeout)
    return pdf if pdf.exists() and pdf.stat().st_size else None

One server per port, each with its own user installation, gives the same isolation without paying startup per file. The trade is supervision: a server that wedges on a malformed document stays wedged, so the pool needs a health check and a restart path — the failure mode described in headless conversion timeout.

Variant Fix 2: Retry the Failures Once, Serially

A small number of failures in a large batch are usually contention rather than bad documents. Retrying them alone separates the two:

# stdlib only
from pathlib import Path

def convert_with_retry(files: list[Path], outdir: Path, workers: int = 4) -> tuple[list, list]:
    first = convert_all(files, outdir, workers=workers)
    failed = [Path("in") / r.source for r in first if r.error]
    if not failed:
        return first, []
    print(f"retrying {len(failed)} file(s) serially")
    second = convert_all(failed, outdir, workers=1)          # serial: no contention left to blame
    still_failing = [r for r in second if r.error]
    succeeded = [r for r in first if not r.error] + [r for r in second if not r.error]
    return succeeded, still_failing

Anything that fails serially is a genuine problem with the document — a corrupt file, a missing font that PDF/A rejects, an unsupported feature — and belongs in a quarantine directory with its error, not in another retry loop.

A batch conversion that reports honestly List the input documents and skip any whose PDF already exists and is newer. Convert in a bounded process pool, each worker with its own profile. Verify every output exists and is non-empty, treating a missing file as a failure regardless of exit code. Retry the failures serially, which removes contention as an explanation. Quarantine whatever still fails, with its error text. Report counts that add up to the input count. List inputs skip already-converted Convert, bounded pool own profile per worker Verify each output exists and non-empty Retry failures serially rules out contention Quarantine the rest file plus error text Report counts in equals out plus failed

Making the Batch Resumable

A two-hour batch that dies at ninety minutes should not start over. Skipping already-converted files costs three lines:

# stdlib only
from pathlib import Path

def pending(files: list[Path], outdir: Path) -> list[Path]:
    remaining = []
    for docx in files:
        pdf = outdir / f"{docx.stem}.pdf"
        if pdf.exists() and pdf.stat().st_size > 0 and pdf.stat().st_mtime >= docx.stat().st_mtime:
            continue                                   # up to date, skip
        remaining.append(docx)
    return remaining

Comparing modification times rather than mere existence means an edited source document is reconverted, which is what makes the check safe to leave in permanently. On a network share, be aware that the two timestamps may come from different clocks; adding a second of slack avoids reconverting the whole batch every run. Pair it with writing each PDF to a temporary name and renaming it into place on success, so an interrupted run never leaves a half-written PDF that the next run mistakes for finished work.

Verification

Prove the batch is complete and the PDFs are real documents, not zero-byte stubs.

# pip install pypdf
from pathlib import Path
from pypdf import PdfReader

def verify_batch(indir: Path, outdir: Path, quarantine: Path | None = None) -> None:
    sources = sorted(indir.glob("*.docx"))
    missing, empty, unreadable = [], [], []
    quarantined = {p.stem for p in quarantine.glob("*.docx")} if quarantine else set()
    for docx in sources:
        if docx.stem in quarantined:
            continue
        pdf = outdir / f"{docx.stem}.pdf"
        if not pdf.exists():
            missing.append(docx.name)
        elif pdf.stat().st_size == 0:
            empty.append(pdf.name)
        else:
            try:
                if len(PdfReader(pdf).pages) == 0:
                    unreadable.append(pdf.name)
            except Exception as error:
                unreadable.append(f"{pdf.name}: {type(error).__name__}")
    assert not missing, f"{len(missing)} PDF(s) missing: {missing[:5]}"
    assert not empty, f"{len(empty)} empty PDF(s): {empty[:5]}"
    assert not unreadable, f"{len(unreadable)} unreadable PDF(s): {unreadable[:5]}"
    print(f"{len(sources)} source(s), {len(quarantined)} quarantined, rest converted and readable")

if __name__ == "__main__":
    verify_batch(Path("in"), Path("out"), Path("quarantine"))

Opening each PDF is the part that matters. A conversion interrupted midway leaves a file of plausible size that no reader can open, and only parsing it catches that. Counting quarantined sources separately keeps the assertion honest: the batch is complete when every input is either converted or explicitly set aside with a reason.

FAQ

Threads or processes? Processes. The work happens in a subprocess either way, but processes keep per-worker state such as the profile path cleanly separated and survive one worker dying.

How many workers? Start at half the core count. LibreOffice is memory-hungry; watch resident memory before raising it, since swapping shows up as timeouts.

Can I convert to PDF/A in a batch? Yes — pass the export filter options shown in fix LibreOffice PDF fonts substituted. Expect a few documents to fail PDF/A validation and land in quarantine.

Should the pool size follow memory or cores? Memory, usually. Each instance holds roughly 150 to 300MB resident depending on the document, so a 2GB container comfortably runs four workers and thrashes at eight regardless of how many cores it reports.

Why does the first conversion take so much longer? It builds the user profile. With disposable per-worker profiles every conversion pays that cost, which is the argument for the persistent-server approach on large batches.

Part of Converting DOCX to PDF with Python.