Extracting Data from Word Documents with Python

Word documents are where a surprising amount of business data lives: supplier questionnaires returned as filled-in templates, contracts with key terms in a table on page two, inspection reports with a signature block in a text box, HR forms built from content controls. Turning a folder of those into a spreadsheet is a common automation request, and the obvious approach — loop over doc.paragraphs and doc.tables with python-docx — loses data in ways that are hard to notice. Paragraphs and tables come back as two separate lists with no indication of which table followed which heading. Horizontally merged cells return the same text several times. Anything inside a text box, a content control, a header, a footnote or a comment is simply absent.

A .docx file is a ZIP archive of XML parts, and python-docx exposes the common parts of the main document through a friendly object model. Where the object model stops, the XML is still there and readable with the lxml tools python-docx is built on. This guide extracts data in document order, handles the table quirks, reaches the content python-docx skips, and validates the result against the document so nothing is silently dropped. It is the reading-side counterpart to Automating Word Document Creation.

Prerequisites

python -m venv .venv && source .venv/bin/activate
pip install "python-docx>=1.1" "pandas>=2.2" lxml
mkdir -p in out

python-docx reads .docx only. Legacy .doc files must be converted first — LibreOffice's headless mode does it in batch, using the same approach as Converting DOCX to PDF with Python with --convert-to docx. Build a test set that includes one document with a text box, one with content controls, one with merged table cells and one with tracked changes; those four features account for nearly all extraction surprises.

Diagnostic: Inventory What the Document Contains

Count every content container in the XML — including the ones python-docx does not surface — and compare with what the object model returns. The gap tells you which extraction steps a document needs.

# pip install "python-docx>=1.1"
import zipfile
from pathlib import Path
from docx import Document

SOURCE = Path("in/supplier-questionnaire.docx")

def inventory(path: Path) -> dict:
    try:
        doc = Document(path)
    except (OSError, KeyError, ValueError) as exc:
        raise SystemExit(f"cannot open {path}: {exc}")
    body = doc.element.body
    counts = {
        "paragraphs (python-docx)": len(doc.paragraphs),
        "tables (python-docx)": len(doc.tables),
        "all w:p in body": len(body.xpath(".//w:p")),
        "all w:tbl in body": len(body.xpath(".//w:tbl")),
        "text boxes": len(body.xpath(".//w:txbxContent")),
        "content controls": len(body.xpath(".//w:sdt")),
        "legacy form fields": len(body.xpath(".//w:fldChar[@w:fldCharType='begin']")),
        "tracked insertions": len(body.xpath(".//w:ins")),
        "tracked deletions": len(body.xpath(".//w:del")),
        "header paragraphs": sum(len(s.header.paragraphs) for s in doc.sections),
    }
    with zipfile.ZipFile(path) as zf:
        names = set(zf.namelist())
    counts["has comments part"] = "word/comments.xml" in names
    counts["has footnotes part"] = "word/footnotes.xml" in names
    return counts

if __name__ == "__main__":
    for key, value in inventory(SOURCE).items():
        print(f"{key:>26}: {value}")

  paragraphs (python-docx): 64
      tables (python-docx): 3
           all w:p in body: 151
         all w:tbl in body: 4
                text boxes: 2
          content controls: 18
        legacy form fields: 0
        tracked insertions: 5
         tracked deletions: 3
         header paragraphs: 2
         has comments part: True
        has footnotes part: False

151 paragraphs exist in the body, python-docx lists 64 at top level: the rest live in table cells, text boxes and content controls. Four tables exist but three are top-level, so one is nested inside a cell. There are tracked changes and comments to decide about. Each of those needs a step below.

Core Implementation

Step 1: Walk the Body in Document Order

Iterate the body's child elements and wrap each as a Paragraph or Table, so headings stay attached to the tables that follow them. python-docx 1.1 provides iter_inner_content() for this; the explicit loop works on every version and makes the logic visible.

# pip install "python-docx>=1.1"
from pathlib import Path
from docx import Document
from docx.oxml.ns import qn
from docx.table import Table
from docx.text.paragraph import Paragraph

def iter_blocks(doc):
    """Yield Paragraph and Table objects in the order they appear in the body."""
    for child in doc.element.body.iterchildren():
        if child.tag == qn("w:p"):
            yield Paragraph(child, doc)
        elif child.tag == qn("w:tbl"):
            yield Table(child, doc)
        elif child.tag == qn("w:sdt"):                      # block-level content control
            content = child.find(qn("w:sdtContent"))
            if content is not None:
                for inner in content.iterchildren():
                    if inner.tag == qn("w:p"):
                        yield Paragraph(inner, doc)
                    elif inner.tag == qn("w:tbl"):
                        yield Table(inner, doc)

def outline_with_tables(path: Path) -> list[tuple[str, str]]:
    doc = Document(path)
    heading, found = "", []
    for block in iter_blocks(doc):
        if isinstance(block, Paragraph) and block.style.name.startswith("Heading"):
            heading = block.text.strip()
        elif isinstance(block, Table):
            found.append((heading, f"table {len(block.rows)}x{len(block.columns)}"))
    return found

Unwrapping block-level content controls in the same loop matters for templates built with Word's Developer tab: whole sections of such documents sit inside w:sdt elements, and a loop that only recognises w:p and w:tbl skips them entirely.

Step 2: Read Tables Without Duplicated Merged Cells

python-docx returns a cell object for every grid position, so a cell merged across three columns appears three times in row.cells. Compare the underlying XML elements to detect repeats:

# pip install "python-docx>=1.1" "pandas>=2.2"
import pandas as pd
from docx.table import Table

def table_rows(table: Table, fill_merged: bool = True) -> list[list[str]]:
    rows = []
    for row in table.rows:
        values, previous = [], None
        for cell in row.cells:
            text = "\n".join(p.text for p in cell.paragraphs).strip()
            if cell._tc is previous:                      # same merged cell repeated
                values.append(text if fill_merged else "")
            else:
                values.append(text)
            previous = cell._tc
        rows.append(values)
    return rows

def table_to_frame(table: Table, header_rows: int = 1) -> pd.DataFrame:
    rows = table_rows(table)
    if len(rows) <= header_rows:
        return pd.DataFrame(columns=rows[0] if rows else [])
    header = [" ".join(filter(None, parts)).strip() or f"col_{i}"
              for i, parts in enumerate(zip(*rows[:header_rows]))]
    return pd.DataFrame(rows[header_rows:], columns=header)

Filling merged cells with their value (rather than blanks) is usually right for data: a "Region" cell merged down five rows describes all five. For headers merged across columns, joining the header rows produces labels like Q1 Revenue — the same flattening idea used for spreadsheets in fix pivot table MultiIndex columns in Excel. The detailed table walkthrough, including nested tables and vertical merges, is in extract tables from Word documents to pandas.

Step 3: Read Content Controls as Named Fields

Content controls carry a tag and a title set by the template author, which makes them the most reliable way to extract form data: you read fields by name instead of by position.

# pip install "python-docx>=1.1"
from docx import Document
from docx.oxml.ns import qn

def content_controls(path) -> dict[str, str]:
    doc = Document(path)
    fields: dict[str, str] = {}
    for sdt in doc.element.body.iter(qn("w:sdt")):
        props = sdt.find(qn("w:sdtPr"))
        tag = props.find(qn("w:tag")) if props is not None else None
        alias = props.find(qn("w:alias")) if props is not None else None
        name = (tag.get(qn("w:val")) if tag is not None else None) or \
               (alias.get(qn("w:val")) if alias is not None else None)
        if not name:
            continue
        texts = [t.text or "" for t in sdt.iter(qn("w:t"))]
        showing_placeholder = props.find(qn("w:showingPlcHdr")) is not None
        fields[name] = "" if showing_placeholder else "".join(texts).strip()
    return fields

w:showingPlcHdr marks a control that still displays its placeholder text ("Click or tap here to enter text"). Treat those as empty, or every unanswered question comes back filled with instructions.

Step 4: Collect Text from Boxes, Headers and Footers

# pip install "python-docx>=1.1"
from docx import Document
from docx.oxml.ns import qn

def textbox_texts(doc) -> list[str]:
    boxes = []
    for box in doc.element.body.iter(qn("w:txbxContent")):
        paragraphs = ["".join(t.text or "" for t in p.iter(qn("w:t"))) for p in box.iter(qn("w:p"))]
        boxes.append("\n".join(p for p in paragraphs if p.strip()))
    return boxes

def header_footer_texts(doc) -> dict[str, list[str]]:
    out = {"header": [], "footer": []}
    for section in doc.sections:
        for kind, part in (("header", section.header), ("footer", section.footer)):
            if part.is_linked_to_previous:
                continue                                      # same as the previous section
            out[kind].extend(p.text for p in part.paragraphs if p.text.strip())
    return out

Documents saved by Word often contain each text box twice — once as a modern DrawingML shape and once as a VML fallback for old readers — so w:txbxContent can appear in pairs with identical text. De-duplicate by content when that matters. Missing text-box content is common enough to have its own page: fix python-docx missing text in text boxes.

Step 5: Decide About Tracked Changes and Comments

A document under review contains both the old and new versions of edited text. python-docx's paragraph.text includes inserted runs but not deleted text — which matches "all changes accepted" for insertions, but only because deleted text lives in w:delText elements that it does not read. Make the choice explicit, and read comments from their own part when reviewers' notes carry data:

# pip install lxml
import zipfile
from pathlib import Path
from lxml import etree

W = {"w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main"}

def comments(path: Path) -> list[dict]:
    with zipfile.ZipFile(path) as zf:
        if "word/comments.xml" not in zf.namelist():
            return []
        root = etree.fromstring(zf.read("word/comments.xml"))
    return [{
        "id": c.get(f"{{{W['w']}}}id"),
        "author": c.get(f"{{{W['w']}}}author"),
        "date": c.get(f"{{{W['w']}}}date"),
        "text": " ".join("".join(p.itertext()) for p in c.findall("w:p", W)).strip(),
    } for c in root.findall("w:comment", W)]

Pairing each comment with the text it annotates, and reading insertions and deletions with their authors, is covered in extract comments and tracked changes from docx.

Edge Cases and Variants

Legacy Form Fields

Older templates use legacy form fields (FORMTEXT, check boxes) rather than content controls. Their values sit in the runs between the field's separate and end markers, and their names in w:ffData/w:name. Read them by walking w:fldChar markers in order; the pattern mirrors content controls but requires tracking state across runs.

Numbered Lists

paragraph.text does not include list numbers — Word generates them at render time from numbering definitions. If "clause 4.2" matters to your extraction, reconstruct numbering from each paragraph's w:numPr level and count occurrences per list, or extract from a PDF rendering of the document where the numbers are real text.

Alternative: docx2python

For bulk text extraction where you do not need python-docx's object model, docx2python reads body, headers, footers, footnotes, endnotes and comments into nested lists in one call:

# pip install docx2python
from pathlib import Path
from docx2python import docx2python

def quick_extract(path: Path) -> dict:
    try:
        with docx2python(path) as content:
            return {"body": content.text, "footnotes": content.footnotes_runs,
                    "comments": content.comments}
    except (OSError, KeyError) as exc:
        raise RuntimeError(f"cannot read {path}: {exc}") from exc

It is convenient for search indexing. For structured extraction — named fields, table frames, document order — the explicit python-docx approach above gives more control.

Validation

Compare what you extracted with an independent count of every text node in the document. If the extracted character count is well below the document's total, a container was missed.

# pip install "python-docx>=1.1"
from pathlib import Path
from docx import Document
from docx.oxml.ns import qn

def all_text_chars(doc) -> int:
    body = doc.element.body
    visible = sum(len(t.text or "") for t in body.iter(qn("w:t")))
    return visible

def verify_coverage(path: Path, extracted: dict, min_ratio: float = 0.97) -> None:
    doc = Document(path)
    total = all_text_chars(doc)
    got = sum(len(str(v)) for v in extracted.get("fields", {}).values())
    got += sum(len(b) for b in extracted.get("blocks", []))
    got += sum(len(str(c)) for frame in extracted.get("tables", []) for c in frame.to_numpy().ravel())
    got += sum(len(t) for t in extracted.get("textboxes", []))
    ratio = got / total if total else 1.0
    assert ratio >= min_ratio, f"extracted {got} of {total} characters ({ratio:.0%}): something was skipped"
    print(f"{path.name}: {ratio:.0%} of body text captured")

Ratios above 100 percent are normal when merged cells are filled or text boxes are duplicated by the VML fallback; ratios well below point at an unhandled container. The first time a new template type arrives, also compare a handful of extracted fields against the document by eye — coverage proves nothing was dropped, not that each value landed under the right name.

Performance and Scale Notes

python-docx parses the whole document XML into memory; typical business documents open in tens of milliseconds, and even 300-page reports take well under a second. Extraction of thousands of files is CPU-bound on XML parsing, so a process pool with one file per task scales linearly with cores. Avoid calling Document(path) more than once per file — pass the loaded document to each extraction step. Large embedded images do not slow text extraction because image parts are not decoded, but they do inflate I/O; reading from a local disk rather than a network share makes a noticeable difference in batch jobs. Write results incrementally, one JSON line per document, so a malformed file late in the batch does not lose the earlier work.

Troubleshooting

Complete Working Script

#!/usr/bin/env python3
# pip install "python-docx>=1.1" "pandas>=2.2"
"""Extract fields, tables and text from every .docx in a folder into JSON Lines."""
import argparse
import json
import sys
from pathlib import Path

from docx import Document
from docx.oxml.ns import qn
from docx.table import Table
from docx.text.paragraph import Paragraph


def iter_blocks(doc):
    for child in doc.element.body.iterchildren():
        if child.tag == qn("w:sdt"):
            content = child.find(qn("w:sdtContent"))
            children = list(content.iterchildren()) if content is not None else []
        else:
            children = [child]
        for el in children:
            if el.tag == qn("w:p"):
                yield Paragraph(el, doc)
            elif el.tag == qn("w:tbl"):
                yield Table(el, doc)


def table_rows(table):
    rows = []
    for row in table.rows:
        rows.append(["\n".join(p.text for p in c.paragraphs).strip() for c in row.cells])
    return rows


def fields(doc):
    out = {}
    for sdt in doc.element.body.iter(qn("w:sdt")):
        pr = sdt.find(qn("w:sdtPr"))
        tag = pr.find(qn("w:tag")) if pr is not None else None
        if tag is None:
            continue
        placeholder = pr.find(qn("w:showingPlcHdr")) is not None
        out[tag.get(qn("w:val"))] = "" if placeholder else "".join(t.text or "" for t in sdt.iter(qn("w:t"))).strip()
    return out


def extract(path: Path) -> dict:
    doc = Document(path)
    blocks, tables, heading = [], [], ""
    for block in iter_blocks(doc):
        if isinstance(block, Paragraph):
            if block.style.name.startswith("Heading"):
                heading = block.text.strip()
            if block.text.strip():
                blocks.append({"heading": heading, "text": block.text.strip()})
        else:
            tables.append({"heading": heading, "rows": table_rows(block)})
    boxes = ["".join(t.text or "" for t in box.iter(qn("w:t")))
             for box in doc.element.body.iter(qn("w:txbxContent"))]
    return {"file": path.name, "fields": fields(doc), "blocks": blocks,
            "tables": tables, "textboxes": sorted(set(b for b in boxes if b.strip()))}


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("src", type=Path)
    ap.add_argument("out", type=Path)
    args = ap.parse_args()
    args.out.parent.mkdir(parents=True, exist_ok=True)
    failures = 0
    with args.out.open("w", encoding="utf-8") as fh:
        for path in sorted(args.src.glob("*.docx")):
            if path.name.startswith("~$"):
                continue                                  # Word lock files
            try:
                fh.write(json.dumps(extract(path), ensure_ascii=False) + "\n")
            except Exception as exc:
                failures += 1
                print(f"{path.name}: {exc}", file=sys.stderr)
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Frequently Asked Questions

Can python-docx read .doc files? No. Convert them to .docx with LibreOffice headless (soffice --headless --convert-to docx) first, then extract.

How do I get checkbox values from a form? Modern checkbox content controls store their state in w14:checkbox/w14:checked inside w:sdtPr; read the w14:val attribute. Legacy checkboxes store it in w:ffData/w:checkBox/w:default or w:checked.

Why is ~$report.docx in my folder? It is Word's lock file for an open document. Skip names starting with ~$; they are not valid packages.

How do I extract hyperlinks and their targets? Link text is in the runs inside a w:hyperlink element, and the target URL is stored in the part's relationships under the element's r:id. Look it up with doc.part.rels[rid].target_ref. Internal links to bookmarks use a w:anchor attribute instead of a relationship id, so check for both.

Is it faster to convert to PDF and extract from that? Rarely, and it loses structure — tables become positioned text. Extract from the .docx directly and use PDF extraction only for documents that exist solely as PDFs.

Related

• Extract Tables from Word Documents to pandas — merged, nested and multi-row header tables
• Fix python-docx Missing Text in Text Boxes — shapes, VML fallbacks and content controls
• Extract Comments and Tracked Changes from docx — reviewer data and revision history
• Find and Replace Text in Word Documents — the same document model, used for editing
• Extracting Text and Metadata from PDFs — when the only copy is a PDF

Part of Word Document Templating & Batch Processing.