Python Doc & Data Automation

Fix python-docx Missing Text in Text Boxes

The contract clearly shows the client name and reference in a box at the top of page one, and the extraction output has neither:

>>> from docx import Document
>>> doc = Document("in/contract-2291.docx")
>>> any("Contract ref" in p.text for p in doc.paragraphs)
False
>>> "\n".join(p.text for p in doc.paragraphs)[:80]
'Terms and Conditions\n\n1. Scope of services\n...'

The same happens with callout shapes, sidebar boxes in reports, labels inside grouped drawings, text in SmartArt, and form answers inside content controls. No error is raised; the text simply is not there.

Root Cause

Document.paragraphs returns only the w:p elements that are direct children of the document body. A text box is a drawing object anchored inside a run of some paragraph: its own paragraphs sit several levels down, inside w:drawing, a DrawingML shape, and finally w:txbxContent. They are real paragraphs in the WordprocessingML namespace, but they are not children of the body, so python-docx's list does not include them. Content controls wrap paragraphs in w:sdt elements, which again moves them out of the body's direct children. SmartArt is stored in a separate diagram data part, not in the document XML at all. Word also writes most text boxes twice for compatibility — once as DrawingML and once as a VML v:textbox fallback — so the obvious XPath fix can return every text box's content twice.

Minimal Diagnostic

Search the raw XML for the missing string and print the chain of element names above it. The ancestor chain names the container that hides it.

# pip install "python-docx>=1.1"
from pathlib import Path
from docx import Document
from docx.oxml.ns import qn

SOURCE = Path("in/contract-2291.docx")
NEEDLE = "Contract ref"

def locate(path: Path, needle: str) -> None:
    try:
        doc = Document(path)
    except Exception as exc:
        raise SystemExit(f"cannot open {path}: {exc}")
    found = False
    for t in doc.element.body.iter(qn("w:t")):
        if needle.lower() in (t.text or "").lower():
            found = True
            chain, el = [], t
            while el is not None and el is not doc.element.body:
                chain.append(el.tag.split("}")[-1])
                el = el.getparent()
            print(" < ".join(chain))
    if not found:
        print("not in document.xml body: check headers, footers, SmartArt or footnotes parts")

if __name__ == "__main__":
    locate(SOURCE, NEEDLE)

t < r < p < txbxContent < txbx < wsp < graphicData < graphic < anchor < drawing < r < p
t < r < p < txbxContent < textbox < shape < pict < Fallback < AlternateContent < r < p

Two hits for one visible box: the DrawingML text box (wsp shape) and the VML fallback (pict/shape/textbox) inside mc:AlternateContent. Both sit below a body paragraph's run, which is why doc.paragraphs shows only that host paragraph, usually empty.

How deep a text box paragraph sits A normal body paragraph is a direct child of w:body and appears in doc.paragraphs. A text box paragraph sits below a host paragraph, a run, w:drawing, wp:anchor, a:graphic, the wps:wsp shape, wps:txbx and w:txbxContent, so it is not listed. The same content usually appears again under mc:AlternateContent Fallback as a VML v:textbox. w:body > w:p listed by doc.paragraphs — usually the empty host paragraph w:r > w:drawing > wp:anchor the shape is anchored to a run in the host paragraph a:graphic > wps:wsp > wps:txbx DrawingML text box shape w:txbxContent > w:p the paragraphs you need, invisible to doc.paragraphs mc:Fallback > v:textbox VML copy of the same text for old readers

Fix: Query Text Boxes and Skip the VML Fallback

Find w:txbxContent elements, ignore those inside an mc:Fallback branch, and read their paragraphs. Changed lines carry comments.

# pip install "python-docx>=1.1"
from pathlib import Path
from docx import Document
from docx.oxml.ns import qn
from docx.text.paragraph import Paragraph

MC_FALLBACK = "{http://schemas.openxmlformats.org/markup-compatibility/2006}Fallback"

def in_fallback(element) -> bool:
    parent = element.getparent()
    while parent is not None:
        if parent.tag == MC_FALLBACK:                               # changed: VML duplicate branch
            return True
        parent = parent.getparent()
    return False

def textbox_paragraphs(doc) -> list[list[Paragraph]]:
    boxes = []
    for box in doc.element.body.iter(qn("w:txbxContent")):         # changed: search all depths
        if in_fallback(box):
            continue                                                # changed: skip the VML copy
        boxes.append([Paragraph(p, doc) for p in box.iterchildren(qn("w:p"))])  # changed: real Paragraphs
    return boxes

if __name__ == "__main__":
    try:
        doc = Document(Path("in/contract-2291.docx"))
    except Exception as exc:
        raise SystemExit(f"cannot open document: {exc}")
    for i, paragraphs in enumerate(textbox_paragraphs(doc)):
        text = "\n".join(p.text for p in paragraphs if p.text.strip())
        print(f"box {i}: {text!r}")

box 0: 'Client: Northwind Traders Ltd\nContract ref: C-2291'

Wrapping the elements as python-docx Paragraph objects keeps access to styles and runs, so the same formatting-aware code you use on body paragraphs works on text box content. Documents created by older tools may contain only VML text boxes, with no DrawingML version; in that case the fallback filter removes everything. Handle it by preferring DrawingML and falling back to VML per shape:

# pip install "python-docx>=1.1"
from docx.oxml.ns import qn

def textbox_texts_any(doc) -> list[str]:
    modern = [b for b in doc.element.body.iter(qn("w:txbxContent")) if not in_fallback(b)]
    chosen = modern if modern else list(doc.element.body.iter(qn("w:txbxContent")))
    return ["\n".join("".join(t.text or "" for t in p.iter(qn("w:t"))) for p in box.iter(qn("w:p")))
            for box in chosen]

Variant Fix 1: Content Controls Hide Paragraphs Too

When the missing text is a form answer, it is usually inside a content control rather than a shape. The ancestor chain from the diagnostic shows sdtContent < sdt. Read controls by their tag to get named values:

# pip install "python-docx>=1.1"
from docx import Document
from docx.oxml.ns import qn

def sdt_values(doc) -> dict[str, str]:
    values = {}
    for sdt in doc.element.body.iter(qn("w:sdt")):
        pr = sdt.find(qn("w:sdtPr"))
        if pr is None:
            continue
        tag = pr.find(qn("w:tag"))
        name = tag.get(qn("w:val")) if tag is not None else None
        if not name:
            continue
        if pr.find(qn("w:showingPlcHdr")) is not None:
            values[name] = ""                                       # placeholder, not an answer
            continue
        content = sdt.find(qn("w:sdtContent"))
        values[name] = "".join(t.text or "" for t in content.iter(qn("w:t"))).strip() if content is not None else ""
    return values

Content controls can also sit inside text boxes, so run this over the whole body (as above) rather than only over top-level paragraphs.

Variant Fix 2: Grouped Shapes, SmartArt and Charts

Text in grouped shapes is still w:txbxContent, just deeper, and the fix above finds it. SmartArt and chart labels are different: their text lives in separate package parts (word/diagrams/data1.xml, word/charts/chart1.xml) using DrawingML's a:t elements. Read those parts from the ZIP when you need them:

# pip install lxml
import zipfile
from pathlib import Path
from lxml import etree

A_T = "{http://schemas.openxmlformats.org/drawingml/2006/main}t"

def smartart_and_chart_text(path: Path) -> dict[str, list[str]]:
    out: dict[str, list[str]] = {}
    with zipfile.ZipFile(path) as zf:
        for name in zf.namelist():
            if name.startswith(("word/diagrams/data", "word/charts/chart")) and name.endswith(".xml"):
                root = etree.fromstring(zf.read(name))
                texts = [el.text for el in root.iter(A_T) if el.text and el.text.strip()]
                if texts:
                    out[name] = texts
    return out

Chart parts also contain cached category labels and series names in c:v elements; if you need chart data, read those instead of the a:t titles.

Where hidden text lives and how to reach it Text boxes and callout shapes keep text in w:txbxContent inside the document part, reached with XPath while skipping mc:Fallback. Content controls keep text in w:sdtContent in the document part, read by tag. Grouped shapes also use w:txbxContent at a deeper level. SmartArt keeps text as a:t in word/diagrams/data parts, read from the ZIP. Chart titles and labels keep text in word/charts parts, read from the ZIP. Headers and footers keep text in their own parts, reached through section.header and section.footer. Content Where the text is How to read it Text box / callout w:txbxContent XPath; skip mc:Fallback Content control w:sdtContent iterate w:sdt by tag Grouped shapes deeper w:txbxContent same XPath SmartArt word/diagrams/data*.xml zipfile + a:t Charts word/charts/chart*.xml zipfile + a:t / c:v Header / footer header*.xml parts section.header

Keeping Text Box Content in Reading Order

Appending all text boxes at the end of the extracted text loses context: a sidebar about payment terms ends up after the signature block. Emit each box's text right after the body paragraph that anchors it, which approximates where a reader sees it:

# pip install "python-docx>=1.1"
from docx import Document
from docx.oxml.ns import qn

def text_in_reading_order(doc) -> list[str]:
    lines = []
    for p in doc.element.body.iter(qn("w:p")):
        if p.getparent() is not doc.element.body:
            continue                                            # only top-level body paragraphs here
        own = "".join(t.text or "" for r in p.iterchildren(qn("w:r")) for t in r.iterchildren(qn("w:t")))
        if own.strip():
            lines.append(own)
        for box in p.iter(qn("w:txbxContent")):
            if in_fallback(box):
                continue
            for inner in box.iter(qn("w:p")):
                text = "".join(t.text or "" for t in inner.iter(qn("w:t")))
                if text.strip():
                    lines.append(f"[box] {text}")
    return lines
Text boxes appended at the end versus after their anchor The left panel shows extraction that appends text boxes at the end, so the client and contract reference box appears after the signature block and the payment terms sidebar appears far from clause 4. The right panel shows extraction in anchor order, where the client box follows the title and the sidebar follows clause 4, matching what a reader sees on the page. Boxes appended at the end Terms and Conditions 1. Scope of services 4. Payment Signed: ________ [box] Client: Northwind [box] Late fee: 2% per month Boxes after their anchor Terms and Conditions [box] Client: Northwind 1. Scope of services 4. Payment [box] Late fee: 2% per month Signed: ________

When a downstream parser looks for "the reference near the title" or "the fee mentioned under payment", order decides whether it finds the right value. Anchoring is a layout hint, not a guarantee — Word lets a box anchored in paragraph 3 float to the top of the page — but for most templates the anchor paragraph is on the same page and near the box. Marking box lines with a prefix keeps them distinguishable for downstream parsing, and for search indexing you can drop the prefix.

Verification

Compare the total text in every w:t element (excluding VML fallbacks) with the text your extraction produced. Anything substantially lower means a container is still missing.

# pip install "python-docx>=1.1"
from pathlib import Path
from docx import Document
from docx.oxml.ns import qn

def verify_all_text_captured(path: Path, extracted_lines: list[str]) -> None:
    doc = Document(path)
    expected = "".join(t.text or "" for t in doc.element.body.iter(qn("w:t")) if not in_fallback(t))
    got = "".join(line.removeprefix("[box] ") for line in extracted_lines)
    squash = lambda s: "".join(s.split())
    missing = len(squash(expected)) - len(squash(got))
    assert missing <= 0.02 * len(squash(expected)), (
        f"{missing} characters not captured; run the locate() diagnostic on a missing phrase")
    print(f"{path.name}: text boxes and body captured ({len(squash(got))} chars)")

if __name__ == "__main__":
    doc = Document("in/contract-2291.docx")
    verify_all_text_captured(Path("in/contract-2291.docx"), text_in_reading_order(doc))

Removing whitespace before comparing avoids false alarms from line breaks and joins. The two percent allowance covers text inside content controls that you deliberately excluded, such as placeholder instructions; tighten it once your extraction includes every container you care about.

FAQ

Will a newer python-docx read text boxes automatically? Not at the time of writing. Shape support in python-docx focuses on inline pictures. The XPath approach is stable because it relies on the file format, not the library's API.

Why does doc.inline_shapes not list my text box? Most text boxes are floating (wp:anchor), and inline_shapes lists only inline ones (wp:inline), and only pictures at that.

Can I edit text inside a text box the same way? Yes. Once wrapped as Paragraph objects, runs can be modified and saved. Replacing text that spans runs has the same pitfalls as body text — see fix python-docx replace text split across runs.

What about text boxes in headers? They live in the header part. Run the same queries on section.header._element instead of the body.

Part of Extracting Data from Word Documents with Python.