Python Doc & Data Automation

Extract Tables from Word Documents to pandas

Forty supplier questionnaires each contain a pricing table, and the goal is one DataFrame with every supplier's prices. The first script — pd.DataFrame([[c.text for c in r.cells] for r in table.rows]) — produces frames with duplicated values across columns, a header row that is really two rows, prices stored as strings like £1,240.00, and for some suppliers IndexError: list index out of range or a table that is simply missing because it sits inside another table's cell.

Root Cause

Word tables are layout objects, not data grids, and python-docx reports them faithfully as layout. A cell merged horizontally across columns occupies several grid positions, and row.cells returns the same cell once per position, so its text repeats. A vertically merged cell is stored as a starting cell followed by continuation cells whose text is empty, so values appear only in the first row of the merge. Header rows are frequently two rows tall with merged group labels on top. Rows can have different numbers of cells when cells were split or deleted in Word, which breaks code that assumes a rectangle. Tables placed inside a cell of an outer table are invisible to doc.tables, which lists only top-level tables. And every cell value is text, including numbers formatted with currency symbols and thousands separators. Converting to a DataFrame reliably means handling each of these explicitly.

Minimal Diagnostic

Report the structural features of each table before converting: grid size, per-row cell counts, merges and nested tables.

# pip install "python-docx>=1.1"
from pathlib import Path
from docx import Document
from docx.oxml.ns import qn

SOURCE = Path("in/questionnaires/northwind.docx")

def table_report(path: Path) -> None:
    try:
        doc = Document(path)
    except Exception as exc:
        raise SystemExit(f"cannot open {path}: {exc}")
    all_tables = doc.element.body.xpath(".//w:tbl")
    print(f"{len(doc.tables)} top-level table(s), {len(all_tables)} including nested")
    for i, table in enumerate(doc.tables):
        widths = [len(row.cells) for row in table.rows]
        h_merges = sum(1 for tc in table._tbl.iter(qn("w:tc"))
                       if tc.find(qn("w:tcPr")) is not None
                       and tc.find(qn("w:tcPr")).find(qn("w:gridSpan")) is not None)
        v_merges = len(table._tbl.xpath(".//w:vMerge"))
        nested = len(table._tbl.xpath(".//w:tc//w:tbl"))
        print(f"table {i}: {len(table.rows)} rows, grid {len(table.columns)} cols, "
              f"cells per row {sorted(set(widths))}, gridSpan {h_merges}, vMerge {v_merges}, nested {nested}")

if __name__ == "__main__":
    table_report(SOURCE)

3 top-level table(s), 4 including nested
table 0: 6 rows, grid 2 cols, cells per row [2], gridSpan 0, vMerge 0, nested 0
table 1: 14 rows, grid 5 cols, cells per row [5], gridSpan 3, vMerge 6, nested 0
table 2: 3 rows, grid 2 cols, cells per row [2], gridSpan 0, vMerge 0, nested 1

Table 1 — the pricing table — has horizontal merges (the header group labels) and vertical merges (a category column spanning several product rows). Table 2 hides a fourth table inside a cell.

Raw row.cells output versus a clean frame The left panel shows the raw extraction, where the merged Price header repeats across three columns, the second header row is treated as data, and the Category column is blank on rows continuing a vertical merge. The right panel shows the cleaned frame with combined header labels such as Price Unit and Price Volume, category values filled down, and prices converted to numbers. [c.text for c in row.cells] Category|Item |Price|Price|Price | |Unit |Vol |Lead Paper |A4 |£4.10|£3.80|2d |A3 |£7.90|£7.20|2d Toner |K-10 |£62 |£58 |5d Clean DataFrame category item price_unit price_vol Paper A4 4.10 3.80 Paper A3 7.90 7.20 Toner K-10 62.00 58.00 dtypes: float64 for prices

Fix: Normalise the Grid, Headers, Merges and Values

Build a rectangular grid from the XML so each grid column is one entry, fill vertical merges downward, join multi-row headers, and convert numeric text. Changed lines carry comments.

# pip install "python-docx>=1.1" "pandas>=2.2"
import re
from pathlib import Path
import pandas as pd
from docx import Document
from docx.oxml.ns import qn
from docx.table import Table

def cell_text(tc) -> str:
    paragraphs = ["".join(t.text or "" for t in p.iter(qn("w:t"))) for p in tc.iter(qn("w:p"))]
    return "\n".join(p for p in paragraphs if p.strip()).strip()

def grid(table: Table) -> list[list[str]]:
    """Rectangular grid: one entry per grid column, merges resolved."""
    n_cols = len(table.columns)
    rows, above = [], [None] * n_cols
    for tr in table._tbl.iter(qn("w:tr")):
        if tr.getparent() is not table._tbl:
            continue                                                  # changed: skip nested tables' rows
        row, col = [], 0
        for tc in tr.iterchildren(qn("w:tc")):
            pr = tc.find(qn("w:tcPr"))
            span_el = pr.find(qn("w:gridSpan")) if pr is not None else None
            span = int(span_el.get(qn("w:val"))) if span_el is not None else 1   # changed: honour gridSpan
            vmerge = pr.find(qn("w:vMerge")) if pr is not None else None
            continues = vmerge is not None and vmerge.get(qn("w:val")) in (None, "continue")
            text = above[col] if continues and above[col] is not None else cell_text(tc)  # changed: fill down
            for _ in range(span):
                if col < n_cols:
                    row.append(text)
                    above[col] = text
                    col += 1
        row.extend([""] * (n_cols - len(row)))                        # changed: pad short rows
        rows.append(row[:n_cols])
    return rows

MONEY = re.compile(r"^[£$€]?\s*-?[\d,]+(\.\d+)?$")

def to_frame(table: Table, header_rows: int = 1) -> pd.DataFrame:
    rows = grid(table)
    header = []
    for i, parts in enumerate(zip(*rows[:header_rows])):
        unique = list(dict.fromkeys(p for p in parts if p))            # changed: drop repeated group labels
        label = "_".join(unique).lower()
        header.append(re.sub(r"\W+", "_", label).strip("_") or f"col_{i}")
    frame = pd.DataFrame(rows[header_rows:], columns=header)
    for col in frame.columns:
        values = frame[col].str.strip()
        if values.replace("", pd.NA).dropna().str.match(MONEY).all():  # changed: numeric-looking column
            frame[col] = pd.to_numeric(values.str.replace(r"[£$€,\s]", "", regex=True), errors="coerce")
    return frame

if __name__ == "__main__":
    doc = Document(Path("in/questionnaires/northwind.docx"))
    prices = to_frame(doc.tables[1], header_rows=2)
    print(prices.head())
    print(prices.dtypes)

Reading w:gridSpan and w:vMerge directly avoids the repeated-cell behaviour of row.cells altogether: each w:tc element is visited once and spread across the grid columns it spans. A vertical merge whose w:val is absent or continue is a continuation, so it takes the value from the row above in the same grid column — for a data table, the category belongs to every row it spans.

Numeric conversion only applies when every non-blank value in the column looks like a number, which stops a column mixing 2d lead times and numbers from being coerced into NaNs.

Resolving merges into a rectangular grid The top header row contains a Price cell spanning three grid columns through gridSpan 3, which becomes the prefix for the three price column names. The second header row holds the sub-labels Unit, Volume and Lead time. In the body, a Category cell starts a vertical merge with vMerge restart and the following rows carry vMerge continue, which are filled with the value Paper. Rows with fewer cells are padded to the grid width. Pricing table, 5 grid columns 1 1 gridSpan=3: Price prefix for three columns 2 2 Second header row Unit, Volume, Lead time 3 3 vMerge: Paper filled down to each row 4 4 Short row padded to 5 cells

Variant Fix 1: Tables Nested Inside Cells

Layout templates often place a small table inside a cell of a larger one. Recurse into cells to find them, and record where each nested table was found so it can be labelled:

# pip install "python-docx>=1.1"
from docx import Document
from docx.table import Table

def all_tables(container, path: str = "body") -> list[tuple[str, Table]]:
    found = []
    for i, table in enumerate(container.tables):
        here = f"{path}/table{i}"
        found.append((here, table))
        for r, row in enumerate(table.rows):
            seen = set()
            for c, cell in enumerate(row.cells):
                if id(cell._tc) in seen:
                    continue                                   # merged cell already visited
                seen.add(id(cell._tc))
                found.extend(all_tables(cell, f"{here}/r{r}c{c}"))
    return found

if __name__ == "__main__":
    doc = Document("in/questionnaires/northwind.docx")
    for location, table in all_tables(doc):
        print(location, len(table.rows), "rows")

_Cell.tables returns tables directly inside a cell, so the recursion mirrors the document's nesting. The grid function above ignores rows of nested tables when converting the outer table, so each table is converted exactly once.

Variant Fix 2: Finding the Right Table by Its Header

Documents from different suppliers put the pricing table at different positions. Select tables by header content instead of index, which survives an extra introductory table being added to the template:

# pip install "python-docx>=1.1" "pandas>=2.2"
from docx import Document

def find_table(doc, required: set[str], header_rows: int = 2):
    for location, table in all_tables(doc):
        rows = grid(table)
        header_text = " ".join(" ".join(r) for r in rows[:header_rows]).lower()
        if all(word in header_text for word in required):
            return location, to_frame(table, header_rows=header_rows)
    return None, None

location, prices = find_table(Document("in/questionnaires/contoso.docx"), {"price", "item", "unit"})
if prices is None:
    raise SystemExit("no pricing table found: template changed?")

Failing loudly when no table matches is better than falling back to doc.tables[1]: a silently wrong table produces plausible-looking numbers.

Combining tables from many documents For each document in the folder, all tables including nested ones are listed. The pricing table is selected by required header words rather than position. It is converted to a frame with merges and numbers handled, tagged with the supplier file name, and appended. Documents without a matching table go to an exceptions list. Finally all frames are concatenated with aligned columns. List tables incl. nested Match header required words to_frame merges and numbers Tag source supplier file name Concatenate aligned columns No match exceptions list

Combining Tables from Many Documents

With selection and conversion in place, the batch is a loop that tags each frame with its source and records documents that did not yield a table:

# pip install "python-docx>=1.1" "pandas>=2.2"
from pathlib import Path
import pandas as pd
from docx import Document

def collect_prices(folder: Path) -> tuple[pd.DataFrame, list[str]]:
    frames, problems = [], []
    for path in sorted(folder.glob("*.docx")):
        if path.name.startswith("~$"):
            continue
        try:
            location, frame = find_table(Document(path), {"price", "item", "unit"})
        except Exception as exc:
            problems.append(f"{path.name}: {exc}")
            continue
        if frame is None:
            problems.append(f"{path.name}: no pricing table")
            continue
        frames.append(frame.assign(supplier=path.stem, source_table=location))
    combined = pd.concat(frames, ignore_index=True, sort=False) if frames else pd.DataFrame()
    return combined, problems

sort=False keeps column order from the first frame; suppliers who renamed a header produce extra columns filled with NaN, which is visible and easy to fix with a rename map — the same alignment problem covered for spreadsheets in fix pandas concat columns misaligned.

Verification

Check each converted frame against its source table: row count, column count, and a sum over numeric columns compared with a sum parsed straight from the XML text.

# pip install "python-docx>=1.1" "pandas>=2.2"
import re
import pandas as pd
from docx.table import Table

def verify_frame(table: Table, frame: pd.DataFrame, header_rows: int) -> None:
    body_rows = len([tr for tr in table._tbl.iterchildren() if tr.tag.endswith("}tr")]) - header_rows
    assert len(frame) == body_rows, f"{len(frame)} rows, table has {body_rows}"
    assert len(frame.columns) == len(table.columns), "column count differs from the grid"
    assert not frame.columns.duplicated().any(), f"duplicate headers {list(frame.columns)}"
    for col in frame.select_dtypes("number").columns:
        assert frame[col].notna().any(), f"{col}: every value failed numeric conversion"
    print(f"frame {frame.shape} matches table structure")

Add a spot check for the first document of each new template: print the frame next to a screenshot or PDF render of the table and compare three or four values by eye. Structure checks catch dropped rows; only a human glance catches a header joined in the wrong order.

FAQ

Can pandas read Word tables directly like read_html? No. Converting the .docx to HTML with LibreOffice and using pd.read_html works for simple tables but handles merges inconsistently. The python-docx approach keeps control.

Why is len(table.columns) wrong for some tables? It reads the table's grid definition, which Word does not always update after manual edits. If rows have more cells than the grid, use the maximum per-row span count instead.

How do I keep line breaks inside cells?cell_text joins paragraphs with newlines. Replace them with spaces for analysis, or keep them for exporting back to Excel where wrapped text is fine.

What about tables inside text boxes? They are in w:txbxContent, outside the body's table list. Find them with XPath as shown in fix python-docx missing text in text boxes and wrap each w:tbl element in Table(element, doc).

Part of Extracting Data from Word Documents with Python.

/html>