Python Doc & Data Automation

Fix Encoded Attachment Filenames in Python

Saved attachments end up with names like these instead of Rechnung Nr 45.pdf or Factura – Septiembre.xlsx:

=?UTF-8?B?UmVjaG51bmcgTnIgNDUucGRm?=
=?iso-8859-1?Q?Factura_=96_Septiembre.xlsx?=
Factura – Septiembre.xlsx
None

Some attachments save as None, others overwrite each other because every name decodes to the same fallback, and a few names crash the job with UnicodeEncodeError when written to disk on a server with an ASCII locale.

Root Cause

Email headers were designed for 7-bit ASCII, so non-ASCII filenames must be encoded, and two incompatible standards are in use. RFC 2047 encoded-words wrap text as =?charset?B?base64?= or =?charset?Q?quoted-printable?=, and many mail clients use them in the name parameter of Content-Type even though the standard does not allow encoded-words inside parameters. RFC 2231 defines the correct parameter encoding — filename*=UTF-8''Rechnung%20Nr%2045.pdf — and allows long names to be split across filename*0*, filename*1* continuations. Python's email package decodes both, but only with the modern policy.default; email.message_from_bytes without a policy uses the legacy compat32 policy, which returns parameter values raw. Some senders then add their own mistakes: a Q-encoded name declared as ISO-8859-1 but containing Windows-1252 characters (the =96 en dash above), UTF-8 bytes sent without any encoding (producing mojibake like –), or no filename at all.

Minimal Diagnostic

Print the raw header parameters and the value each API returns for every attachment in a problem message. The comparison shows which encoding the sender used and whether the policy is the cause.

# stdlib only
import email
from email import policy
from pathlib import Path

RAW = Path("samples/problem-message.eml")          # save the message source from the mail client

def diagnose(path: Path) -> None:
    data = path.read_bytes()
    legacy = email.message_from_bytes(data)                        # compat32 policy
    modern = email.message_from_bytes(data, policy=policy.default)
    legacy_parts = [p for p in legacy.walk() if p.get_content_disposition() in ("attachment", "inline")]
    modern_parts = list(modern.iter_attachments())
    for lp, mp in zip(legacy_parts, modern_parts):
        print("Content-Type raw:       ", lp.get("Content-Type"))
        print("Content-Disposition raw:", lp.get("Content-Disposition"))
        print("legacy get_filename():  ", repr(lp.get_filename()))
        print("modern get_filename():  ", repr(mp.get_filename()))
        print("-" * 60)

if __name__ == "__main__":
    diagnose(RAW)

Content-Type raw:        application/pdf; name="=?UTF-8?B?UmVjaG51bmcgTnIgNDUucGRm?="
Content-Disposition raw: attachment; filename*=UTF-8''Rechnung%20Nr%2045.pdf
legacy get_filename():   'Rechnung Nr 45.pdf'
modern get_filename():   'Rechnung Nr 45.pdf'
------------------------------------------------------------
Content-Type raw:        application/vnd.openxmlformats-officedocument.spreadsheetml.sheet; name="=?iso-8859-1?Q?Factura_=96_Septiembre.xlsx?="
Content-Disposition raw: attachment
legacy get_filename():   '=?iso-8859-1?Q?Factura_=96_Septiembre.xlsx?='
modern get_filename():   'Factura \x96 Septiembre.xlsx'

The first attachment has a proper RFC 2231 filename that both policies decode. The second has only an RFC 2047 name in Content-Type: the legacy policy returns it raw, and the modern policy decodes it — but as ISO-8859-1, where byte 0x96 is an invisible control character rather than the en dash the sender meant.

Two filename encodings on the wire The left panel shows the RFC 2231 form in Content-Disposition, filename star equals UTF-8 two quotes then percent-encoded text, including continuation parameters for long names, which both legacy and modern policies decode. The right panel shows an RFC 2047 encoded-word inside the Content-Type name parameter, which is common but non-standard; the legacy policy returns it raw and the modern policy decodes it, subject to the declared charset being correct. RFC 2231 (standard) Content-Disposition: attachment; filename*=UTF-8''Rechnung%20Nr%2045.pdf long names: filename*0*=UTF-8''Very%20long%20 filename*1*=name.pdf RFC 2047 in name= (common) Content-Type: application/pdf; name="=?UTF-8?B?UmVjaG51bmc...?=" legacy policy: returned raw modern policy: decoded wrong charset: mojibake

Which Senders Produce Which Form

The encoding you receive depends on the software that sent the message, which explains why one supplier's attachments always break and another's never do. A small sample of real messages from your own mailbox is the best guide, but the common patterns are stable:

Filename encodings by sending software Modern desktop and webmail clients usually send an RFC 2231 filename star parameter, sometimes together with an RFC 2047 name parameter, and both decode with policy.default. Multifunction scanners and copiers often send RFC 2047 encoded-words only, which decode with policy.default. Legacy ERP and billing systems frequently declare ISO-8859-1 while sending Windows-1252 characters, which need charset repair. Some scripts and old systems send raw UTF-8 bytes without any encoding, which produce mojibake that needs repair, or omit the filename entirely, which needs a fallback name. Sender type Typical form With policy.default Desktop / webmail RFC 2231 filename* decodes Scanners RFC 2047 in name= decodes Legacy ERP latin-1 label on cp1252 needs charset repair Home-made scripts raw UTF-8 or no name repair or fallback

Because the problem follows the sender, log the raw Content-Disposition header the first time each new sender domain appears. When a name breaks later, that one log line shows immediately whether the sender changed software or your code regressed.

Fix: Parse with the Modern Policy and Repair Misdeclared Charsets

Use policy.default everywhere, fall back to decoding encoded-words manually if a raw one survives, repair the ISO-8859-1/Windows-1252 mix-up, and always produce a safe, unique on-disk name. Changed lines are commented.

# stdlib only
import email
import hashlib
import re
import unicodedata
from email import policy
from email.header import decode_header, make_header
from email.message import EmailMessage
from pathlib import Path

ENCODED_WORD = re.compile(r"=\?[\w\-]+\?[bBqQ]\?.*?\?=")
C1_CONTROLS = re.compile("[\x80-\x9f]")

def decode_filename(part: EmailMessage) -> str | None:
    name = part.get_filename()                                     # changed: modern policy decodes both RFCs
    if name and ENCODED_WORD.search(name):
        name = str(make_header(decode_header(name)))               # changed: leftover encoded-words
    if name and C1_CONTROLS.search(name):
        try:
            name = name.encode("latin-1").decode("cp1252")         # changed: 0x96 etc. were cp1252
        except (UnicodeEncodeError, UnicodeDecodeError):
            pass
    if name and "â€" in name or (name and "Ã" in name):
        try:
            name = name.encode("cp1252").decode("utf-8")           # changed: undo UTF-8 read as cp1252
        except (UnicodeEncodeError, UnicodeDecodeError):
            pass
    return unicodedata.normalize("NFC", name) if name else None     # changed: one form for accents

def disk_name(part: EmailMessage, payload: bytes, index: int) -> str:
    name = decode_filename(part)
    if not name:                                                   # changed: never save as "None"
        ext = {"application/pdf": ".pdf", "text/csv": ".csv"}.get(part.get_content_type(), ".bin")
        name = f"attachment-{index}{ext}"
    base = Path(name.replace("\\", "/")).name
    stem = re.sub(r"[^\w.\- ]+", "_", Path(base).stem).strip(" ._")[:80] or "attachment"
    digest = hashlib.sha256(payload).hexdigest()[:8]
    return f"{stem}_{digest}{Path(base).suffix.lower()}"          # changed: unique and filesystem-safe

def save_all(raw_message: bytes, out_dir: Path) -> list[Path]:
    msg = email.message_from_bytes(raw_message, policy=policy.default)   # changed: modern policy
    out_dir.mkdir(parents=True, exist_ok=True)
    saved = []
    for i, part in enumerate(msg.iter_attachments(), start=1):
        payload = part.get_payload(decode=True) or b""
        if not payload:
            continue
        dest = out_dir / disk_name(part, payload, i)
        dest.write_bytes(payload)
        saved.append(dest)
    return saved

if __name__ == "__main__":
    try:
        for path in save_all(Path("samples/problem-message.eml").read_bytes(), Path("out/attachments")):
            print(path.name)
    except OSError as exc:
        raise SystemExit(f"cannot save attachments: {exc}")

Rechnung Nr 45_3fa91c2e.pdf
Factura – Septiembre_8b10d4a7.xlsx

The \w class in Python's re is Unicode-aware, so accented letters and non-Latin scripts survive sanitising while path separators, control characters and shell metacharacters do not. Normalising to NFC matters on macOS-originated mail, where é may arrive as e plus a combining accent — visually identical, but a different filename to your file system and to any later lookup.

Variant Fix 1: Messages Already Parsed with the Legacy Policy

Code that received Message objects from elsewhere — an older library, a pickled queue item — cannot re-parse with a new policy easily. Decode the filename parameters by hand with email.utils.collapse_rfc2231_value:

# stdlib only
from email.header import decode_header, make_header
from email.message import Message
from email.utils import collapse_rfc2231_value

def legacy_filename(part: Message) -> str | None:
    raw = part.get_param("filename", header="content-disposition") or part.get_param("name")
    if raw is None:
        return None
    value = collapse_rfc2231_value(raw)                    # handles filename*= and continuations
    if "=?" in value:
        value = str(make_header(decode_header(value)))     # handles encoded-words inside quotes
    return value

get_param returns a tuple for RFC 2231 values and a string otherwise; collapse_rfc2231_value accepts both. Prefer converting the pipeline to policy.default when you can — it also decodes subjects and addresses, which the legacy path leaves to you.

Filename repair steps in order First parse with policy.default so RFC 2231 and most RFC 2047 names decode. Second decode any encoded-words still present. Third repair misdeclared charsets such as Windows-1252 characters labelled ISO-8859-1, and UTF-8 bytes read as Windows-1252. Fourth normalise to Unicode NFC. Fifth fall back to a generated name when no filename exists. Sixth strip directories and unsafe characters and append a content hash for uniqueness. policy.default decodes RFC 2231 filename* and most RFC 2047 name= values Leftover encoded-words decode_header + make_header for anything still =?...?= Charset repair cp1252 characters labelled latin-1; UTF-8 read as cp1252 NFC normalise one byte sequence per visible name Fallback name attachment-N.ext from the content type Safe + unique base name only, unsafe chars replaced, hash suffix

Variant Fix 2: UnicodeEncodeError When Writing to Disk

A correctly decoded name like Отчёт.xlsx still fails on servers whose file system encoding is ASCII, typically containers without a locale:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)

Check sys.getfilesystemencoding() in the job's environment; if it is ascii, set a UTF-8 locale for the process (LANG=C.UTF-8) or enable Python's UTF-8 mode with PYTHONUTF8=1. When neither is possible, transliterate names for disk while keeping the original in your metadata:

# pip install unidecode
from unidecode import unidecode

def ascii_disk_name(name: str) -> str:
    return unidecode(name).replace(" ", "_")               # 'Отчёт.xlsx' -> 'Otchiot.xlsx'

Store the decoded original name in the state database or manifest next to the ASCII path. Users searching for "Отчёт" then still find the file, and the file system never sees a character it cannot store.

Recording Original and Stored Names

Downstream steps — and people answering "where is the invoice Contoso sent on Tuesday" — need both names. Keep a manifest row per attachment:

# pip install "pandas>=2.2"
from datetime import datetime, timezone
from pathlib import Path
import pandas as pd

def manifest_rows(msg, saved: list[tuple[str | None, Path]]) -> pd.DataFrame:
    return pd.DataFrame([{
        "received": msg["Date"].datetime.astimezone(timezone.utc).isoformat() if msg["Date"] else None,
        "sender": msg["From"].addresses[0].addr_spec if msg["From"] else None,
        "subject": str(msg["Subject"] or ""),
        "original_name": original,
        "stored_as": path.name,
        "bytes": path.stat().st_size,
    } for original, path in saved])

With the modern policy, msg["Date"].datetime and msg["Subject"] are already decoded objects, so the manifest contains readable subjects rather than encoded-words. Writing it as CSV with a UTF-8 BOM keeps non-ASCII names readable in Excel, as described in fixing encoding errors in CSV files.

Verification

Build test messages covering every encoding form, run them through the saving code, and assert the names that come out. Constructing messages with the standard library makes the tests independent of any mail server.

# stdlib only
import tempfile
from email.message import EmailMessage
from pathlib import Path

def make_message(filename_header: str, payload: bytes = b"%PDF-1.7 test") -> bytes:
    msg = EmailMessage()
    msg["From"], msg["To"], msg["Subject"] = "[email protected]", "[email protected]", "test"
    msg.set_content("see attachment")
    msg.add_attachment(payload, maintype="application", subtype="pdf", filename="placeholder.pdf")
    raw = msg.as_bytes()
    return raw.replace(b'filename="placeholder.pdf"', filename_header.encode())

CASES = {
    "filename*=UTF-8''Rechnung%20Nr%2045.pdf": "Rechnung Nr 45",
    'filename="=?UTF-8?B?UmVjaG51bmcgTnIgNDUucGRm?="': "Rechnung Nr 45",
    'filename="=?iso-8859-1?Q?Factura_=96_Septiembre.pdf?="': "Factura – Septiembre",
    'filename="../../etc/passwd.pdf"': "passwd",
}

def test_filenames() -> None:
    for header, expected_stem in CASES.items():
        with tempfile.TemporaryDirectory() as tmp:
            saved = save_all(make_message(header), Path(tmp))
            assert len(saved) == 1, f"{header}: nothing saved"
            name = saved[0].name
            assert name.startswith(expected_stem), f"{header}: got {name!r}"
            assert saved[0].parent == Path(tmp), f"{header}: escaped the output folder"
    print(f"{len(CASES)} filename cases passed")

if __name__ == "__main__":
    test_filenames()

The path-traversal case belongs in the same test file as the encoding cases: sanitising and decoding happen in the same function, and a change to one should never silently weaken the other.

FAQ

Why does the subject decode but the filename does not? With the legacy policy, msg["Subject"] is still raw too; you may be decoding it elsewhere. With policy.default, both decode automatically.

Can I trust the declared charset at all? Usually. The cp1252 repair only runs when C1 control characters appear, which never occur in real filenames, so it does not alter correct names.

What if two attachments in one message have the same name? The content hash suffix keeps them apart unless the files are byte-identical, in which case saving one copy is correct.

Should I keep the original filename at all? Yes — suppliers refer to documents by name. Keep it in the manifest even when the stored name is sanitised or transliterated.

Part of Processing Email Attachments Automatically.