Fixing UnicodeDecodeError in CSV Files: 'utf-8' codec can't decode byte 0x96

The error UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 142: invalid start byte stops your script cold every time you try to load a legacy or Windows-exported CSV with pandas. This page gives you a deterministic workflow: detect the real encoding, apply the right fix, handle edge cases like BOM and mixed encodings, and verify the result.

Root Cause

Pandas defaults to UTF-8. Byte 0x96 is a valid Windows-1252 character — the en-dash – — but it is an illegal start byte in UTF-8. The moment the C parser hits that byte it raises and halts; there is no partial load. The same pattern applies to 0x91–0x9F (curly quotes, em-dash, ellipsis, bullet) that Windows applications encode in the cp1252 range. Regional ERP exports, older Excel CSV saves, and accounting software from any era are the common culprits. For a broader look at which parser handles which file quirks, see Best Python Libraries for CSV Parsing.

Step 1 — Detect the Encoding with chardet

Before guessing, measure. chardet reads a binary sample and returns a codec name and a confidence score.

# pip install chardet pandas
from pathlib import Path
import chardet
import pandas as pd

csv_path = Path("data/export.csv")

try:
    raw_sample = csv_path.read_bytes()[:50_000]  # first 50 KB is enough
    result = chardet.detect(raw_sample)
    print(result)  # {'encoding': 'Windows-1252', 'confidence': 0.73, 'language': ''}
except OSError as e:
    raise SystemExit(f"Cannot read file: {e}") from e

A confidence score above 0.7 is reliable enough to use directly. Below 0.7 the detector is uncertain — fall back to the manual ladder in the next section. Always read in binary mode (read_bytes() / 'rb') so no premature decoding occurs before chardet has seen the raw bytes.

Step 2 — Detect with charset-normalizer (Alternative)

charset-normalizer is a pure-Python alternative with no C extension dependency, making it friendlier in restricted environments. Its API mirrors chardet intentionally.

# pip install charset-normalizer pandas
from pathlib import Path
from charset_normalizer import detect
import pandas as pd

csv_path = Path("data/export.csv")

try:
    raw_sample = csv_path.read_bytes()[:50_000]
    result = detect(raw_sample)
    detected_encoding = result.get("encoding")
    confidence = result.get("confidence", 0.0)
    print(f"Detected: {detected_encoding}  confidence: {confidence:.2f}")
except OSError as e:
    raise SystemExit(f"Cannot read file: {e}") from e

Both libraries return the same dictionary shape: {"encoding": str | None, "confidence": float}. Use either; the rest of the workflow is identical.

Step 3 — Encoding Detection Flow

Step 4 — Fix with Explicit Encoding

cp1252 — Windows and Excel Exports

cp1252 is the right first choice for any file that originated on a Windows machine or was saved by Excel's "Save As CSV" option. It maps 0x80–0x9F to typographic characters that UTF-8 cannot represent in those byte positions.

# pip install pandas
from pathlib import Path
import pandas as pd

csv_path = Path("data/export.csv")

try:
    df = pd.read_csv(csv_path, encoding="cp1252", engine="python")
    print(f"Loaded {len(df):,} rows")
except (UnicodeDecodeError, OSError) as e:
    raise SystemExit(f"Load failed: {e}") from e

latin-1 — Universal Single-Byte Fallback

latin-1 (ISO-8859-1) maps every byte 0x00–0xFF directly to the first 256 Unicode code points. It never raises a UnicodeDecodeError, making it the final safety net when you have no idea what encoding was used. The trade-off: cp1252 characters in 0x80–0x9F come through as control characters rather than typographic symbols, so use it only after cp1252 fails.

# pip install pandas
from pathlib import Path
import pandas as pd

csv_path = Path("data/export.csv")

try:
    df = pd.read_csv(csv_path, encoding="latin-1", engine="python")
    print(df.dtypes)
except OSError as e:
    raise SystemExit(f"Cannot open file: {e}") from e

utf-8-sig — BOM Stripping

Files exported from Excel's UTF-8 CSV option often carry a three-byte BOM (EF BB BF) at the start. When pandas reads these with encoding='utf-8', the first column name gains a leading  — your DataFrame has a column named order_id instead of order_id. Using encoding='utf-8-sig' strips the BOM automatically before parsing.

# pip install pandas
from pathlib import Path
import pandas as pd

csv_path = Path("data/excel_utf8_export.csv")

try:
    df = pd.read_csv(csv_path, encoding="utf-8-sig")
    print(df.columns.tolist())  # ['order_id', 'amount', ...] — no BOM prefix
except (UnicodeDecodeError, OSError) as e:
    raise SystemExit(f"Load failed: {e}") from e

When exporting data to CSV formats from pandas you can write a BOM-free UTF-8 file with df.to_csv(path, encoding='utf-8', index=False) — this prevents the BOM problem entirely for downstream consumers.

Step 5 — Handle Residual Bad Bytes with encoding_errors

Even with the correct encoding, some files contain isolated corrupted bytes — a copy-paste artifact, a transmission glitch. Use encoding_errors to control what happens when the parser hits them.

encoding_errors='replace'

Substitutes each undecodable byte with U+FFFD (the replacement character �, displayed as �). The file loads completely; you can then find affected cells and convert them to pd.NA.

# pip install pandas
from pathlib import Path
import pandas as pd

csv_path = Path("data/mixed.csv")

try:
    df = pd.read_csv(
        csv_path,
        encoding="utf-8",
        encoding_errors="replace",
        engine="python",
    )
    # Replace replacement characters with proper NA
    df = df.replace("�", pd.NA)
    print(f"Rows with any NA after replacement: {df.isna().any(axis=1).sum()}")
except OSError as e:
    raise SystemExit(f"Cannot open file: {e}") from e

encoding_errors='backslashreplace'

Converts each bad byte to a Python escape sequence like \x96. Useful for debugging — you can see exactly which bytes are causing trouble — but leave this only as a diagnostic mode, not in production pipelines.

# pip install pandas
from pathlib import Path
import pandas as pd

csv_path = Path("data/mixed.csv")

try:
    df = pd.read_csv(
        csv_path,
        encoding="utf-8",
        encoding_errors="backslashreplace",
        engine="python",
    )
    # Find cells containing escape sequences to spot bad bytes
    mask = df.apply(lambda col: col.astype(str).str.contains(r"\\x", regex=True))
    print("Cells with escaped bytes:")
    print(df[mask.any(axis=1)])
except OSError as e:
    raise SystemExit(f"Cannot open file: {e}") from e

Never use encoding_errors='ignore'. It silently drops the offending bytes entirely. A value like caf\xe9 becomes caf — a shorter string — which can cause column misalignment across rows, truncated values, and data loss that has no visible signal in the DataFrame.

Step 6 — Automated Fallback Ladder

Combine detection with a ranked fallback sequence. This covers the full workflow: detect → high-confidence shortcut → manual ladder → give up gracefully.

# pip install chardet pandas
from pathlib import Path
import chardet
import pandas as pd

FALLBACK_ENCODINGS = ["utf-8-sig", "cp1252", "latin-1"]

def load_csv_robust(csv_path: Path) -> pd.DataFrame:
    """Load a CSV file, auto-detecting encoding with a manual fallback ladder."""
    try:
        raw_sample = csv_path.read_bytes()[:50_000]
    except OSError as e:
        raise SystemExit(f"Cannot read file: {e}") from e

    detected = chardet.detect(raw_sample)
    enc = detected.get("encoding")
    confidence = detected.get("confidence", 0.0)

    if enc and confidence > 0.7:
        try:
            return pd.read_csv(csv_path, encoding=enc, engine="python")
        except (UnicodeDecodeError, LookupError):
            pass  # fall through to manual ladder

    for encoding in FALLBACK_ENCODINGS:
        try:
            df = pd.read_csv(csv_path, encoding=encoding, engine="python")
            print(f"Loaded with fallback encoding: {encoding}")
            return df
        except UnicodeDecodeError:
            continue

    raise ValueError(
        f"Could not decode {csv_path.name} with any known encoding. "
        "Inspect with a hex editor."
    )

if __name__ == "__main__":
    df = load_csv_robust(Path("data/export.csv"))
    print(df.head())

The same pattern applies when reading Excel files with Python — .xlsx files are always UTF-8 internally, but legacy .xls and CSV saves from Excel carry cp1252 or regional codecs.

Step 7 — Verify the Fix

A row-count assertion and a spot-check catch silent failures before data reaches downstream code. If the file loaded with encoding_errors='replace' but you expected clean data, the assertion surfaces the problem.

# pip install pandas
from pathlib import Path
import pandas as pd

csv_path = Path("data/export.csv")
EXPECTED_ROWS = 10_000  # set from source system record count

try:
    df = pd.read_csv(csv_path, encoding="cp1252", engine="python")
except (UnicodeDecodeError, OSError) as e:
    raise SystemExit(f"Load failed: {e}") from e

# Row count assertion
assert len(df) == EXPECTED_ROWS, (
    f"Expected {EXPECTED_ROWS} rows, got {len(df)}. "
    "Check for header/footer rows or encoding truncation."
)

# Spot-check for replacement characters (indicates residual bad bytes)
replacement_count = df.apply(lambda c: c.astype(str).str.contains("�")).sum().sum()
if replacement_count > 0:
    print(f"WARNING: {replacement_count} cells contain replacement characters.")

print("Verification passed.")

Quick-Reference: Encoding Choices

Common Mistakes

Encoding issues are one facet of a larger ingestion problem. The Cleaning Messy CSV Data with Pandas guide continues from a successfully loaded DataFrame into type coercion, whitespace normalization, and duplicate handling.

FAQ

Why does UnicodeDecodeError name byte 0x96 specifically?0x96 is the cp1252 code point for an en-dash (–). In UTF-8 a byte in the 0x80–0xBF range is only ever a continuation byte, never a start byte, so when the decoder meets 0x96 where a new character should begin it raises immediately. The position number (position 142) is the byte offset, not the row or column — open the file in binary mode and slice raw[135:150] to see the surrounding context. Other frequent offenders are 0x91/0x92 (curly quotes), 0x97 (em-dash), and 0x85 (ellipsis).

Should I use chardet or charset-normalizer? Both return the same {"encoding", "confidence"} shape and are interchangeable in the workflow above. Prefer charset-normalizer when you want a pure-Python dependency with no C extension (it is also what modern requests ships with); prefer chardet if you already have it pinned or need its slightly different heuristics on East Asian encodings. Neither is a substitute for knowing the source system — a confident detection of Windows-1252 on a file you know came from Excel just confirms what cp1252 would have fixed directly.

Why does my first column name start with  or ? That is a UTF-8 byte-order mark (EF BB BF) that pandas read literally because you used encoding='utf-8' instead of encoding='utf-8-sig'. Excel's "CSV UTF-8" export adds it. Switch to utf-8-sig to strip it during parsing, or rename the column with df.columns = df.columns.str.replace('', '') after the fact.

Is latin-1 a safe default since it never raises? It is safe from crashing but not from corruption. latin-1 maps bytes 0x80–0x9F to C1 control characters instead of the typographic symbols Windows put there, so an en-dash silently becomes an invisible control character rather than –. Use it only as the final rung of the fallback ladder, after utf-8-sig and cp1252 have both failed, and inspect the result for control characters.

Why do I still get a UnicodeDecodeError even after passing encoding='cp1252'? Two common causes: the file is genuinely mixed-encoding (rows appended from different systems), or you are on the C parser, which has narrower codec fallback behaviour. Add engine='python', and if that still fails combine the correct base encoding with encoding_errors='replace' to isolate the bad rows rather than losing the whole load — then convert the resulting � cells to pd.NA.

Related

• Cleaning Messy CSV Data with Pandas — post-ingestion cleaning workflows
• Best Python Libraries for CSV Parsing — chardet, charset-normalizer, and parser comparisons
• Reading Excel Files with Python — encoding considerations for .xls and legacy Excel formats
• Exporting Data to CSV Formats — write BOM-free UTF-8 to prevent downstream encoding errors
• Python for Excel & CSV Data Processing — full pipeline overview

Part of Cleaning Messy CSV Data with Pandas.