Python Doc & Data Automation

Fix: matplotlib No Display Error in Headless Jobs

The chart code runs perfectly in a notebook and dies the first night it runs unattended:

Traceback (most recent call last):
  ...
tkinter.TclError: no display name and no $DISPLAY environment variable

Other environments produce different wording for the same problem — UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown, ImportError: Cannot load backend 'TkAgg' which requires the 'tk' interactive framework, or in a container, a silent hang while something waits for an X server that will never arrive.

Root Cause

matplotlib draws through a backend, and it chooses one at import time. If it finds a GUI toolkit installed — Tk, Qt, GTK — it picks the matching interactive backend, because the usual case is a person looking at a window. An interactive backend needs a display connection, so in cron, a systemd unit, a container or a CI runner it fails as soon as a figure is created or shown. The non-interactive Agg backend renders straight to a pixel buffer and needs no display at all. The subtlety is timing: the backend is decided when matplotlib.pyplot is first imported, so setting it afterwards has no effect, and an import buried in a helper module can settle the choice before the main script gets a say.

When the backend gets decided The process starts and the first import of matplotlib pyplot triggers backend selection. Matplotlib checks the MPLBACKEND environment variable, then any explicit use call made before the import, then falls back to detecting an installed GUI toolkit. Once pyplot is imported the choice is fixed. A use call made after the import is ignored, which is why moving the call to the top of the main script does not help when a helper module imported earlier already pulled pyplot in. Process starts no backend yet, blue, nothing decided MPLBACKEND checked environment wins, green, set this in the job matplotlib.use() honoured only before pyplot, green, must come first pyplot imported backend is now fixed, amber, GUI toolkit detected if nothing set Later use() call silently ineffective, red, the usual failed fix

Minimal Diagnostic

Report which backend was selected, what could have selected it, and whether a display exists.

# pip install matplotlib
import os
import sys

def backend_report() -> None:
    print(f"MPLBACKEND={os.environ.get('MPLBACKEND', '(unset)')}")
    print(f"DISPLAY={os.environ.get('DISPLAY', '(unset)')}")
    print(f"pyplot already imported: {'matplotlib.pyplot' in sys.modules}")
    import matplotlib
    print(f"matplotlib {matplotlib.__version__}, backend now: {matplotlib.get_backend()}")
    print(f"config dir: {matplotlib.get_configdir()}")
    print(f"cache dir:  {matplotlib.get_cachedir()}")
    writable = os.access(matplotlib.get_cachedir(), os.W_OK) if \
        os.path.isdir(matplotlib.get_cachedir()) else False
    print(f"cache writable: {writable}")
    rc = matplotlib.matplotlib_fname()
    print(f"rc file: {rc}")

if __name__ == "__main__":
    backend_report()

MPLBACKEND=(unset)
DISPLAY=(unset)
pyplot already imported: True
matplotlib 3.9.2, backend now: TkAgg
config dir: /home/worker/.config/matplotlib
cache dir:  /home/worker/.cache/matplotlib
cache writable: False
rc file: /usr/lib/python3/dist-packages/matplotlib/mpl-data/matplotlibrc

Two problems: pyplot was already imported before this function ran, so the backend is fixed at TkAgg with no display, and the cache directory is not writable — which produces a separate, slower failure where matplotlib rebuilds its font cache on every run.

Fix: Choose the Backend Before pyplot Is Imported

The environment variable is the most reliable place, because it is read before any Python code runs.

# in the cron entry, systemd unit or Dockerfile
MPLBACKEND=Agg

# Dockerfile
FROM python:3.12-slim
ENV MPLBACKEND=Agg \
    MPLCONFIGDIR=/tmp/matplotlib \
    HOME=/tmp
RUN pip install --no-cache-dir matplotlib pandas \
    && python -c "import matplotlib.pyplot"     # warm the font cache into the image

Where the environment cannot be controlled, set it in code — but before importing pyplot, and in the entry point rather than a helper:

# pip install matplotlib
import matplotlib
matplotlib.use("Agg")                        # changed: must precede the pyplot import
import matplotlib.pyplot as plt              # changed: import after, not before

import io
from pathlib import Path

def render_chart(labels: list[str], values: list[float], out: Path | None = None) -> bytes:
    figure, axes = plt.subplots(figsize=(7.5, 3.0), dpi=110)
    try:
        axes.bar(labels, values, color="#2563eb")
        axes.set_ylabel("documents")
        axes.spines[["top", "right"]].set_visible(False)
        figure.tight_layout()
        buffer = io.BytesIO()
        figure.savefig(buffer, format="png", bbox_inches="tight")
    finally:
        plt.close(figure)                    # changed: release it even if savefig raised
    data = buffer.getvalue()
    if out is not None:
        out.parent.mkdir(parents=True, exist_ok=True)
        out.write_bytes(data)
    return data

Closing the figure in a finally block is the second half of the fix. pyplot keeps a reference to every figure created through it, so a job rendering one chart per customer leaks memory until it is killed — a failure that looks like a memory problem in the data code rather than in the charting.

MPLCONFIGDIR matters in containers that run as a non-root user with no writable home. Without it, matplotlib warns on every run and rebuilds its font cache, adding several seconds to a job that should take milliseconds.

Where each environment needs the setting A cron entry takes the variable on its own line above the schedule, and needs an absolute PATH. A systemd unit takes an Environment directive and usually a writable state directory. A Dockerfile takes an ENV instruction and benefits from warming the font cache at build time. CI runners often set it in the workflow environment block. A library that cannot control the environment must call use before importing pyplot, in its entry point. Environment Set it here Also needed cron MPLBACKEND=Agg line absolute PATH too systemd Environment= directive writable MPLCONFIGDIR Docker ENV instruction warm the font cache CI runner workflow env block cache the font dir Library code use() before pyplot entry point only Notebook, then script do not rely on rcParams test headless

Variant Fix 1: An Import Elsewhere Already Fixed the Backend

When a helper module imports pyplot at module level, the backend is settled before the entry point runs. Find the culprit rather than guessing:

# stdlib only
import sys

def who_imported_pyplot() -> None:
    if "matplotlib.pyplot" not in sys.modules:
        print("pyplot not yet imported — safe to choose a backend")
        return
    module = sys.modules["matplotlib.pyplot"]
    importers = [name for name, mod in sys.modules.items()
                 if mod is not None and getattr(mod, "plt", None) is module]
    print(f"pyplot already imported; modules holding it: {importers[:10]}")

The durable fix is the environment variable, which cannot be lost to import order. Failing that, move chart imports inside the functions that draw, so nothing pulls pyplot in until a chart is actually requested — which also shortens startup for the runs that produce no charts at all.

Variant Fix 2: Fonts Differ Between the Laptop and the Server

Charts that render correctly locally often come out with boxes instead of glyphs, or a different typeface, on a slim container image. That is the same font-substitution problem that affects document conversion:

# pip install matplotlib
import matplotlib
from matplotlib import font_manager

def pin_font(preferred: str = "DejaVu Sans") -> str:
    available = {f.name for f in font_manager.fontManager.ttflist}
    chosen = preferred if preferred in available else "DejaVu Sans"
    matplotlib.rcParams["font.family"] = "sans-serif"
    matplotlib.rcParams["font.sans-serif"] = [chosen]
    matplotlib.rcParams["axes.unicode_minus"] = False        # avoids a missing-glyph box for minus
    return chosen

DejaVu Sans ships with matplotlib itself, so it is always present — which makes it the right default for a server even when something prettier is installed locally. Turning off axes.unicode_minus replaces the Unicode minus sign with an ASCII hyphen, removing the single most common missing-glyph box on charts with negative values.

Making a charting job run unattended Set the backend through the environment so import order cannot break it. Point the config directory somewhere writable. Pin the font family to one that ships with matplotlib. Render to an in-memory buffer rather than a file, so the chart can be embedded directly in a report. Close every figure in a finally block. Finally assert the produced bytes are a valid image of the expected size before treating the job as successful. MPLBACKEND=Agg in the job environment Writable MPLCONFIGDIR no font cache rebuild Pin the font DejaVu Sans always exists Render to a buffer embed without a temp file Close every figure in a finally block Assert the bytes valid PNG, right size

Rendering Many Charts Without Growing

A report with one chart per customer means hundreds of figures in a single run, and the interaction between pyplot and memory is the thing that turns a working script into a nightly out-of-memory kill. Reusing one figure is faster than creating and closing hundreds:

# pip install matplotlib
import io
import matplotlib.pyplot as plt

def chart_series(datasets: list[tuple[str, list[str], list[float]]]) -> dict[str, bytes]:
    figure, axes = plt.subplots(figsize=(7.5, 3.0), dpi=110)
    results = {}
    try:
        for name, labels, values in datasets:
            axes.clear()                                 # reuse the axes rather than a new figure
            axes.bar(labels, values, color="#2563eb")
            axes.set_title(name)
            axes.spines[["top", "right"]].set_visible(False)
            buffer = io.BytesIO()
            figure.savefig(buffer, format="png", bbox_inches="tight")
            results[name] = buffer.getvalue()
    finally:
        plt.close(figure)
    return results

axes.clear() resets the data and the labels while keeping the figure, the canvas and the font cache warm, which on a few hundred charts is several times faster than building each one from scratch. The one thing it does not reset is anything set on the figure rather than the axes — a suptitle or a figure-level legend persists, so set those inside the loop too or not at all.

Why the Error Sometimes Does Not Appear

A confusing property of this failure is that it is intermittent across environments that look identical. Three reasons account for most of it.

A machine with no GUI toolkit installed selects Agg on its own, so the code works by accident; installing any package that pulls in Tk later breaks it. A job run interactively over SSH with X forwarding has a DISPLAY, so testing by hand succeeds while the same command under cron fails. And savefig alone sometimes works under an interactive backend, while show or tight_layout triggers the display connection — so a script can run for months and break the day a layout call is added.

None of these are worth diagnosing individually. Setting MPLBACKEND=Agg in every non-interactive environment removes the whole class, and costs nothing where it was not needed.

Verification

Prove the chart rendered, in the right backend, without leaking figures.

# pip install matplotlib pillow
import io
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image

def verify_charting(render, expected_min_width: int = 600) -> None:
    backend = matplotlib.get_backend().lower()
    assert backend in {"agg", "pdf", "svg", "ps", "cairo", "module://matplotlib_inline.backend_inline"}, \
        f"interactive backend {backend!r} — will fail without a display"
    before = len(plt.get_fignums())
    data = render(["09", "10", "11"], [120.0, 340.0, 98.0])
    after = len(plt.get_fignums())
    assert after == before, f"{after - before} figure(s) left open — add plt.close(figure)"
    assert data[:8] == b"\x89PNG\r\n\x1a\n", "output is not a PNG"
    with Image.open(io.BytesIO(data)) as image:
        assert image.width >= expected_min_width, f"chart is {image.width}px wide, expected >= {expected_min_width}"
        extrema = image.convert("L").getextrema()
        assert extrema[0] != extrema[1], "chart is a single flat colour — nothing was drawn"
    print(f"backend {backend}, {len(data) / 1024:.0f}KB PNG, {image.width}x{image.height}, no leaked figures")

if __name__ == "__main__":
    verify_charting(render_chart)

The flat-colour check catches a failure the others miss: a chart that renders as a blank white rectangle because the data was empty or every value was NaN. That produces a valid PNG of the right size, so only looking at the pixels reveals it — and a report full of blank charts is the kind of thing that survives to production unnoticed.

FAQ

Is Agg the only non-interactive backend? No — pdf, svg, ps and cairo are also non-interactive and write those formats directly. Agg is the right default for PNG output.

Does plt.close("all") work instead of closing each figure? It does, but only at the end. Inside a loop it is per-figure closing that keeps memory flat.

Can I still use the same code in a notebook? Yes. Setting MPLBACKEND only in the job environment leaves the notebook on its inline backend.

Why does the first chart take three seconds? Font cache rebuilding. Set MPLCONFIGDIR somewhere writable and warm it at image-build time.

Part of Generating Reports from Pipeline Data.