Python Doc & Data Automation

Fix Cron Not Running a Python Script

The script works when you run it and produces nothing when cron does:

$ python3 /home/reports/pipeline/run.py     # works: writes out/report.pdf
$ crontab -l
0 6 * * * python3 /home/reports/pipeline/run.py
# 06:00 arrives, no report, no error, no log

Or the system log carries a single unhelpful line:

CRON[41208]: (reports) CMD (python3 /home/reports/pipeline/run.py)
/bin/sh: 1: python3: not found
ModuleNotFoundError: No module named 'pandas'

Cron did run it. The job failed in an environment that shares almost nothing with your shell, and its output went nowhere you looked.

Root Cause

An interactive shell sources profile files that set PATH, activate virtualenvs, export credentials and define the working directory. Cron sources none of that. It runs the command with /bin/sh, a PATH of roughly /usr/bin:/bin, HOME set, and essentially nothing else.

Four consequences follow, and each produces a different symptom. A python3 installed under ~/.pyenv or /usr/local/bin is not on cron's PATH, so the command is not found. A virtualenv that was never activated means the system interpreter runs without your dependencies, hence ModuleNotFoundError. Relative paths resolve against HOME rather than the project, so inputs are missing and outputs land somewhere unexpected. And unless output is redirected, cron mails it to the local user — a mailbox nobody reads on a server without a mail transfer agent, so it is discarded.

What your shell provides that cron does not Two columns compare environments. The interactive shell has a full PATH including pyenv and local binaries, an activated virtualenv, the project as the working directory, and a terminal to print to. Cron has a minimal PATH of usr bin and bin, no virtualenv, the home directory as the working directory, and output sent to local mail which is usually discarded. Each difference maps to one of the four symptoms. Interactive shell Cron PATH includes ~/.pyenv, /usr/local/bin python3 resolves PATH = /usr/bin:/bin python3: not found virtualenv activated pandas importable no virtualenv ModuleNotFoundError cwd = the project directory data/input.csv found cwd = $HOME FileNotFoundError output goes to the terminal you see the traceback output goes to local mail discarded — the real problem

Minimal Diagnostic

Before changing the job, capture what cron actually sees. Schedule a probe for two minutes' time.

# Add temporarily with: crontab -e
* * * * * /usr/bin/env > /tmp/cron_env.txt 2>&1; pwd >> /tmp/cron_env.txt; \
          command -v python3 >> /tmp/cron_env.txt 2>&1

# After a minute
cat /tmp/cron_env.txt

HOME=/home/reports
LANG=en_GB.UTF-8
PATH=/usr/bin:/bin
LOGNAME=reports
SHELL=/bin/sh
/home/reports              <- working directory, not the project
/usr/bin/python3           <- the SYSTEM interpreter, not the venv

That file answers every question at once: the PATH, the working directory and which interpreter will run. Remove the probe entry once you have it.

Confirm cron is even firing, which is a different failure:

# Debian/Ubuntu
grep CRON /var/log/syslog | tail -20
# systemd systems
journalctl -u cron --since "1 hour ago" | tail -20

No CMD line for your job means the schedule never matched — a syntax error in the crontab, or a file without a trailing newline, which some cron implementations ignore entirely.

The Fix: Absolute Paths, the venv Interpreter, Redirected Output

# crontab -e
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=""

# 06:00 daily — absolute interpreter, absolute script, absolute log
0 6 * * * cd /home/reports/pipeline && \
  /home/reports/pipeline/.venv/bin/python run.py \
  >> /home/reports/pipeline/logs/cron.log 2>&1

Every element of that line is doing work. cd into the project makes relative paths inside the script resolve as they do interactively. Calling .venv/bin/python directly runs the virtualenv's interpreter with its site-packages — no activation script needed, and it is more robust than sourcing activate from a non-interactive shell. >> appends rather than truncating, so yesterday's failure is still there tomorrow. And 2>&1 redirects stderr into the same file, which is where the traceback lives.

Wrapping the job in a small shell script keeps the crontab line short and makes the job runnable by hand in exactly the form cron uses:

#!/usr/bin/env bash
# /home/reports/pipeline/run_daily.sh
set -euo pipefail          # fail on error, on unset variables, and through pipes

PROJECT="/home/reports/pipeline"
cd "$PROJECT"

# Secrets live in a file readable only by this user, not in the crontab
set -a
[ -f "$PROJECT/.env" ] && . "$PROJECT/.env"
set +a

mkdir -p "$PROJECT/logs"
exec "$PROJECT/.venv/bin/python" run.py "$@"

0 6 * * * /home/reports/pipeline/run_daily.sh >> /home/reports/pipeline/logs/cron.log 2>&1

set -euo pipefail matters more under cron than anywhere else: without it, a failing step is followed by the next one and the job reports success having done half the work.

Variant Fix 1: Logging From Inside the Script

Redirecting stdout captures print statements and tracebacks. Structured logging makes the log usable months later.

# Standard library only
import logging
import sys
from datetime import datetime, timezone
from pathlib import Path

PROJECT = Path(__file__).resolve().parent          # absolute, whatever the cwd is
LOG_DIR = PROJECT / "logs"

def configure_logging(name: str = "pipeline") -> logging.Logger:
    """Log to a dated file and to stderr, so cron's redirect catches it too."""
    LOG_DIR.mkdir(parents=True, exist_ok=True)
    logger = logging.getLogger(name)
    logger.setLevel(logging.INFO)
    if logger.handlers:
        return logger

    formatter = logging.Formatter(
        "%(asctime)s %(levelname)-8s %(name)s %(message)s",
        datefmt="%Y-%m-%dT%H:%M:%S%z",
    )
    stamp = datetime.now(timezone.utc).strftime("%Y%m%d")
    file_handler = logging.FileHandler(LOG_DIR / f"{name}-{stamp}.log", encoding="utf-8")
    file_handler.setFormatter(formatter)
    stream_handler = logging.StreamHandler(sys.stderr)
    stream_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    logger.addHandler(stream_handler)
    return logger

if __name__ == "__main__":
    log = configure_logging()
    log.info("run started")
    try:
        # ... the actual work ...
        log.info("run finished: %d row(s) published", 1204)
    except Exception:
        log.exception("run failed")                # logs the full traceback
        raise SystemExit(1)

Path(__file__).resolve().parent is the single most valuable line for cron reliability: it makes every path in the script independent of the working directory, so the job behaves identically however it is invoked. Combined with the log directory, it removes the two most common "works for me" differences.

Variant Fix 2: Overlapping Runs

A job scheduled every five minutes that occasionally takes six will run twice concurrently, and two copies writing the same output file corrupt it. A lock file prevents it.

# flock is in util-linux on most distributions
*/5 * * * * /usr/bin/flock -n /tmp/pipeline.lock /home/reports/pipeline/run_daily.sh \
  >> /home/reports/pipeline/logs/cron.log 2>&1

-n makes the second invocation exit immediately rather than queueing, which is almost always what you want: a five-minute job that queues will accumulate a backlog it can never clear. The Python-level equivalent, for portability:

# Standard library only
import fcntl
from contextlib import contextmanager
from pathlib import Path

@contextmanager
def single_instance(lock_path: Path):
    """Exit quietly if another copy of this job is already running."""
    lock_path.parent.mkdir(parents=True, exist_ok=True)
    handle = lock_path.open("w")
    try:
        fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        handle.close()
        raise SystemExit(0)          # not an error: the previous run is still working
    try:
        yield
    finally:
        fcntl.flock(handle, fcntl.LOCK_UN)
        handle.close()

Variant Fix 3: Timezone and DST

Cron uses the system timezone. On a machine set to UTC, 0 6 * * * is 06:00 UTC — 07:00 in British Summer Time. Worse, on a machine set to a DST-observing zone, a job scheduled at 01:30 runs twice on the clocks-back night and not at all on the clocks-forward night.

# Make the timezone explicit in the crontab (supported by cronie and Vixie cron)
CRON_TZ=Europe/London
0 6 * * * /home/reports/pipeline/run_daily.sh >> /var/log/pipeline.log 2>&1

timedatectl                     # confirm what the system thinks the timezone is

Schedule anything DST-sensitive outside the 01:00–03:00 window, and make the job idempotent so a double run is harmless — which for a report pipeline usually means writing to a dated filename and overwriting rather than appending.

Symptom to fix, at a glance Four rows pair a symptom with its fix. Python not found is fixed by calling the interpreter by absolute path. Module not found is fixed by using the virtualenv interpreter rather than the system one. File not found is fixed by changing into the project directory and resolving paths from the script location. Nothing at all in the log is fixed by appending both stdout and stderr to a file, because cron mails output to a mailbox nobody reads. Symptom Fix python3: not found absolute interpreter path ModuleNotFoundError .venv/bin/python, not system python3 FileNotFoundError on data/… cd into the project; resolve from __file__ nothing in any log append stdout AND stderr: >> log 2>&1

Verification

Reproduce cron's environment deliberately rather than hoping tomorrow's run works.

# Run the job with cron's environment, from cron's working directory
env -i HOME="$HOME" LOGNAME="$USER" PATH=/usr/bin:/bin SHELL=/bin/sh \
  /bin/sh -c 'cd "$HOME" && /home/reports/pipeline/run_daily.sh'
echo "exit code: $?"

If that succeeds, the scheduled run will too. Add a heartbeat so silence is itself detectable:

# Standard library only
import json
from datetime import datetime, timezone
from pathlib import Path

HEARTBEAT = Path("/home/reports/pipeline/logs/heartbeat.json")

def write_heartbeat(status: str, rows: int = 0, error: str | None = None) -> None:
    """Record the outcome of this run so a monitor can detect a missed one."""
    HEARTBEAT.parent.mkdir(parents=True, exist_ok=True)
    HEARTBEAT.write_text(json.dumps({
        "status": status,
        "rows": rows,
        "error": error,
        "finished_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }, indent=2))

def assert_recent(max_age_hours: float = 26.0) -> None:
    """Fail when the last successful run is older than expected."""
    assert HEARTBEAT.exists(), "no heartbeat file — the job has never completed"
    record = json.loads(HEARTBEAT.read_text())
    finished = datetime.fromisoformat(record["finished_at"])
    age_hours = (datetime.now(timezone.utc) - finished).total_seconds() / 3600
    assert record["status"] == "ok", f"last run failed: {record.get('error')}"
    assert age_hours <= max_age_hours, f"last successful run was {age_hours:.1f} hours ago"
    print(f"heartbeat OK — {record['rows']:,} row(s), {age_hours:.1f}h ago")

A job that stops running produces no error anywhere, so the only way to notice is to check that it did run. Point an external monitor at the heartbeat file, or push to a dead-man's-switch service, and a silently disabled crontab becomes an alert instead of a discovery three weeks later. The wider logging and alerting design is in scheduling and logging automation jobs.

Reading a crontab schedule field by field The five schedule fields are shown in order with their ranges: minute zero to fifty nine, hour zero to twenty three, day of month one to thirty one, month one to twelve, and day of week zero to six. The worked example, zero six star star one to five, means six in the morning on weekdays only. A note records that the hour is interpreted in the system timezone unless CRON_TZ is set. 0 6 * * 1-5 — six in the morning, weekdays 0 minute 0-59 6 hour 0-23 * day 1-31 * month 1-12 1-5 weekday 0-6 The hour is read in the SYSTEM timezone unless CRON_TZ is set Day-of-month and day-of-week are ORed when both are restricted — a classic surprise

FAQ

Should I use systemd timers instead of cron? On a systemd host, yes for anything important. Timers give you journalctl logs, dependency ordering, Environment= and WorkingDirectory= directives, and automatic handling of missed runs with Persistent=true. Cron remains fine for simple jobs on shared hosts.

Why does source .venv/bin/activate not work in a crontab? Because cron runs /bin/sh, where source is not a builtin, and because the activation only affects the shell that runs it. Calling .venv/bin/python directly achieves the same result with none of the fragility.

Where do credentials belong? In a file readable only by the job's user, sourced by the wrapper script — not in the crontab, which is world-readable on some systems, and not in the repository. Better still, use the platform's secret store and read it at runtime.

Why does my job run at the wrong hour? The system timezone differs from your local one, or DST has shifted it. Check timedatectl, set CRON_TZ explicitly, and avoid scheduling in the 01:00–03:00 window.

Can I stop cron emailing me? Set MAILTO="" at the top of the crontab, and redirect output to a file instead. Do not simply discard it with >/dev/null 2>&1 — that is how failures become invisible.

Part of Scheduling and Logging Automation Jobs.