Python Doc & Data Automation

Run a Document Watcher as a systemd Service

The folder watcher runs fine in a terminal, so it gets started with nohup python watcher.py & on the document server. It survives until the next reboot, or until it crashes on a malformed PDF at 2 a.m. and nobody notices for three days. Turning it into a systemd service is the obvious next step, and the first unit file fails in new ways: status=203/EXEC, ModuleNotFoundError: No module named 'watchdog', a watcher that cannot see files its own user just wrote, or a service that takes 90 seconds to stop and loses the file it was processing.

× doc-watcher.service - Document drop-folder watcher
     Active: failed (Result: exit-code) since Thu 2026-09-17 09:02:11 UTC
    Process: 4121 ExecStart=python watcher.py /srv/drop/incoming (code=exited, status=203/EXEC)

Root Cause

systemd starts processes in a deliberately minimal environment, which is exactly what makes services reproducible and exactly what breaks scripts written for an interactive shell. There is no activated virtualenv, no PATH entries from .bashrc, no working directory unless one is set, and no environment variables except those declared in the unit. status=203/EXEC means the ExecStart program could not be executed — typically a relative path or a bare python that systemd cannot resolve. ModuleNotFoundError means the system interpreter ran instead of the virtualenv's. Permission problems appear because the service runs as a different user, often with a sandboxed file system when hardening options are copied from examples. And slow, lossy stops happen because systemd sends SIGTERM and waits, while a script that only handles KeyboardInterrupt ignores the signal until the 90-second default timeout ends with SIGKILL mid-file.

Minimal Diagnostic

Ask systemd what it actually ran, with which environment, and read the service's own output. These three commands answer almost every service failure.

systemctl status doc-watcher.service --no-pager
systemctl show doc-watcher.service -p ExecStart -p User -p WorkingDirectory -p Environment -p EnvironmentFiles
journalctl -u doc-watcher.service -n 50 --no-pager

To reproduce the service's environment interactively, run the same command line as the service user with an empty environment:

sudo -u docbot env -i PATH=/usr/bin:/bin /opt/doc-watcher/.venv/bin/python -c \
  "import sys, watchdog; print(sys.executable, watchdog.__version__)"
sudo -u docbot test -w /srv/drop/incoming && echo "writable" || echo "NOT writable"

If the import fails here, the service will fail the same way; if the directory is not writable for the service user, moving processed files will fail.

Interactive shell versus systemd environment In an interactive shell the virtualenv is activated so python resolves to the venv interpreter, PATH includes user directories, the working directory is wherever the user is, environment variables come from the shell profile, and Ctrl+C raises KeyboardInterrupt. Under systemd there is no activated venv, PATH is minimal, the working directory is the root directory unless set, only variables declared in the unit exist, and stopping sends SIGTERM which Python does not turn into an exception. Interactive shell python -> .venv/bin/python PATH from .bashrc cwd = where you cd'd vars from shell profile stop: Ctrl+C -> exception systemd service python -> not resolved PATH = minimal cwd = / unless set only Environment= vars stop: SIGTERM, no exception

Fix: A Complete Unit File and a Signal-Aware Script

Install the application into a fixed directory with its own virtualenv, run it as a dedicated user, and describe everything the process needs in the unit. Comments mark each line that fixes one of the failures above.

sudo useradd --system --home /opt/doc-watcher --shell /usr/sbin/nologin docbot
sudo mkdir -p /opt/doc-watcher /srv/drop/{incoming,processed,failed} /var/lib/doc-watcher
sudo python3 -m venv /opt/doc-watcher/.venv
sudo /opt/doc-watcher/.venv/bin/pip install "watchdog>=4.0" pdfplumber openpyxl
sudo cp watcher.py /opt/doc-watcher/
sudo chown -R docbot:docbot /srv/drop /var/lib/doc-watcher

# /etc/systemd/system/doc-watcher.service
[Unit]
Description=Document drop-folder watcher
After=network-online.target remote-fs.target
Wants=network-online.target

[Service]
Type=simple
User=docbot
Group=docbot
WorkingDirectory=/opt/doc-watcher
# absolute interpreter path from the virtualenv: fixes 203/EXEC and ModuleNotFoundError
ExecStart=/opt/doc-watcher/.venv/bin/python -u watcher.py /srv/drop/incoming
# secrets and settings live outside the unit, readable only by root and the service
EnvironmentFile=/etc/doc-watcher/env
Environment=PYTHONUNBUFFERED=1
# restart on crashes, but stop retrying if it crash-loops
Restart=on-failure
RestartSec=10
StartLimitIntervalSec=300
StartLimitBurst=5
# give the script time to finish the current file after SIGTERM
KillSignal=SIGTERM
TimeoutStopSec=60
# hardening that still allows the folders the watcher needs
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/srv/drop /var/lib/doc-watcher
MemoryMax=1G

[Install]
WantedBy=multi-user.target

ProtectSystem=strict makes the whole file system read-only for the service except the paths listed in ReadWritePaths — copy it from an example without that line and every file move fails with Read-only file system. -u and PYTHONUNBUFFERED=1 make log lines reach the journal immediately instead of in 4 KB bursts. StartLimitBurst stops a service that crashes on start from restarting forever and filling the journal. Put remote-fs.target in After= whenever the watched folder is a network mount, so the watcher does not start against an empty mount point.

The script needs to turn SIGTERM into a clean stop: finish the current file, stop the observer, close state.

# pip install "watchdog>=4.0"
import logging
import signal
import sys
import threading
from pathlib import Path

log = logging.getLogger("doc-watcher")
stop = threading.Event()

def handle_signal(signum, frame) -> None:
    log.info("received %s, finishing current file and stopping", signal.Signals(signum).name)
    stop.set()                                                    # changed: cooperative shutdown

def main() -> int:
    logging.basicConfig(level=logging.INFO, stream=sys.stdout,    # changed: stdout goes to journald
                        format="%(levelname)s %(name)s %(message)s")
    signal.signal(signal.SIGTERM, handle_signal)                  # changed: systemd stop
    signal.signal(signal.SIGINT, handle_signal)                   # Ctrl+C when run by hand
    folder = Path(sys.argv[1])
    if not folder.is_dir():
        log.error("%s is not a directory (mount missing?)", folder)
        return 2                                                  # non-zero: systemd counts a failure
    observer, worker_thread = start_watcher(folder, stop)         # observer + worker from the watcher guide
    try:
        while not stop.wait(timeout=5):
            if not worker_thread.is_alive():
                log.error("worker thread died; exiting so systemd restarts the service")
                return 1
    finally:
        observer.stop()
        observer.join(timeout=20)
        worker_thread.join(timeout=30)                            # changed: let the current file finish
    log.info("stopped cleanly")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Logging without timestamps is intentional: journald adds its own, and duplicated timestamps make logs harder to read. Returning a non-zero exit code when the worker thread dies turns a silently stuck watcher into a restart that shows up in systemctl status.

Graceful stop sequence systemd sends SIGTERM to the watcher process. The signal handler sets a stop event. The worker finishes the file it is processing and records it in the state store. The main thread stops the watchdog observer and joins the worker. The process exits with code 0 within the 60 second TimeoutStopSec, so systemd never needs to send SIGKILL. systemd Watcher main Worker thread State store SIGTERM stop.set() record finished file exit 0 before timeout Without a SIGTERM handler, systemd waits TimeoutStopSec and then kills the process mid-file

Enable and start it:

sudo mkdir -p /etc/doc-watcher && sudo install -m 600 -o root -g root env.example /etc/doc-watcher/env
sudo systemctl daemon-reload
sudo systemctl enable --now doc-watcher.service
journalctl -u doc-watcher.service -f

Variant Fix 1: Network Share Not Mounted Yet

A watcher on /mnt/scans that starts before the SMB mount exists watches the empty local directory under the mount point, sees nothing, and never recovers. Declare the mount as a dependency so systemd starts the watcher only when it is present:

# additions to [Unit]
RequiresMountsFor=/mnt/scans

# /etc/fstab — mount on first access and tolerate a slow file server at boot
//fileserver/scans  /mnt/scans  cifs  credentials=/etc/doc-watcher/smb,uid=docbot,gid=docbot,_netdev,x-systemd.automount,x-systemd.mount-timeout=30  0  0

RequiresMountsFor adds both ordering and a requirement on the mount unit, and _netdev marks the mount as needing the network. Pair this with the polling observer, because SMB mounts do not deliver change notifications to the client — see Watching Folders for Incoming Documents. The script's own is_dir() check remains a useful second line of defence.

Variant Fix 2: Get Told When It Fails

Restart=on-failure hides crashes by recovering from them. Add a failure hook so repeated failures notify someone, using a templated helper unit:

# addition to doc-watcher.service [Unit]
OnFailure=notify-failure@%n.service

# /etc/systemd/system/[email protected]
[Unit]
Description=Send failure notice for %i

[Service]
Type=oneshot
User=docbot
EnvironmentFile=/etc/doc-watcher/env
ExecStart=/opt/doc-watcher/.venv/bin/python /opt/doc-watcher/notify.py "%i"

# /opt/doc-watcher/notify.py — pip install requests
import os
import subprocess
import sys
import requests

unit = sys.argv[1]
logs = subprocess.run(["journalctl", "-u", unit, "-n", "30", "--no-pager"],
                      capture_output=True, text=True).stdout
try:
    requests.post(os.environ["ALERT_WEBHOOK_URL"], timeout=10,
                  json={"text": f"{unit} failed on {os.uname().nodename}\n```{logs[-3000:]}```"})
except (KeyError, requests.RequestException) as exc:
    print(f"alert not sent: {exc}", file=sys.stderr)

OnFailure fires when the unit enters the failed state — after start limits are exhausted, or on a failure without restart — so a single transient crash that restarts cleanly does not page anyone. The service user needs membership in the systemd-journal group to read the journal. The same alerting pattern for scheduled jobs is in add retries and failure alerts to automation jobs.

Service lifecycle with restart and alert The service is enabled at boot and starts after the network and mounts. While running, a crash triggers a restart after ten seconds. If it crashes five times within five minutes the start limit is reached and the unit enters the failed state. OnFailure then starts the notification unit, which posts recent journal lines to a webhook so a person can investigate. Boot after mounts Running watching folder Crash restart in 10 s 5 crashes / 5 min start limit hit OnFailure webhook alert

Operating the Service Day to Day

A few commands cover almost all routine work, and writing them into the job's runbook saves the next person from searching:

# is it healthy, and since when?
systemctl status doc-watcher.service --no-pager
# follow the log, or show errors from the last day only
journalctl -u doc-watcher.service -f
journalctl -u doc-watcher.service --since "24 hours ago" -p err --no-pager
# deploy a new version: install, then restart (graceful stop, then start)
sudo cp watcher.py /opt/doc-watcher/ && sudo systemctl restart doc-watcher.service
# after editing the unit file
sudo systemctl daemon-reload && sudo systemctl restart doc-watcher.service
# see effective settings including drop-in overrides
systemctl cat doc-watcher.service

Use drop-in overrides (sudo systemctl edit doc-watcher.service) for per-server changes such as a different watched path or memory limit, rather than editing the packaged unit; they survive redeployment of the main unit file and show up clearly in systemctl cat.

Verification

Check the three behaviours that matter: it starts at boot, it restarts after a crash, and it stops gracefully without losing work.

# starts at boot
systemctl is-enabled doc-watcher.service            # expect: enabled

# restarts after a crash
PID=$(systemctl show -p MainPID --value doc-watcher.service)
sudo kill -9 "$PID"; sleep 15
systemctl show -p NRestarts --value doc-watcher.service   # expect: incremented
systemctl is-active doc-watcher.service                    # expect: active

# stops gracefully while a file is being processed
cp big-sample.pdf /srv/drop/incoming/ && sleep 1
time sudo systemctl stop doc-watcher.service               # expect: well under TimeoutStopSec
journalctl -u doc-watcher.service -n 5 --no-pager | grep "stopped cleanly"
ls /srv/drop/processed/*/ | grep big-sample                # the file finished, not half-processed
sudo systemctl start doc-watcher.service

kill -9 simulates a hard crash that no signal handler can intercept, which is exactly the case Restart=on-failure exists for. If the stop takes close to the timeout and the journal lacks the "stopped cleanly" line, the signal handler is not wired up or the worker ignores the stop event.

FAQ

Should I use Type=notify instead of simple? Only if the script calls sd_notify (for example via the sdnotify package) after startup checks pass. It makes systemctl start wait until the watcher is genuinely ready, which helps dependent services.

Can the service run under my own user account? A user service (systemctl --user) works on desktops, but stops when you log out unless lingering is enabled. On servers, a dedicated system user is cleaner and survives staff changes.

Where do the logs go if the disk fills? journald rotates by size (SystemMaxUse in journald.conf). Keep application data out of logs and ship important events to your monitoring system rather than relying on local retention.

What about Windows servers? Use a Windows service wrapper such as NSSM, or run the watcher from Task Scheduler at startup, as described in run Python scripts with Windows Task Scheduler.

Part of Watching Folders for Incoming Documents.