The Cron Job Failure Runbook: What to Check When a Scheduled Task Silently Dies

A cron job that fails loudly is easy: you get an error, a stack trace, a nonzero exit code in a log somewhere. A cron job that just stops running — no error, no output, nothing in your monitoring, the file it’s supposed to touch just isn’t touched — is the harder case, because there’s no starting thread to pull. This runbook is ordered: work through it top to bottom, since each step rules out an entire class of cause before you spend time on the next.

1. Confirm cron itself is running

Before anything about your job, confirm the daemon is alive.

# systemd-based distros (most current Linux)
systemctl status cron      # Debian/Ubuntu
systemctl status crond     # RHEL/CentOS/Fedora/Amazon Linux

# non-systemd or a quick process check anywhere
ps aux | grep -E '[c]ron'

If the service is stopped, nothing downstream matters — restart it (systemctl enable --now cron/crond) and figure out why it stopped separately (check journalctl -u cron for a crash, or whether something OOM-killed it — see step 9). On containers, this is a common one: a minimal container image often doesn’t run a cron daemon at all unless you explicitly installed and started one as PID 1 or via a supervisor.

2. Confirm the crontab is what you think it is, for the right user

Cron jobs are per-user. It’s easy to edit the wrong one, or edit as the wrong user without noticing.

crontab -l                 # current user's crontab
crontab -l -u deploy       # a specific user's crontab (needs privilege)
sudo crontab -l -u root    # root's crontab is separate from your own

Also check the system-wide locations, which are easy to forget exist and each have slightly different syntax (they include a user field the per-user crontab doesn’t):

cat /etc/crontab
ls /etc/cron.d/
ls /etc/cron.{hourly,daily,weekly,monthly}/

If the job you’re debugging was added by a deploy script, config management tool, or a teammate, confirm it’s actually present in the file cron is reading — not in a .bak file next to it, not in a crontab for a user that no longer runs anything, not commented out.

3. Check the cron log — and know that it may not exist by default

This is the step people lose the most time on, because many current distributions do not log cron activity anywhere by default.

Debian/Ubuntu: historically /var/log/cron.log via rsyslog’s cron.* facility, but the /etc/rsyslog.d/50-default.conf line that populates it is commented out on a lot of default installs. Check journalctl -u cron first — cron’s own stderr goes to the systemd journal regardless of rsyslog config.
RHEL/CentOS/Fedora/Amazon Linux: /var/log/cron (not cron.log), populated by rsyslog by default in most current setups. grep CRON /var/log/cron or tail -f /var/log/cron while you wait for the next scheduled run.
Any systemd-based distro, regardless of rsyslog config: journalctl -u cron (or -u crond) shows cron’s own execution log — when it forked a job, for which user, with what exit status of the fork itself. journalctl -u cron --since "1 hour ago" for a scoped view.
Alpine / busybox cron / minimal containers: often no logging at all unless you pass -L <path> to crond explicitly.

journalctl -u cron --since today | grep -i yourjobname
grep CRON /var/log/syslog 2>/dev/null   # older Debian/Ubuntu without journald

If none of these show your job even attempting to run at the expected time, the problem is upstream of your script entirely — go back to steps 1–2. If the log shows cron did fork the job, the problem is inside your script’s execution, and you move to steps 4 onward.

4. The empty-environment trap

This is the single most common cause of “works when I run it by hand, fails under cron.” Cron runs jobs with a minimal environment — not your login shell’s environment, not the environment your SSH session has. In particular, PATH is typically just /usr/bin:/bin (or similarly minimal), not the PATH your interactive shell built up from .bashrc, .profile, nvm, rbenv, pyenv, or a Python virtualenv activation.

Symptoms: command not found for something that works fine when you type it yourself, or a script that silently uses the system Python/Node instead of the version you meant.

Fix by being explicit rather than relying on inherited environment:

# in the crontab itself — sets PATH for every job below it
PATH=/usr/local/bin:/usr/bin:/bin

# or, inside the job command itself
30 2 * * * /bin/bash -lc '/path/to/script.sh'   # -l loads a login shell's env

# or, most robust: don't depend on PATH at all — use absolute paths
30 2 * * * /usr/local/bin/node /home/deploy/app/job.js

To see exactly what environment a cron job actually runs with, temporarily add a diagnostic job:

* * * * * env > /tmp/cron-env.txt

Run it once, compare /tmp/cron-env.txt against env from an interactive shell. The diff is usually the whole story — missing PATH entries, no HOME set to what you expect, no LANG/LC_*, none of your shell aliases or functions.

5. Permissions and the executable bit

ls -l /path/to/script.sh          # check the x bit
stat -c '%U:%G %a' /path/to/script.sh   # owner, group, mode

The job runs as the crontab’s owner, not as you (unless you’re debugging root’s crontab and you’re root). A script that’s executable by you but not by the cron job’s user fails with a permission error that, depending on how output is handled (step 7), you may never see. Same applies to any files or directories the script reads from or writes to — a script that writes a log file to a directory only writable by your interactive user will fail under a service account’s more restricted permissions.

6. Relative paths and working directory

Cron does not run your job from the directory the script lives in, or from any directory you’d guess — it typically starts in the crontab owner’s home directory (implementation detail varies, but never assume it matches wherever you were cd’d when you tested manually). A script that does cd ../data or opens ./config.yml relative to “wherever I happen to be” will behave differently under cron than under your terminal.

Fix: cd explicitly to a known directory as the first line of the script, or use absolute paths throughout, or set the working directory explicitly in the command:

30 2 * * * cd /home/deploy/app && ./run.sh

7. Output redirection — find out where stdout and stderr actually went

By default, cron mails job output to the crontab owner’s local mail account — which, on most modern servers, has no mail transport configured, so the output goes nowhere observable. This is the classic “silent” failure: the script did print an error, it just went to /var/spool/mail/<user> (if even that’s configured) instead of anywhere you look.

Check for local mail first:

mail -u deploy          # or: cat /var/spool/mail/deploy

Then fix it going forward by redirecting explicitly instead of relying on cron’s mail behavior:

30 2 * * * /path/to/script.sh >> /var/log/myjob.log 2>&1

Redirecting only stdout and not stderr (>> file.log without 2>&1) is a common half-fix — errors still vanish into cron’s mail path while normal output looks fine in the log, which reads as “it ran successfully” right up until it didn’t.

8. Exit codes

A script can print nothing wrong and still have failed, or print something alarming-looking and still have exited 0. Check the actual exit status rather than trusting output:

30 2 * * * /path/to/script.sh >> /var/log/myjob.log 2>&1; echo "exit: $?" >> /var/log/myjob.log

If the script pipes through other commands, remember exit code semantics: $? after a pipeline reflects only the last command by default. Use set -o pipefail at the top of bash scripts so a failure earlier in a pipe doesn’t get masked by a trailing command that succeeds.

9. Overlapping runs and locking

If a job sometimes takes longer than its own interval, a new invocation can start while the previous one is still running. Depending on what the job does, this ranges from harmless (idempotent, no shared state) to actively corrupting (two processes racing to write the same file, or double-processing a queue). It rarely shows up as a clean “failure” — more often as intermittent, hard-to-reproduce wrongness or a process count that quietly climbs.

Use flock to make a job self-excluding:

*/5 * * * * flock -n /tmp/myjob.lock -c /path/to/script.sh

-n makes it non-blocking — if the lock is held, this run exits immediately rather than queueing up behind the previous one, which is almost always what you want for a periodic job.

10. Resource limits and OOM

A job killed by the kernel’s out-of-memory killer, or one that hits a ulimit, doesn’t get a chance to log anything on its way out — it just stops. Check for this after the fact:

dmesg | grep -i 'killed process'
journalctl -k | grep -i oom
grep oom /var/log/syslog     # older Debian/Ubuntu without journald for kernel logs

Cron’s minimal environment also inherits ulimit values from wherever cron itself was started (often very early in boot), which can be more restrictive than your interactive shell’s — check ulimit -a inside a cron-launched diagnostic job (same env-dump trick from step 4 works for ulimit -a > /tmp/cron-ulimit.txt) if you suspect a file-descriptor or memory-limit mismatch.

11. Clock and timezone drift

Cron schedules run against the system clock in whatever timezone cron itself is configured for — not necessarily the timezone you were thinking in when you wrote the schedule, and not necessarily UTC even on a cloud instance.

timedatectl                 # current timezone, whether NTP sync is active
date                        # what "now" means to this machine right now

If timedatectl shows NTP synchronized: no or a large offset, the job may be running — just not when you expect, and any timestamp-dependent logic inside the job (comparing against “today,” generating a filename with the date) can be silently wrong even while the schedule itself fires correctly.

12. SELinux and AppArmor

On distributions that enforce mandatory access control, a script that runs fine manually can be blocked when invoked by cron specifically, because the process’s security context under cron can differ from your interactive session’s context.

# SELinux (RHEL/CentOS/Fedora)
sudo ausearch -m avc -ts recent          # recent denials
sudo sealert -a /var/log/audit/audit.log # human-readable summary, if setroubleshoot is installed

# AppArmor (Debian/Ubuntu)
sudo aa-status                            # confirm whether a relevant profile is loaded/enforcing
sudo dmesg | grep -i apparmor             # denials show up here

A denial here often looks identical to a permissions failure from step 5 — same symptom (the job can’t read/write/execute something it should be able to), different root cause and different fix (audit policy, not file mode bits).

Quick-reference order

Step	Check	Rules out
1	`systemctl status cron`/`crond`	Daemon not running
2	`crontab -l -u <user>`, `/etc/cron.d/`	Wrong user, wrong file, job not present
3	`journalctl -u cron`, `/var/log/cron*`	Job never fired
4	Diagnostic `env` job	Missing PATH/env vars
5	`ls -l`, `stat` on script and its I/O paths	Permission denied
6	Working-directory assumptions in the script	Relative-path failures
7	Explicit `>> log 2>&1`	Output going to unread local mail
8	`$?` after the command	Silent nonzero exit
9	`flock`	Overlapping runs corrupting state
10	`dmesg \| grep -i killed`, `ulimit -a`	OOM kill, resource limits
11	`timedatectl`	Clock/timezone drift
12	`ausearch`/`aa-status`, `dmesg`	MAC policy denial

FAQ

My cron job works when I run the command manually but not under cron — where do I start? Step 4, almost always. The single biggest gap between “works manually” and “fails under cron” is the environment cron gives the job — a minimal PATH, no shell profile sourced, no assumptions about HOME or working directory. Dump env from a cron-run job and diff it against your interactive shell before looking anywhere else.

Where does cron log by default? It depends on the distro, and on a lot of current systems the answer is “nowhere obvious.” Debian/Ubuntu often route cron logging through rsyslog to /var/log/cron.log, but the config line for it is commonly commented out by default. RHEL-family systems typically log to /var/log/cron. On any systemd-based system, journalctl -u cron (or -u crond) works regardless of rsyslog configuration and is the most reliable first check.

Why did my job’s output just disappear? By default cron mails stdout/stderr to the crontab owner’s local mail account, and most servers have no local mail transport set up, so the mail goes nowhere you’ll see it. Redirect explicitly with >> /var/log/myjob.log 2>&1 in the crontab line rather than relying on cron’s mail behavior.

How do I stop a slow job from overlapping with its next scheduled run? Wrap it in flock -n /tmp/myjob.lock -c '/path/to/script.sh'. The -n flag makes the lock non-blocking, so an overlapping invocation exits immediately instead of queuing up and compounding the backlog.

If you’re maintaining this checklist by memory across a fleet of machines, that’s the actual failure mode this runbook exists to route around — not any single step above, but not having visibility into which step is the problem until after something already broke. norc gives you run history and failure alerts across every machine you manage, local or remote, so you find out at step 0. See pricing or how it compares to monitoring-only tools on /compare.