RT Robert Truesdale

August 21, 2026

Scripts, Cron Jobs, Alerts, and the Part Everyone Forgets

You’ve got your script running. Cron’s in place. PagerDuty’s pinging your phone. You’re done, right?

No. You’re just getting started—and most people stop here and wonder why things break quietly or why their “set-and-forget” system becomes a maintenance nightmare in six months.

I’ve seen this play out too many times: a clever cron job that backups a database, a monitoring script that checks uptime, an alert that fires… and then sits dormant until something doesn’t alert when it should. Or worse, it alerts too much, until the team starts ignoring it.

The part everyone forgets isn’t the code. It’s the context—and the discipline to keep it alive.

Let’s talk about what actually works in real-world ops, not what reads well in a slide deck.

—

The Script Is the Easy Part

You write the script. Bash, Python, whatever you’re comfortable with. It does the thing: checks disk space, rotates logs, hits an API, whatever.

But here’s what you don’t write into the script:

  • Why it exists
  • What happens if it fails silently
  • Who owns it now—and in six months
  • How to test it without breaking something

I had a backup script once that ran nightly. It logged to /var/log/backup.log. Great. But when the backup volume filled, it still exited with code 0. No alert. No warning. Just “success” in the logs. For weeks.

The fix wasn’t better error handling in the script. It was adding a failure mode check: if the log file hasn’t been modified in 25 hours, send a low-priority alert. Not because the script was broken—but because context was missing.

Write the script. Then write the failure story.

—

Cron Isn’t a Black Box—It Has Teeth

Cron jobs are deceptively simple. Put a line in crontab, walk away.

But cron has gotchas:

  • Timezone mismatches (cron uses system time; your logs might not)
  • PATH issues (your script runs fine in your shell, not in cron’s minimal env)
  • Overlap (if a job runs every 5 minutes and takes 7, you get a backlog of zombies)
  • No visibility into why it ran (or didn’t)

I once inherited a cron that ran a health check every minute. It was fine until the server’s clock drifted 2 minutes behind NTP. Suddenly, the job ran at 23:58, 00:58, 01:58… and missed the 00:00 window where other services reset. No alerts fired. Everything looked fine—until it wasn’t.

Use systemd timers instead of cron where possible. They give you logging, dependencies, and overlap handling out of the box. If you must use cron:

  • Always log to a dedicated file with timestamps
  • Set MAILTO="ops@yourdomain" so failures hit your inbox
  • Add a lockfile or use flock to prevent overlap

Example:

# Bad: runs every 5 min, no overlap protection
*/5 * * * * /opt/scripts/health_check.sh

# Better: uses flock, logs to journal
*/5 * * * * /usr/bin/flock -n /tmp/health_check.lock /opt/scripts/health_check.sh

—

Alerts Without Context Are Noise

You’ve built the script. Cron’s running it. Now you add an alert: “Disk > 90% → page.”

But what does “disk” mean? /? /var? /data? What if it’s a tmpfs? What if it’s a RAID array where one disk is degraded but space is fine?

Alerts need diagnostic context, not just thresholds.

I worked on a monitoring setup where the alert fired: “CPU high.” The fix? Add a label: cpu_breakdown=system,user,idle,iowait. Suddenly, you see it’s not a spike in user load—it’s iowait from a slow disk. That changes the response.

Also: don’t alert on every failure. Alert on unusual patterns or expected behaviors not happening.

Example:

  • ✅ Alert: “Backup log not updated in 26 hours”
  • ❌ Alert: “Backup failed (exit code 1)” — unless you always expect it to succeed
  • ✅ Alert: “No new entries in log for 1 hour” (if logs are supposed to rotate hourly)

And always include a runbook link or a --help flag in the alert itself. Example Slack message: > ⚠️ backup_stale: /data/backup.log last updated 27h ago > Runbook: https://internal/wiki/backup-stale > Last run: 2026-04-02T14:03:22Z (exit 0)

—

The Part Everyone Forgets: Ownership and Decay

Here’s the truth: scripts, cron jobs, and alerts decay.

Not because they’re bad code—but because the world moves.

  • That API endpoint gets deprecated
  • The log path changes after a package upgrade
  • The person who wrote it leaves
  • The server gets migrated to a new cloud region, and cron’s timezone breaks

I’ve seen a perfectly working cron job break for three months because the new sysadmin assumed it was a legacy thing and didn’t touch it—until the backup volume filled and the whole app went down.

So:

  • Assign owners to every automated job. Not “the team,” but one person.
  • Add a # MAINTAINED_BY header in scripts. Update it when ownership changes.
  • Run a quarterly “cron audit”: list all cron entries, check if they’re still needed, if logs are clean, if alerts are firing.
  • Kill jobs you don’t use. Dead code is technical debt with interest.

Bonus: add a --check-config flag to your scripts. It verifies dependencies, config paths, and permissions before running. Saves hours of debugging.

—

Failure Modes: What Breaks Quietly

The worst failures don’t scream—they whisper.

Here are the ones I see most often:

  • Silent success: Script exits 0 but does nothing (e.g., API returns 200 OK with an empty body)
  • Timezone drift: Cron runs at the wrong time, logs misaligned with other systems
  • Path hell: /usr/local/bin not in $PATH in cron, so python fails but python3 works
  • Log rotation gaps: Your script logs to app.log, but logrotate rotates it during the run—half the log is lost
  • Rate limits: Your “backup” script hits an external API 100x/day. After 6 months, the API blocks you—and you don’t know until the backup is empty

One client had a monitoring script that looked like it was checking service health. It pinged the server on port 80. But it never checked if the app was actually responding correctly. The server was up. The app was down. No alert.

The fix? Add a /healthz endpoint that returns status and a timestamp. Then the script checks both connectivity and freshness.

Test failure modes on purpose. Break things in staging. Then watch your alerting.

—

What I Would Do First

You don’t need to overhaul everything tomorrow. Start small. Pick one cron job or script you run daily—and do this:

  • Add a --check-config flag
  • Verify config files exist
  • Check permissions on log paths
  • Test write access to log dir
  • If anything fails, exit 1 before doing real work
  • Add a failure-mode check
  • After the main work, ask: “Did I actually do the thing I’m supposed to?”
  • Example: For a backup script, check that the backup file size > 0 and timestamp is recent
  • If not, exit 1 even if the main command succeeded
  • Log to systemd journal (or a structured file)
  • Include: timestamp, job name, exit status, and a human-readable “what I did”
  • Example: backup completed: 12 files, 2.3GB, last file: db_20260402.sql
  • Add a 1-line runbook link in the script’s --help
  • # Runbook: https://internal/wiki/backup-stale
  • If it breaks, the first thing the on-call person sees isn’t panic—they see a link

Then, every quarter:

  • Run crontab -l -u root (and others)
  • For each job:
  • Who owns it?
  • When was it last tested?
  • Is it still needed?
  • If not, kill it.

The goal isn’t more automation. It’s reliable automation.

Scripts, cron, alerts—they’re just tools. The part everyone forgets is that they’re part of a system. And systems need maintenance, not just magic.

You don’t need to be a guru. You just need to keep the lights on—and the context alive.