I've been running automation in production for over fifteen years. I've seen scripts that were supposed to run daily for years without touching them, and I've seen scripts that broke within a week because someone changed a file path on the server and nobody noticed for three days. The difference between those two outcomes isn't how clever the script is. It's whether anyone knew it was broken.
That's the part everyone forgets: the monitoring. Not just whether the job ran, but whether it actually did what it was supposed to do.
The Myth of Set-and-Forget
Here's what happens. You write a script. Maybe it's backing up a database, rotating logs, pulling data from an API, or generating a report. You test it a few times, it works, you throw it in cron or your automation tool of choice, and you move on to the next thing.
Six months later, someone asks why the data looks wrong, or why the backup is three weeks old, or why the report has been blank. You dig in and find the script has been failing silently since March. The cron job is still firing. The exit code is being ignored. The error output went to /dev/null because nobody told it to go somewhere useful.
This isn't a beginner mistake. I've seen senior engineers do this. The script worked in testing, so the job is "done." But testing isn't production, and "working last Tuesday" isn't "working right now."
What You're Actually Monitoring
Most people check if their cron ran. That's not enough. You need to check three things:
Did it run? The cron fired, the script executed, no obvious crash. This is the bare minimum and what most people do.
Did it succeed? The script completed without errors. This requires actually checking exit codes, which most shell scripts don't bother with unless you force them to.
Did it do the thing? The output is correct, the file exists with the right size, the database has the expected rows, the API returned what you expected. This is where most automation falls apart.
There's a fourth level that's even better but nobody does: did it do the thing at the right time and in the right order? If your backup script runs at the same time as your log rotation, you might be backing up files that are being truncated mid-write. Dependencies matter.
Practical Alerting That Doesn't Suck
The standard approach is to email on failure. This works until your inbox has 400 emails about a script that fails every Tuesday and has for two years, and you've tuned it out. Email alerts for persistent failures are noise.
Here's what actually works:
Escalating alerts. If a job fails once, ping the on-call. If it fails three times in a row, escalate. This filters out the transient blips while catching real problems.
Success verification. For critical jobs, have a separate check that confirms the output is correct. Your backup script can return success even if the disk was full and nothing was written. The check needs to verify the backup file exists and has reasonable size.
Output capture. I'm a fan of logging to a file and stdout, with stdout going to a central log aggregator. If you're not collecting your script output somewhere searchable, you're making debugging harder for yourself.
For the content site I run on the side, I have a simple health check that verifies the daily content generation script actually created new pages. It checks the timestamp on the output files. If the most recent file is older than 26 hours, I get a Slack message. Simple, effective, and it catches failures that "did the script run?" checks would miss.
Failure Modes Nobody Talks About
Scripts don't just fail because of code bugs. They fail because the world changes around them:
Credentials expire. That API key you hardcoded two years ago? It expired last month. Now your data sync is failing silently.
Dependencies disappear. A script that calls an external tool will break when that tool gets updated and changes its output format. Or when the server is rebuilt and the tool isn't installed.
Resource exhaustion. The script that ran fine when the database had 10,000 rows starts timing out at 10 million. It still "runs," but it doesn't finish in the allocated cron window.
Timezone confusion. Daylight saving time happens. Cron uses system time. If your job runs at 2 AM and the clocks fall back, it might run twice or not at all depending on your setup.
Path changes. Someone reorganized the directory structure. Your script still runs because cron doesn't care if the path is wrong—it just returns exit code 127 and moves on.
The most insidious failures are the ones where the script runs successfully but produces wrong output. You need to verify the result, not just the exit code.
What I Would Do First
If you're looking at your automation setup and wondering where to start, here's the priority order:
- Find your silent failures. Check your logs for anything that's been erroring out with nobody noticing. You probably have some.
- Add basic success verification. For your most critical jobs, add a simple check: does the output file exist? Does it have content? Does the database have new rows? Pick something that takes five minutes to implement.
- Fix alerting to not suck. Move alerts out of email if that's where they are. Use something that escalates. Set up a daily "jobs that failed" summary if you have enough automation that individual alerts are noise.
- Document the dependencies. What does this script need? What does it produce? What breaks if it stops working? This takes fifteen minutes and saves hours of debugging.
- Schedule time to review. Set a calendar reminder every quarter to look at your automation portfolio. Kill what doesn't work. Update what needs updating. Check if the APIs still exist.
The goal isn't perfect monitoring. It's knowing when something breaks before a human notices and asks you about it.