Most IT folks have been burned by a cron job that stopped working three months ago and nobody noticed until production was on fire. That's not a hypothetical—it's a Tuesday. I've seen it happen at startups, at enterprises, and on personal side projects where someone was just trying to automate a backup.
The conversation around automation always focuses on the cool part: the script, the schedule, the alert that fires. Nobody talks about what happens when any of that breaks and nobody knows. The script that runs at 2 AM, depends on a credential that's about to expire, calls an API that changed its response format, or runs on a server nobody remembers exists.
That's the part everyone forgets.
The Script Is the Easy Part
Writing a script to back up a database, sync files, or check if a service is up—that's straightforward. You can bang that out in an afternoon. The hard part starts after you walk away.
Here's what actually happens in most environments I've seen:
You write a script. You set up a cron job. You add an alert so you get emailed if it fails. Then you move on to the next fire. Six months later, the script is still running—or it isn't—and nobody has a clue which one is true because:
- The email address on the alert is wrong or goes to someone who left
- The script fails but the error gets swallowed because you didn't handle it right
- The alert fires every night and everyone mute it after week two
- The server the script runs on loses power and nobody knows
This is the operational debt of automation. It's invisible until it isn't.
The Alert Problem Nobody Wants to Discuss
Alerts sound good in theory. In practice, most alert setups I've encountered are broken in one of three ways:
Too many alerts, too little signal. The team configures alerts for everything—every script, every check, every API call. Within a week, everyone's inbox looks like a denial-of-service attack. Then they either mute everything or create a rule to archive all automated emails. The alert that actually mattered gets lost.
No escalation path. An alert fires at 2 AM. It goes to a distribution list. Nobody reads it until 9 AM. Meanwhile, the backup job failed, the disk filled up, and now you have a real problem that could have been a non-issue six hours earlier.
The alert checks the wrong thing. I've seen setups where the script alerts on "did it run" rather than "did it succeed." The cron fires, the script starts, and then hangs on a network call. The alert says "job started" and everyone goes back to sleep while nothing actually gets done.
The fix isn't more alerts. It's fewer alerts with better signal, an actual escalation path, and checks that verify the outcome, not just the trigger.
Failure Modes You'll Hit
Let me walk through some real ones:
Dependency rot. Your script calls an API using a library you installed via pip three years ago. The library had a security update. Now your script crashes because the old version is incompatible with the new API. Nobody noticed because the script ran successfully for three years and nobody was checking the output.
Credential expiry. The script uses a service account with a password that expires every 90 days. Nobody updated the credential. The script has been failing silently for six weeks. You find out when someone asks why the backup from two months ago is missing.
Timezone drift. Your cron job runs at 2 AM because that's when traffic is low. Except the server's clock is wrong, or someone moved the server to a different region, and now it's running at 2 PM when traffic is highest. The script still runs. It just runs at the wrong time and nobody thought to verify the schedule.
The script that eats itself. I watched a cleanup script get a bug where it deleted its own log file, then kept running and deleting nothing because the log file was its only state tracking mechanism. It ran successfully for three weeks before anyone noticed the disk was still full.
These aren't edge cases. These are the actual things that break in production.
What Nobody Remembers to Document
Here's the part that gets skipped every time: theoperational manual for the automation.
I'm not talking about a wiki page nobody reads. I'm talking about the bare minimum someone needs to figure out what broke at 3 AM without calling you:
- What the script does, in one sentence
- Where it runs and how to log in
- What credentials it uses and where they live
- What the alert means and what to do about it
- How to manually run it if needed
- What the expected output looks like
You don't need a runbook for a five-line script. But if the script touches production data, external services, or anything that would cause a problem if it broke—document it. Not for the auditors. For the person who's going to fix it at 2 AM while you're on vacation.
The Maintenance Reality
This is the part that kills most automation projects: they require maintenance.
Scripts are not "set and forget." They're more like pets than livestock. They need:
- Periodic checks that they're actually working
- Dependency updates when upstream APIs or libraries change
- Credential rotation before expiry
- Log rotation so the disk doesn't fill up
- Someone who gives a damn
The honest truth is that most environments I've worked in can't sustain this. There's always a higher priority fire. The automation that seemed critical six months ago becomes someone else's problem, then nobody's problem, then a silent failure nobody notices until something else breaks.
If you can't commit to maintaining it, don't automate it. A manual process that works is better than an automated process that's broken.
What I Would Do First
If you're looking at your automation setup and realizing you have no idea what's actually running, here's where I'd start:
- Find everything. Audit your cron jobs, systemd timers, and scheduled tasks. List every automated script you can find and what server it runs on.
- Check the logs. Not for errors—check that logs are being written at all. If a script ran yesterday and there's no log entry, that's your first problem.
- Verify the alerts. Trigger a test failure and see if anyone actually gets notified. Then verify that the notification includes enough information to act on.
- Pick one critical path. Pick the automation that would hurt most if it broke—a backup, a sync, a health check—and read through the script like it's someone else's code. Look for the failure modes I listed above.
- Document that one. Write down the five things someone would need to fix it. Put it somewhere people will actually look.
Don't try to fix everything at once. You'll burn out and nothing will get better. Fix one thing, prove it works, then move to the next.
The goal isn't perfect automation. The goal is automation that doesn't surprise you at the worst possible moment.