RT Robert Truesdale

May 19, 2026

Scripts Run, Alerts Fire, and Then What?

You've got a script. It runs on cron. You set up an alert when it fails. You're done, right?

Wrong. You've built the machine, but you haven't built the system around it. And that system—the part everyone forgets—is where production actually lives.

After 20 years of watching ops teams automate themselves into a corner, here's what I see happening: someone writes a backup script, adds a cron entry, configures a "backup failed" alert, and walks away proud. Then at 2am on a Sunday, the alert fires, whoever's on call has no idea what the script does or what to do about it, and the whole "automation" thing becomes a liability instead of a win.

This isn't about your script failing. It's about everything around it that you forgot to build.

—

The Monitoring Gap: "Running" vs. "Working"

Here's the first place automation projects die: the difference between "the script ran" and "the job succeeded."

Your cron job has an exit code. That's it. That's the whole story. If your backup script runs but runs out of disk space halfway through, cron sees exit code 0 and moves on. Nothing fires. Nothing alerts. You find out three weeks later when someone asks for a file from two months ago and it's not there.

You need to check the results, not just the run.

For a backup script, that means verifying the output file exists and has a reasonable size. For a data sync script, it means checking record counts or checksums. For an AI workflow processing content, it means validating that output actually got generated—not just that the script didn't crash.

What this looks like in practice:

#!/bin/bash
./run_backup.sh
if [ $? -ne 0 ]; then
  echo "Backup script failed" | mail -s "ALERT" [email protected]
  exit 1
fi

# Check the actual output
if [ ! -f /backup/daily.tar.gz ]; then
  echo "Backup file missing" | mail -s "ALERT" [email protected]
  exit 1
fi

# File exists but is it reasonable?
SIZE=$(stat -f%z /backup/daily.tar.gz 2>/dev/null)
if [ "$SIZE" -lt 1000000 ]; then
  echo "Backup file suspiciously small: $SIZE bytes" | mail -s "WARNING" [email protected]
fi

This isn't glamorous. It doesn't sound like AI. It is the work that keeps you employed.

—

The Documentation Deficit

You wrote the script. You know what it does. Your replacement won't. Neither will you, six months from now, when you're on call and the alert fires and you have to figure out fast.

I'm not saying write a novel. I'm saying write the four things that matter:

  • What this does – one sentence
  • What it depends on – filesystem, network, credentials, other scripts
  • What failure looks like – what actually breaks when this goes wrong
  • How to fix it – the first three things to check

Put this in a README next to the script. Put it in a runbook. Put it somewhere that survives you taking a sick day.

I've seen teams lose days to "who wrote this and what was it doing?" That time has a cost. Documenting isn't optional—it's the tax you pay for having automation.

—

Error Handling That Breaks in Production

Your script has a try/catch equivalent? Great. What does it do when something fails? Does it exit silently? Write to a log nobody reads? Send an alert that goes to a dead email address?

Here's what actually happens: your script hits an edge case you didn't anticipate. It fails. It sends an alert. The alert says "script failed" with no context. You fix it. Three months later, it fails again on a different edge case. You fix it again. This is not automation. This is bug whack-a-mole.

Build in explicit error handling that tells you what went wrong:

try:
    result = process_content(input_file)
    if not result.success:
        raise ProcessingError(f"Content processing failed: {result.error}")
except FileNotFoundError as e:
    raise ProcessingError(f"Input file missing: {input_file}") from e
except PermissionError as e:
    raise ProcessingError(f"Permission denied on output: {output_path}") from e

Same pattern in bash—capture the actual error, wrap it with context, pass it up the chain. When this fails at 2am, you want the error message to tell you exactly where to look.

—

Alert Fatigue Is Real

You added an alert for everything. Now you get fifty emails a day. You stop reading them. The real problem gets lost in the noise.

This is the part everyone forgets: alerts need tuning, or they become useless.

Ask yourself two questions about every alert:

  • Does this require human action? If the answer is no, it's a log entry, not an alert. Your script retrying three times and succeeding? That's a log entry. Your script retrying three times and still failing? That's an alert.
  • Will I act on this at 2am? If you're going to snooze it or ignore it, don't create it. Create it when you have a real response planned.

For a content-site automation workflow processing 200 articles a day, you don't need an alert when each individual article fails. You need an alert when the failure rate spikes—when suddenly 15% of articles fail instead of the normal 2%. That's an actionable signal. Individual failures are noise.

Cut your alerts by 80%. Your on-call self will thank you.

—

The Maintenance Reality

Your script ran fine for six months. Then it broke. Nothing changed in the script. Nothing changed in the system. It just… stopped working.

Welcome to maintenance reality. Things break not because you broke them, but because the world changed around them:

  • An API changed its response format
  • A certificate expired
  • A dependency updated and changed behavior
  • Disk usage grew and hit a threshold you didn't monitor
  • The third-party service you rely on changed without telling anyone

Your automation needs health checks, not just success checks. Run a sanity check periodically that validates the whole chain: credentials still work, dependencies respond, outputs are being generated. Catch drift before it becomes an outage.

This is why "set it and forget it" is a lie we tell ourselves. Automation shifts work from "do it manually" to "maintain the automation." That second part is where most teams underinvest.

—

What I Would Do First

If you're looking at your systems right now and realizing you've got a pile of scripts and alerts but no structure around them, here's where to start:

  • Pick one critical job – your most important automation. Backup, sync, whatever you can't live without.
  • Add result validation – verify the output, not just the exit code. Check file size, record count, whatever proves it actually worked.
  • Write the four-line README – what it does, dependencies, failure mode, how to fix it. Put it next to the script.
  • Tune your alerts – if you can't explain why you'd wake up for it, delete the alert. Make the ones that remain actionable.
  • Set a calendar reminder – six months from now, review this job. Check if credentials need rotating, dependencies changed, health checks still pass.

That's it. One job, done right. Then do the next one. This isn't a project you finish—it's a discipline you maintain.

The scripts will run. The alerts will fire. Make sure someone knows what happens next.