You’ve seen it: a 400-line Bash script named backup_and_notify.sh, tucked into /opt/scripts, running nightly via cron. You call it a “system” because it feels like one—multiple stages, error handling, Slack alerts. But call it that too often, and you stop seeing the cracks.
Here’s the difference, in operational terms:
- A script is a sequence of steps that runs once, finishes, and leaves the system in a known state.
- A system is a persistent capability that absorbs change, handles failure gracefully, and continues operating—even when parts break.
That backup script? It’s a script. It’ll run fine until it doesn’t—until a new filesystem type appears, or curl gets blocked by a firewall, or the Slack webhook rotates credentials and the script silently fails. You’ll only notice when backups stop and the audit review is next week.
Let’s talk about why this distinction matters—especially when you’re tired, understaffed, and expected to “automate everything.”
—
Scripts Are Fast. Systems Are Survivable.
A script is like a checklist you write on a napkin: “1. Stop service. 2. Backup DB. 3. Copy to S3. 4. Restart service.” It’s fast, it’s clear, and it works—until the checklist assumptions change.
A system, by contrast, has feedback loops. It knows if it succeeded. It knows if the world changed out from under it.
Example: I once inherited a CI pipeline that deployed to 50+ microservices. The “pipeline” was a Python script that called kubectl apply in a loop. It ran fine for months—until one namespace had a network partition. The script retried five times, then exited with code 0 because the last service deployed. All others were half-deployed. No rollback. No visibility. Just a broken cluster.
What we built instead was a deployment system:
- A state tracker (etcd-backed) that recorded intent vs. reality
- A reconciler loop that compared current state to desired state every 30 seconds
- An alerting hook that fired if drift exceeded thresholds for >5 minutes
- A dry-run mode that never touched production but validated the plan
The script took 12 lines. The system took 800. But the system kept working during a major AWS outage last winter—while the old script would’ve left half the cluster offline.
The difference? The system waited. It retried intelligently. It told you when it was stuck.
—
The Maintenance Tax Hits Scripts Hard
Scripts don’t decay—they just become brittle.
Every time you change a dependency (API version, config format, auth method), a script needs a manual patch. Systems absorb that change through abstraction layers and versioned contracts.
I ran a content site for years. Early on, I used a script to fetch SEO metrics from Google Search Console, parse JSON, and email a weekly summary. Worked great—until Google changed their API rate limits, then changed the response schema without deprecation warnings.
The script broke. I patched it. Broke again. Broke again. By the third time, I had three versions of the script in three branches, each tied to a specific date range.
So I rebuilt it as a data pipeline system:
- A thin adapter layer that isolated the Google API calls
- A schema validator that logged mismatches but didn’t crash
- A cache layer (Redis) so partial failures didn’t lose the day’s data
- A config file (YAML) that defined what to fetch, not how
Now, when Google changes their API, I update one adapter and test it against a local mock server. The rest of the system doesn’t care.
This isn’t “enterprise overkill.” It’s time insurance. You’re not paying for a system to be perfect—you’re paying so it doesn’t break when you’re on call at 2 a.m.
—
When “System” Means “Don’t Reinvent the Wheel”
Here’s a truth no one admits: most “systems” you build should be built on top of existing systems.
You don’t need to write your own scheduler. Use cron, systemd timers, or a proper orchestrator (like Argo Workflows). You don’t need to write your own alerting. Use Prometheus + Alertmanager. You don’t need to build a config store—use Consul, etcd, or even a versioned YAML repo with drift detection.
I tried building a “lightweight” task queue in Python for a small project. It worked—until two tasks collided on shared state. Then I added locking. Then retries. Then dead-letter queues. Then monitoring. Six weeks in, I had half a RabbitMQ clone.
I replaced it with RabbitMQ. Took 3 hours. My code went from 1,200 lines to 120—and I stopped debugging race conditions.
Rule of thumb: If your “system” needs any of the following, lean on existing infrastructure:
- Retries with backoff
- Circuit breakers
- State persistence
- Metrics or tracing
- Multi-node coordination
These are hard. The people who built those tools have spent years fixing edge cases you haven’t hit yet. Use them.
—
The Hidden Cost of “It Works on My Machine”
A script passes the “works on my machine” test. A system must pass “works on your machine, next year, with new staff.”
I watched a team build a homegrown backup system using rsync and Bash. It worked for 18 months. Then:
- They upgraded to a new NAS with ACLs enabled
- The script didn’t handle
setfaclfailures gracefully - A critical directory was left without execute permissions
- The next reboot failed silently
The root cause? The script used set -e but didn’t log which command failed. Debugging took 4 hours.
A system would’ve:
- Logged each step with timestamps and exit codes
- Had a health-check endpoint (
/healthz) that confirmed backups were recent and readable - Used
rsync --checksum --archive --verboseso changes were visible in logs - Integrated with a monitoring system to fail noisily if backup age exceeded SLA
It’s not about adding features. It’s about adding visibility and recoverability.
—
What I Would Do First
You don’t need to refactor everything tomorrow. Start small. Pick one script that’s caused you pain in the last 6 months—especially one you’ve patched more than twice.
Ask these questions before rewriting:
- What breaks it? List 3 ways this script fails silently.
- Who notices? Is it you? Or does it only fail after a user complains?
- What if it ran twice? Does it idempotently handle re-runs? (Spoiler: Most scripts don’t.)
- What does “done” look like? Does it leave a trail you can inspect later?
Then, pick one system property to add:
- Logging: Add timestamps, status codes, and a unique run ID to every log line. (Use
loggeror JSON stdout.) - Health checks: Add a file or endpoint that says “last successful run: 2026-04-03T14:22:11Z”
- Idempotency: Make the script safe to run multiple times (e.g.,
rsync --deleteinstead ofcp -r) - Alerting: Wrap it in a wrapper that sends a Slack message only on failure, with the error snippet.
For example, here’s how I wrapped that backup script:
#!/bin/bash
RUN_ID=$(date +%s)-$$
LOG_FILE="/var/log/backup-${RUN_ID}.log"
# Log everything, including exit codes
exec > >(tee -a "$LOG_FILE") 2>&1
echo "[INFO] Starting backup (run $RUN_ID)"
# ... actual backup steps ...
# At the end:
if [ $? -eq 0 ]; then
echo "[INFO] Backup complete (run $RUN_ID)"
echo "$(date -Iseconds) SUCCESS" > /var/run/backup_status
else
echo "[ERROR] Backup failed (run $RUN_ID)" >&2
echo "$(date -Iseconds) FAILED" > /var/run/backup_status
# Only alert on failure
curl -X POST "$SLACK_WEBHOOK" -d '{"text":"Backup failed: '"$RUN_ID"'"}'
exit 1
fiThat took 10 minutes. It didn’t rewrite the backup logic—but it turned a black box into something you can debug in under 5 minutes.
—
Next Steps
- Audit your scripts: Run
find /opt -name "*.sh" -mtime -180. Pick one. - Add logging and status: Use the pattern above—just 5 lines of boilerplate.
- Test failure modes: Kill a service mid-run. What happens? Does it clean up? Does it leave partial state?
- Measure: Track how often it breaks, and how long it takes to fix.
You’ll know you’ve crossed the line into a system when:
- You can explain its behavior to someone new in under 10 minutes
- It fails noisily instead of silently
- You trust it more than your memory
That’s not magic. It’s just care. And in infrastructure, care is the only thing that scales.
The goal isn’t to build “systems” for ego—it’s to build things that outlive your shift.