I've been running AI agents in production for about three years now. Not because I'm an early adopter—I hate being an early adopter—but because clients kept asking and I got tired of saying no without knowing what I was saying no to.
Here's what I've learned: AI agents are useful tools. They're not magic. They're not going to replace your job, your team, or your brain. But they'll absolutely amplify what you can do—if you understand what they're actually good at and where they'll wreck your day if you look away.
This article is for the 9-5 sysadmin, the infrastructure lead, the builder who's tired of hype. I'm going to show you where agents work, where they break, and how to keep them on a leash without micromanaging them into uselessness.
—
What AI Agents Actually Do Well
Let me cut through the noise. Agents are good at three things:
1. Repetitive tasks that have clear inputs and outputs. If you've got a process that runs the same way every time with minor variations, an agent can handle it. I'm thinking log rotation scripts that need adjustment based on disk usage patterns, or ticket categorization that routes incoming requests to the right queue. The agent doesn't get bored. It doesn't start cutting corners on the 47th iteration.
2. Cross-system glue work. This is where agents shine. You know that integration between your monitoring system and your incident management tool that nobody built because it would take two teams and six sprints? An agent can write that glue code, maintain it, and flag when it breaks. It's not glamorous work, but it's work that never gets done because it's never "priority enough."
3. First-pass drafting and research. Agents are decent at pulling together information from multiple sources, summarizing docs, generating first drafts of runbooks, or drafting config file templates. They're not writing your production documentation—but they'll get you 70% of the way there so you're editing instead of starting from zero.
The common thread: agents are good at bounded, repetitive work where you can verify the output without needing deep context. They excel at "do this exact thing, every time, check the result" tasks.
—
Where Agents Fall Apart
Now the important part—what breaks.
1. Context that lives in your head. Agents don't know what you know. They don't understand that the "test environment" is actually production because someone misnamed it in 2019. They don't know that the VP skips the approval process when it's 4pm on Friday. Every assumption an agent makes that isn't in the prompt is a potential fire. You will debug agents making wrong assumptions more than you'd expect.
2. Anything requiring judgment about real-world consequences. An agent can generate a rollback script. It cannot decide whether you should run it at 2am on a Tuesday. It can flag that disk usage hit 90%. It cannot decide whether to page someone or wait until morning. These decisions require experience, risk assessment, and organizational context that lives in humans, not in tokens.
3. Tasks without clear success criteria. If you can't describe what "done" looks like, your agent will make something up—and you'll hate the result. Agents need boundaries. They're great at "extract these three fields from these emails and put them in this spreadsheet." They're terrible at "make this better."
4. Long-running processes with failure recovery. An agent that runs for 30 seconds is predictable. An agent that runs for 3 hours across 40 steps will fail in ways you haven't imagined. Network timeouts, API rate limits, partial state, race conditions—the failure modes are real, and they're not theoretical. I've had agents get stuck in loops, call the same API 200 times because of a bad retry condition, and corrupt data because a step failed silently.
—
The Maintenance Reality Nobody Talks About
Here's what the AI vendors don't tell you: agents are like pets. They need maintenance.
Every agent I've deployed needs ongoing care:
- Prompt tuning. What worked last month stops working when an API changes or your data format shifts. You'll be tweaking prompts quarterly, minimum.
- Output validation. You cannot trust agent output without checking it. I don't care how good the model is. I don't care what the marketing says. You will build validation layers, and those layers will catch problems the agent couldn't see.
- Failure monitoring. When your agent breaks, it often breaks silently. You need to know. I set up explicit "I succeeded" and "I failed" check-ins with notifications. If the agent stops checking in, I know something's wrong before the downstream system screams.
- Version drift. Your agent's behavior will drift over time as models update. What generated perfect YAML in January might add extra fields in March. You're locked in an arms race with your own tools.
This is the work nobody celebrates. It's not sexy. But it's the difference between an agent that runs for a week and an agent that runs for a year.
—
A Real Example: Automating Server Status Checks
Let me give you a concrete example from my own work.
I built an agent last year that checks server health across a client's infrastructure. It pulls metrics from three different monitoring systems, compares them against baseline thresholds, and generates a morning report. Before the agent, someone spent 20 minutes every morning manually checking dashboards. Now the agent handles it in about 90 seconds.
Here's what the agent does well:
- Connects to three APIs without complaint
- Formats the output into a readable report
- Flags anomalies based on configurable thresholds
- Runs on a schedule without being asked
Here's where I keep a leash:
- The agent cannot reboot servers. Ever. That decision stays human.
- If more than 3 servers show warnings, the agent pings me before sending the report. The threshold forces a human to look at cascading failures.
- Output validation checks that the report contains the expected sections. If the API returns garbage, the agent tells me it failed instead of sending a broken report.
The leash is simple: the agent does the tedious gathering and formatting. I do the thinking. That's the split that works.
—
Failure Modes You'll Hit
Let me save you some debugging time. You'll see these:
The "everything looks fine" failure. The agent completes its task successfully—but the output is wrong in a way it can't detect. Wrong threshold, wrong server, wrong format. You'll learn to validate outputs for correctness, not just presence.
The infinite loop. The agent hits a condition it doesn't know how to handle and keeps trying. I've seen agents call the same failing API 500 times because the retry logic didn't have a max attempt count. Set hard limits. Always.
The permission escalation. The agent does something with credentials that worked in dev but fails in prod because the prod service account has fewer rights. Test in the actual environment with actual permissions before you trust the agent in production.
The silent data corruption. The agent writes to a database with a subtle schema mismatch. Rows get written. They're just wrong. You won't notice until someone complains three weeks later. Write-only agents without read-back validation are dangerous.
—
What I Would Do First
If you're thinking about deploying an agent, here's where I'd start:
- Pick one boring, repetitive task. Don't try to automate your entire incident response. Find something that takes 15 minutes a day and always runs the same way. Documentation updates. Log rotation checks. Ticket triage.
- Build the leash before the agent. Decide what the agent cannot do. Write those boundaries down. Test that the boundaries hold.
- Validate everything. Build output checking even if it feels redundant. Catch the failures yourself before your users do.
- Start with a short leash. Let the agent run, watch it closely, and tighten the constraints based on what you see. You'll learn more from two weeks of observation than from any documentation.
- Accept the maintenance. Budget time every month for agent updates, prompt tuning, and failure investigation. If you can't maintain it, don't deploy it.
Agents are useful. They're not magic. They'll amplify your work if you give them clear boundaries and actually watch them run. The people who get value from agents aren't the ones who trust them most—they're the ones who understand exactly where the leash needs to be.
Start small. Watch everything. Adjust as you go. That's it.