Can You Build AI SRE? Why DIY Agents Fail in Production
Sang Lee
October 22, 2026
It's 2:47 AM. Checkout error rate just crossed 7%. Your homegrown AI agent, the one that nailed the demo last quarter, posts a confident summary in Slack: the database looks unhealthy, probably a noisy neighbor. The on-call engineer spends 40 minutes chasing it. The real cause was a connection pool change in a deploy that went out four minutes before the first error.
So, can you build AI SRE yourself? You can build the demo in a week. Building the system that is right at 3 AM, every night, is a different project.
Direct Answer: Yes, you can build AI SRE in-house, but most DIY agents fail in production for seven predictable reasons: missing tribal knowledge, telemetry without context, plausible but wrong root causes, no evals, investigations that die at handoff, agents that only start after a human is paged, and unsafe production access. Closing those gaps is a multi-quarter platform build, not a prompt.
Overview
- What "building AI SRE" actually means and where the demo stops
- Why production is a different problem than AI coding
- The seven reasons DIY AI SRE agents fail, and how Tony SRE handles each one
- What it costs to build and maintain an AI SRE in-house
- Build vs. buy, and when building still makes sense
- Common mistakes teams make on the DIY path
- FAQ for engineering leaders evaluating AI SRE
Can you build AI SRE in-house?
Yes. Wire a frontier model to your Datadog, GitHub, and Kubernetes APIs, give it a system prompt and a few tools, and it will investigate an alert well enough to impress a room. A capable engineer can get there in days.
The problem is what the demo hides. A demo runs on one engineer's laptop, against a handful of alerts that engineer already understands, with nothing at stake. An AI SRE (an AI agent that investigates alerts, finds root cause, and coordinates incident response) runs against every alert, for every team, at 3 AM, with revenue on the line.
The table shows the core gap: a DIY demo proves a model can reason about an alert, while a production AI SRE has to be correct, shared, governed, and affordable across the whole organization. Everything below is about the distance between those two columns.
Takeaway: Building an AI SRE demo is easy. Building one your on-call engineers trust is a platform.
Why AI SRE is harder than AI coding
AI coding agents work because code is a cooperative input. It is bounded, typed, versioned, and testable in seconds. Production is none of those things, so the same model that writes good code produces confident but shallow incident analysis.
The contrast in the table explains why AI SRE is not just "AI coding, but for ops": production has no compact, self-describing input and no fast pass or fail signal, so the model needs a system around it that supplies context and checks its work.
The stakes are real. Splunk and Oxford Economics estimate that downtime costs Global 2000 companies $400B a year. And the work itself is the kind of repetitive, interrupt-driven load that Google's SRE book defines as toil, which is exactly why teams want to automate it.
Takeaway: A model that is great at code is a starting point for AI SRE, not the finished product.
7 reasons DIY AI SRE agents fail in production
These failure modes rarely show up one at a time. In a real incident they stack, which is why a DIY agent that looked solid in testing falls apart on its first bad night.
1. The knowledge your agent needs was never written down
In most orgs, the same five engineers solve every hard incident. What makes them good is not in a runbook: the batch job that always spikes memory at 2 AM, the Helm default that silently resets a pool limit, the service everyone knows not to restart. A DIY agent only sees what its builder fed it, usually one team's docs and prompt files.
Worse, whatever you do index starts decaying the week you index it. Services get renamed, ownership moves, and a stale runbook is more dangerous than none because the agent trusts it.
How Tony SRE handles it: Tony learns from your team's conversations, service mapping, and past incidents, and every correction an engineer makes becomes memory. When someone asks "have we seen this before?", Tony can point to the earlier incident and the fix the original engineer applied. That knowledge stays when your senior engineers rotate off or leave.
2. Telemetry is not context
The default DIY move is to give the agent more logs. But graphs show what broke, not why your team fixed it that way last time. The deciding evidence is often a Slack thread, a deploy note, or a comment from another team months ago.
How Tony SRE handles it: Tony reads logs, metrics, deploys, and Slack threads together, through 3,000+ integrations including Datadog, New Relic, GitHub, Jira, ServiceNow, and Backstage. Context beats more logs.
3. The Tidy Answer Trap
The Tidy Answer Trap: a DIY agent returns a clean, well-cited explanation where every piece of evidence is real, but the conclusion is wrong because the root cause sits upstream of everything it checked.
General-purpose models are tuned to produce the most coherent answer, and incident investigation punishes coherence without causality. Nothing in the answer is hallucinated, so spot-checking the citations passes. A senior engineer distrusts a tidy answer; a newer on-call engineer acts on it.
The result is either a wrong fix during a live incident, or an engineer who ends up verifying the agent's first guess instead of investigating. Either way, trust erodes and the tool quietly stops getting used.
How Tony SRE handles it: Tony starts the second the alert fires and returns a root cause with the evidence attached: the deploy diff, the pool metrics, the error logs. At Shutterstock, an incident that looked like it was three hours old turned out to have started 11 days earlier. Tony found it.
4. No evals, so every change is a guess
Without an evaluation harness, you cannot tell whether your agent is right 50% or 90% of the time, or whether last week's prompt change made it better or worse. Every model upgrade becomes a blind migration. A useful eval program needs to check the answer, the path the agent took to get there, whether it abstained when evidence was missing, and whether its stated confidence matches reality.
That is research-grade ML infrastructure, and it needs to be regenerated continuously as your systems change.
How Tony SRE handles it: [Confirm with product before publishing: how Vibranium evaluates Tony's root cause accuracy and calibration, and how it regression-tests model upgrades.]
5. Investigations die at the handoff
Most DIY agents are single-player. The investigation lives in one engineer's chat session, so what the agent learned at 2 AM is gone by the 4 AM handoff. Production incidents are multiplayer: they pull in owners from several teams, and someone has to run the room.
How Tony SRE handles it: Tony acts as incident commander. It opens the incident channel, declares severity, assigns roles, pulls in service owners with a summary, posts status updates on a fixed cadence, and syncs the timeline to Jira. The postmortem draft is ready when the incident closes, and the investigation lives where the whole team can see it.
6. The Post-Page Gap
The Post-Page Gap: the time between an alert firing and a useful investigation starting, because the DIY agent only runs after a human has already been paged, woken up, and asked it a question.
Almost every DIY build bolts the agent onto an existing pager. PagerDuty or Opsgenie wakes a human first, the human opens a laptop, and only then does the agent start working. You have automated part of the investigation without changing who gets woken up, or when.
How Tony SRE handles it: Tony runs on Vibe OnCall, which has paging built in: schedules, rotations, escalation policies, handoffs, and mobile alerts. It replaces PagerDuty and Opsgenie, and Vibranium migrates your schedules for you. Because paging and investigation live in one system, Tony takes the page first and your engineers make the decisions.
7. Production access without production-grade guardrails
An agent with production access is a new security principal. Most DIY builds run it under a human's credentials or a broad service account, log tool calls but not the reasoning behind them, and have no policy for which actions need approval. That works in a demo and fails a security review.
This maps directly to what OWASP calls excessive agency in LLM applications: an agent with more permissions or autonomy than the task requires.
How Tony SRE handles it: Tony runs on a dedicated AWS Bedrock AgentCore runtime with per-customer memory isolation. Vibe OnCall is SOC 2 compliant, stores no credentials (not even a token), and keeps data auditable and deletable. You decide what Tony can reach, and Tony asks before anything irreversible. In a suggested workflow, read-only steps run automatically while a rollback is marked as needing approval.
What it costs to build AI SRE in-house
The expensive part of a DIY AI SRE is people, not tokens. Most teams budget for model usage and forget that the system needs a permanent team to keep it accurate as models, services, and owners change.
Seven of the eight cost areas in this table are permanent, which is the main reason DIY AI SRE costs more in year two than teams expect: the build ends, but the maintenance, evals, and compliance work never do.
A simple way to size it: multiply the steady-state headcount you would need by your fully loaded engineering cost. As an illustration, if you assume four engineers to maintain the agent, its evals, and its integrations, at an assumed $250K fully loaded cost each, that is about $1M a year before infrastructure, tokens, or your pager contract. Swap in your own numbers; the point is that the line item is headcount.
Takeaway: If you build AI SRE, budget it like a product team, not a project.
Build vs. buy AI SRE
For most teams, buying wins on time to value and total cost; building wins only when AI operations is itself your product or your environment cannot use a vendor.
The comparison comes down to ownership: building means your team owns the agent, its knowledge pipeline, its evals, and a separate pager indefinitely, while buying moves that work to a vendor and leaves your engineers making the decisions.
When building still makes sense
- AI operations tooling is your product, and the harness is your differentiation
- Regulatory or air-gapped constraints rule out any external vendor
- You already run a mature ML platform team with eval infrastructure in place
- You want a narrow, single-team assistant and accept that it will not scale org-wide
Takeaway: Build the parts that differentiate your business. For most teams, the AI SRE harness is not one of them.
What not to do when building AI SRE
If you do build, avoid the mistakes that sink most DIY attempts.
- Don't judge it by the demo. Test against last quarter's real incidents, including the ones your team got wrong at first, not the alerts the builder already understands.
- Don't fix accuracy with more context. Stuffing the window with logs raises token cost and still misses the Slack thread that held the answer. Retrieve what the current hypothesis needs.
- Don't run it under a human's credentials. Give the agent its own identity, scoped read and write paths, and an audit trail that records why it acted, not just what it called.
- Don't ship without a way to say "I don't know." An agent that never abstains will eventually hand a newer engineer a confident wrong answer during a Sev1.
- Don't leave the pager out of the design. If the agent only starts after a human is paged, you have not changed the 3 AM experience. You have added a chatbot to it.
- Don't treat it as a one-time project. Models, services, and owners change every month. Without a standing owner, accuracy decays quietly.
How Tony SRE closes the gap
Tony SRE is the AI twin of your smartest on-call engineer, built on Vibe OnCall. Instead of solving one of the problems above, Tony takes on four roles in one hire:
- Smartest on-call engineer: picks up the alert the second it fires and brings back the root cause, with evidence.
- Fastest incident commander: sets severity and roles, pulls in owners, posts status on a clock, and drafts the timeline and postmortem.
- Always-available subject matter expert: knows every service, owner, past incident, and fix, and stays when senior engineers leave.
- Most proactive SRE: spots repeat incidents, flags missing runbooks, and turns repeat fixes into workflows you approve in one click.
Tony lives in Slack or Microsoft Teams and is entirely conversational: plain-English instructions become workflows, and corrections become memory. See our guide to what a Tier 0 on-call layer is for how investigation before paging works.
In production at Shutterstock
Shutterstock's incident process was already strong. The slow part was figuring out which systems were hit, how far the impact spread, and who owned what. That is the part Tony took over, across 300+ engineers.
- 70% fewer manual incident minutes
- 60% reduction in MTTD and MTTR (mean time to detect and mean time to resolve)
- 11 days: an incident that looked three hours old had actually started 11 days earlier, and Tony found it
Read the full Shutterstock case study.
Takeaway: Your best engineer cannot be on every incident. Tony SRE can. Vibranium is opening an exclusive beta to 20 teams in Q4 2026. Join the beta or book a demo.
FAQ
Can you build AI SRE with an off-the-shelf LLM?
Yes, you can build a working prototype with any frontier model and a few API tools. What an off-the-shelf model lacks is your organization's context, a way to verify its own root cause, evals, shared incident state, and production-grade access controls. Those are what take quarters to build.
Why do DIY AI agents give confident but wrong root causes?
General-purpose models are optimized to produce the most coherent answer, not the most causal one. They anchor on the first plausible explanation and cite real evidence for it, which is why the error survives a quick citation check. We call this the Tidy Answer Trap.
How long does it take to build an AI SRE in-house?
A demo takes days; a production system takes multiple quarters and a permanent team. The initial build is usually one to two quarters, and maintenance, evals, and compliance continue indefinitely after that.
Does an AI SRE replace on-call engineers?
No, it changes what on-call engineers spend time on. Tony SRE takes the page, investigates, and coordinates the incident, while engineers make the decisions and approve anything irreversible.
Do I still need PagerDuty or Opsgenie with an AI SRE?
With most AI SRE tools, yes, because they investigate but do not page. Tony SRE runs on Vibe OnCall, which includes schedules, escalations, and mobile alerting, so it replaces PagerDuty and Opsgenie, and Vibranium migrates your schedules.
Is it safe to give an AI agent production access?
It is safe only with scoped identities, auditing, and approval gates for risky actions. Tony SRE runs with per-customer isolation on a dedicated runtime, stores no credentials, and asks before anything irreversible.
