Quick Summary
This insight explains how Bacancy set up Automated Incident Response with Azure SRE Agent for an Australia-based SaaS company running container apps on Azure, handling the first response on their production alerts. It covers the problem their on-call team faced, what we configured, how we kept it safe, and the results they saw in the first three months.
Table of Contents
Microsoft addressed a challenge that many Azure teams have been dealing with for years when it introduced Azure SRE Agent. Most organizations already have skilled engineers who can respond to incidents, but identifying the root cause often takes longer than the actual fix. The longer the investigation phase, the longer the disruption, the higher the operational cost, and the greater the burden on on-call teams.
That was exactly the situation one of our clients was facing. Their team had no shortage of experience and could handle incidents effectively, but too much time was being spent collecting information, analyzing logs, and narrowing down the source of the problem. They approached us shortly after Azure SRE Agent was launched with one objective, i.e., automate a significant part of their incident response workflow and reduce the time spent on diagnosis.
Before introducing the solution, we carried out an Azure environment assessment. While reviewing their setup, we identified a few architectural issues that could affect both reliability and the effectiveness of automation. We documented our findings and shared practical recommendations to address them.
The engagement eventually extended beyond implementing Azure SRE Agent. Alongside the automation rollout, we helped the client address the improvements identified during the assessment. As a result, they moved toward Automated Incident Response with Azure SRE Agent and also strengthened the overall health of their Azure environment.
Azure SRE Agent sits on top of existing monitoring tools rather than replacing them. It doesn’t generate alerts itself; instead, it responds when alerts are triggered by the current observability stack. Our role was to position the agent correctly within this workflow and ensure it could act immediately when incidents were detected.
Since its effectiveness depends entirely on available context, most of the setup focused on connecting it to the same sources that engineers use during incident investigations.
We set up the following connections for the client:
The starting point was their incident platform, i.e., Azure Monitor, which is where their alerts already came from. Once we connected it, the agent picked up an incident the moment it fired, without anyone needing to pass it along.
We configured the agent with read-only access to the client’s Azure resources. This lets it query metrics, check resource health, and read configurations across the services it would investigate, without allowing it to make changes at this stage.
Reading the resources showed the agent what state things were in, but most incidents only start to make sense once you look at the logs. By connecting Log Analytics and Application Insights, the agent could work from the same telemetry and diagnostic data that their engineers used during an incident.
For the cases where the logs pointed to the application itself, the agent needed to see the code behind it. We connected their repositories, which made the biggest difference to how well it investigated, since it could now follow an error to the file and the recent change that caused it rather than stopping at the symptoms.
The last piece was the knowledge that the team had built up over time. We added their runbooks and documentation, so the agent’s investigations followed how their systems were actually built and run.
A few of their tools sat outside Azure, such as a non-Microsoft monitoring platform. For those, the agent connects through the Model Context Protocol, which gives us one consistent way to add a tool whenever the client needs it, without a separate integration each time.
A connected agent still needs rules, it would treat every alert the same way. We set those rules through response plans, which decide which incidents the agent acts on and how it responds to each one. Each plan does two things, and the two work together.
The first job of a plan is to narrow down what the agent responds to. We scoped each one by severity, service, and title, so the agent acted on the incidents that mattered and left routine noise alone.
Once a plan matched an incident, it also decided who should handle it. Different problems call for different expertise, so we could send a database incident to a setup built around database work, and a deployment issue to one built around their code and release history.
Each plan also carries an autonomy level, which decides how far the agent can go on its own. That setting was central to keeping the agent safe, so we cover how we handled it in the section below.
For the client to rely on the agent, they needed to see what it was doing, so we treated visibility as part of the setup rather than something added later. The agent’s Operations Hub gave them a single dashboard for this. It shows the incidents the agent handled, the state of every connection, and anything waiting on a person’s approval, so a gap in the setup or a failing connection shows up here early instead of during the next incident.
Hire Azure developers from Bacancy to connect, configure, and roll it out safely.
With the setup in place, the agent took the first response to an incident. It moves through the same steps each time: it acknowledges the alert, gathers context, works out the cause, checks what it has seen before, applies and verifies a fix, then reports back and records what it learned.
The agent acknowledges the alert within seconds of it firing. The acknowledgement marks the incident as taken, so two people do not start working on the same problem.
The agent queries every connected source at the same time rather than one after another. The logs, metrics, deployments, and resource health come together before it begins reasoning, so it works from the full picture.
The agent forms a few explanations for the failure, then tests each against the evidence and drops the ones that do not hold. It works through them until the evidence points to one cause, and keeps the reasoning trail so the team can follow it.
The agent checks its memory of earlier incidents as it investigates. If the same symptoms appeared before, apply the fix that worked then, so a known problem does not need a fresh investigation.
Once it reaches the cause, the agent either carries out the fix or proposes it for approval, depending on the response plan. After the fix is in place, it checks that the problem is resolved rather than assuming the action worked.
The agent sends the team a summary with the cause, the impact, and the action it took or recommended. It also records the symptoms, the cause, and the steps that worked, so that knowledge is ready next time.
Handing an agent the ability to act on production is not something a team does on trust alone, and the client was right to be careful about it. We built the rollout so the agent earned that trust in stages, and so the riskier an action was, the harder it was for the agent to take it without a person involved.
We put the agent on their live production incidents from the start, but kept it on a tight rein and loosened that only as it proved itself:
The agent also sorts every action by how risky it is, which gave us a second layer of control on top of the run mode:
This meant the actions that could affect production always had a human gate in the early stages, regardless of how routine they seemed.
Some boundaries do not depend on configuration, because the agent enforces them itself:
Every action the agent takes is logged to Application Insights, down to each command it runs and each approval a person gives. The client has a complete record of what the agent did, when it did it, and what led to it. This made the agent’s work easy to review after the fact, and it gave the team the evidence they needed to keep extending the agent’s autonomy with confidence rather than on faith.
Get Azure managed services from Bacancy to run, tune, and watch over it.
The client’s main goal was to reduce the time spent diagnosing incidents. Operations Hub gave them a complete view of what changed.
| Measure | Before | After |
|---|---|---|
| Time to find the cause | 30 to 60 minutes of manual work | Around 4 minutes, gathered automatically |
| Incidents resolved by the agent alone | None | 47%, with the rest investigated and handed over |
| Median time to mitigate | About 70 minutes | Around 18 minutes, shown side by side |
| Engineering time saved | - | Roughly 90 hours over the first three months |
| Quality of resolutions | Varied by who was on call | 4.3 out of 5, scored automatically |
For the team, the biggest shift was in how incidents started. Engineers were no longer spending their first 30 or 40 minutes collecting logs, checking telemetry, and piecing together what had happened. Much of that groundwork had already been done before they got involved.
The Automated Incident Response with Azure SRE Agent rollout also addressed issues outside of incident management. Several architectural improvements identified during the assessment were implemented before rollout, helping create a stronger foundation for the automation that followed.
This insight covered how we set up Automated Incident Response with Azure SRE Agent for an Australia-based SaaS company running container apps on Azure. The work came down to a few clear stages:
The outcome was an incident response that found the cause in minutes rather than hours, and a team that no longer spent the start of every incident working out what had gone wrong.
The Azure SRE Agent can do a great deal, but that capability only turns into results once it is set up around a specific environment and run with the right guardrails in place. If you are planning to bring it into your own Azure setup, you can opt for services from Azure Consulting Company to run that assessment first and build the automation on a solid footing.
Azure SRE Agent is an AI agent from Microsoft that helps investigate production incidents, identify root causes, recommend or apply fixes, and reduce operational toil across Azure environments. It acts on existing alerts rather than monitoring on its own.
It acknowledges an alert the moment it fires, gathers logs and telemetry from connected sources, tests possible causes, checks past incidents, then applies or proposes a fix depending on its permissions. Once done, it reports a summary of the cause, impact, and action taken.
No. It works alongside tools like Azure Monitor, Log Analytics, and Application Insights, using the alerts they generate rather than replacing them. The agent has no monitoring of its own.
It connects to your incident platform, Azure resources, logs and telemetry, source code repositories, and runbooks. For tools outside Azure, it connects through the Model Context Protocol (MCP).
Yes, when set up with the right guardrails. You can start in Review mode, where the agent investigates and proposes actions that need human approval before anything changes, then widen its autonomy as it proves reliable.
In Review mode, the agent proposes changes and waits for a person to approve them. In Autonomous mode, it carries out the response on its own for the incident types you have approved for it.
No. All Key Vault commands are blocked, so the agent has no path to credentials. It acts only within the permissions assigned to its managed identity.
It works hypothesis-driven rather than searching at random. The agent forms a few possible explanations, tests each against logs, metrics, deployments, and resource health, and discards the ones that do not hold until the evidence points to one cause.
Yes. It keeps a memory of past incidents, including the symptoms, cause, and steps that worked, and applies that knowledge when the same problem appears again, so it does not investigate a known issue from scratch.
Depending on its permissions and response plans, it can investigate incidents, send notifications, and make changes such as restarting or scaling resources. Riskier actions need approval in Review mode, and some actions, like deletes, are blocked entirely.
It depends on the size of the environment, the integrations involved, and the governance required. Most rollouts begin with an assessment, then connection and configuration, a review period, and a phased move toward autonomy.
It speeds up root-cause analysis, lowers mean time to resolution, reduces operational overhead, and keeps incident handling consistent. It also eases the load on on-call engineers by taking the first response.
Yes. Through MCP, it can work with supported third-party monitoring, observability, and operational tools, which gives one consistent way to extend its reach without a separate integration each time.
Yes. Every command it runs and every approval a person gives is logged to Application Insights, so you have a full record of what it did, when, and why, for review and compliance.
An assessment surfaces architectural issues, monitoring gaps, and permission needs that would otherwise limit how well the automation works. Fixing these first means the agent runs on a healthy, predictable setup.