Autonomous Root Cause Analysis for AWS Systems Manager Patch Failures Using AWS DevOps Agent

Introduction

When AWS Systems Manager Patch Manager reports failures across hundreds of managed nodes spanning multiple accounts and regions, operations teams face a time-consuming investigation: logging into each account, correlating events, analyzing logs, and determining whether failures share a common root cause. A single patch cycle failure can consume hours of engineering time.

In this post, we show you how to build an event-driven solution that automatically triggers root cause analysis for patch failures using AWS DevOps Agent. When AWS Systems Manager reports a patch operation failure, it emits a status-change event containing the command ID, instance ID, and error status. Amazon EventBridge captures this event, an AWS Lambda function enriches it with SSM command output, instance metadata, and fleet-wide failure rates, and AWS DevOps Agent conducts an autonomous investigation by correlating logs, configuration state, and infrastructure changes to produce a root cause report within minutes.

Overview of solution

By the end of this walkthrough, you will have a fully deployed pipeline that:

• Captures patch failure events from AWS Systems Manager across multiple accounts and Regions
• Enriches each failure with SSM command output, fleet failure rates, and instance metadata
• Triggers AWS DevOps Agent to autonomously investigate and report root cause without manual log correlation

Before you begin, review the prerequisites below.

The solution uses a centralized architecture. Workload accounts forward patch failure events to a central primary account where the Lambda function and AWS DevOps Agent space reside.

Figure 1 – Solution architecture

The event flow works as follows:

1. AWS Systems Manager executes AWS-RunPatchBaseline on target managed nodes.
2. When a patch operation fails, Systems Manager emits a status-change event to Amazon EventBridge.
3. An EventBridge rule in the workload account forwards the event to the primary account’s event bus (cross-account, cross-region).
4. An EventBridge rule in the primary account invokes the Lambda function to deduplicate, enrich, and forward the failure for investigation.
5. Lambda checks Amazon DynamoDB for duplicate events (preventing repeated investigations for the same failure), enriches the event with fleet context by assuming a role back into the workload account, retrieves the HMAC (Hash-based Message Authentication Code) signing key from AWS Secrets Manager encrypted with the solution’s AWS Key Management Service (AWS KMS) key and sends an HMAC-signed request over HTTPS to the AWS DevOps Agent webhook endpoint. The HMAC signature authenticates the request, ensuring only your Lambda function – not arbitrary sources can trigger investigations in the Agent Space.
6. AWS DevOps Agent investigates autonomously and produces a root cause report.

Walkthrough

In this walkthrough, we deploy the solution and validate the end-to-end flow. The walkthrough covers the following steps:

• Create the AWS DevOps Agent space and generic webhook
• Deploy the primary account stack
• Deploy workload account stacks

You can download the AWS CloudFormation templates from the companion GitHub repository.

Prerequisites

For this walkthrough, you should have the following prerequisites:

• An AWS account with an active AWS DevOps Agent space
• Workload accounts registered as secondary sources in your Agent Space
• One or more workload accounts with Systems Manager managed nodes
• AWS Organizations (recommended for multi-account deployments) – you will need your Organization ID or a list of workload account IDs
• Permissions to create AWS CloudFormation stacks and StackSets

Step 1: Create the AWS DevOps Agent space

1. Open the AWS DevOps Agent console in your primary account.
2. Choose Create Agent Space.
3. In the Agent Space Name field, enter a name for your Agent Space (for example, patch-rca-production).
4. Give this Agent Space AWS resource access – AWS DevOps Agent requires an AWS Identity and Access Management (IAM) role to discover and access AWS resources in this account during investigations. For this post, let’s select Auto-create a new AWS DevOps Agent role. To configure a custom IAM role instead, see Configuring primary account access in AWS DevOps agent user guide.
5. Enable Web App – The Web App is where personnel interact with AWS DevOps Agent for incident investigations and reviewing recommendations. For this post, let’s select Auto-create a new AWS DevOps Agent role.
6. After completing all sections, choose Create.

Register each workload account where you patch instances by following these steps:

1. After the Agent Space is created, within the Capabilities tab – scroll to Cloud section, and under the Secondary sources, choose Add.
2. Select AWS within Add source account and choose Add.
3. Review the Role and permissions and select Next.
4. Provide the AWS Account ID of the secondary account and choose Add.

Next, configure the webhook by following the steps from AWS documentation:

1. Within the Capabilities tab of Agent Space – scroll to the Webhooks section and select Add webhook.
2. Now verify the data schema and select Next.
3. Review the instructions and select Next.
4. Choose Generate URL and secret key, then choose Download .csv file and save it in a secure location.
5. Note the webhook URL and HMAC (Hash-based Message Authentication Code) secret. You need these values in the next step.
6. Select I’ve saved and stored my URL and secret key, choose Add.

Step 2: Deploy the primary account stack

Deploy primary-account.yaml in the account where your Agent Space lives. This stack creates the Lambda function, EventBridge rule, Amazon DynamoDB deduplication table, AWS Secrets Manager secret, and AWS KMS encryption key.

To deploy the primary stack:

1. Download the CloudFormation templates from the companion GitHub repository.
2. Open the AWS CloudFormation console in your primary account.
3. Choose Create stack, then choose With new resources (standard).
4. Choose Upload a template file, select primary-account.yaml, and choose Next.
5. Enter a Stack name (for example, devops-agent-patch-primary).
6. In the parameters section, enter the webhook URL and webhook secret (HMAC secret) from Step 1.
7. Under OrganizationId, enter your AWS Organizations ID (for example, o-abc123) to allow workload accounts to forward events. Leave blank if providing explicit account IDs in SecondaryAccountIds parameter instead.
8. Choose Next, acknowledge the IAM capabilities, choose Next, and then choose Submit.

The following table describes the key parameters.

To use existing instance tags instead of adding new ones, set InvestigateTagKey and InvestigateTagValue to match your existing tags (for example, env and prod, or patch-group and critical-servers).

Step 3: Deploy workload account stacks

Deploy secondary-account.yaml to your workload accounts using AWS CloudFormation StackSets. It creates an IAM role for cross-account event enrichment and an EventBridge forwarding rule in each account and region you target, routing patch failure events to your primary account.

The workload stack creates the following IAM role:

DevOpsAgentPatchEnrichmentRole : assumed by the primary account’s Lambda function to gather context about the failed patch operation. It grants read-only access to:

• Amazon Elastic Compute Cloud (Amazon EC2) instance metadata and tags
• SSM command invocation output and history
• CloudWatch metrics and alarms

The role’s trust policy restricts access exclusively to the Lambda role in your primary account using an aws:PrincipalAccount condition. No other account or principal can assume this role.

This walkthrough uses service-managed permissions to create a CloudFormation StackSet. If you choose to use self-service permissions, the deployment steps may vary slightly.

To deploy using StackSets:

1. Open the AWS CloudFormation console in your management or CloudFormation delegated administrator account.
2. Choose StackSets, then choose Create StackSet.
3. For Permissions, choose Service-managed permissions.
4. For Specify template, choose Upload a template file, select Choose file, choose the secondary-account.yaml file, and then choose Next.
5. Enter a StackSet name (for example, devops-agent-patch-workload).
6. In the parameters section, enter your primary account ID and primary region (for example, us-east-1), and then choose Next.
7. Acknowledge the IAM capabilities and choose Next.
8. Under Deployment targets, choose Deploy to organization for full coverage, or choose Deploy to organizational units and enter specific OU IDs to limit scope.
9. Under Specify regions, select the region you specified as PrimaryRegion in step 6. If you patch instances in additional regions, you will add them after this deployment (see the steps below).
10. Leave the defaults for the other options and choose Next and then choose Submit.

Note: Amazon EventBridge events are Regional. Deploy the workload template in every Region where you run patched instances – the PrimaryRegion parameter routes events to your primary account automatically.

Adding Additional regions:

Multiple regions: If you patch instances in regions beyond the one deployed above, add them to the StackSet:

1. In the AWS CloudFormation console, choose StackSets and select the StackSet you created.
2. Choose Actions, then choose Edit StackSet details.
3. Leave everything as it is on the template page and choose Next.
4. On the Specify Stackset details page under Parameters section, change CreateIAMRoles to false and choose Next.
5. Leave everything as it is on the Configure stack set options page, acknowledge the IAM capabilities and choose Next.
6. On the Deployment options page, under Specify regions, add the remaining regions where you run patched instances.
7. Choose Next and then choose Submit.

The IAM enrichment role is global and was already created in the first deployment. Setting CreateIAMRoles to false ensures subsequent regions only create the regional EventBridge forwarding rule without attempting to recreate the existing IAM role.

How it works

After deployment, the Lambda function performs filtering and enrichment before triggering an investigation.

Filtering includes:

• Tag check: only instances matching the configured tag key/value proceed
• Deduplication: DynamoDB prevents duplicate investigations within a configurable window (default: 10 minutes)
• Per-command throttle: limits investigations per command ID (default: 5) to prevent floods
• Transient failure filtering: known benign patterns (reboot exits, lock contention) are skipped

Enrichment adds fleet context before sending the webhook:

• Fleet failure rate: how many instances in the same command failed versus succeeded
• Instance metadata: name, platform, OS version, IAM role
• SSM command output: the error message from the failed operation
• Priority classification: derived from instance tags (for example, env=prod maps to CRITICAL)

Validate the solution

By default, the solution investigates instances tagged with auto_investigate =true. You can change this to use any existing tag by setting the InvestigateTagKey and InvestigateTagValue parameters in Step 2.

To validate the solution:

1. Open the AWS Systems Manager console in a workload account.
2. In the left navigation pane, choose Run Command.
3. Choose Run a command.
4. Search for and select AWS-RunPatchBaseline document.
5. Under Target selection, choose Specify instance tags and enter your configured tag key and value.
6. Under Command parameters, set Operation to Install.
7. Choose Run.

If any tagged instance fails, a new investigation appears in the AWS DevOps Agent console within 60 seconds. In this example, a patch scan targeted three instances, two failed with different underlying causes, while one succeeded.

Note: If all targeted instances patch successfully, no investigation is triggered. The solution activates only when Systems Manager emits a failure event.

Figure 2 – Incident Response Dashboard with linked patch failure investigations

As shown in Figure 2, the parent investigation (batch-worker-01) completed in six minutes and contains a linked child investigation for data-proc-02, both triggered automatically via Event Channel. AWS DevOps Agent links related investigations when multiple instances fail in the same time window, providing a consolidated view of fleet-wide failures.

Figure 3 – Investigation timeline — initial triage and related failure linking

In Figure 3, the agent detects a second failure arriving within seconds, links the investigations, and escalates the fleet impact from 33% to 67%. It loads relevant skills and recalls patterns from prior investigations to accelerate root cause analysis.

Figure 4 – Fleet correlation identifying two independent failure signatures

As shown in Figure 4, rather than assuming all failures share the same root cause, AWS DevOps Agent examines each failure independently. It identifies that batch-worker-01 returned empty output in 1ms (disk exhaustion signature) while data-proc-02 ran for 18 seconds before failing with a DNS resolution error. A third managed node with the same OS and configuration patched successfully, confirming the failures are node-specific.

Figure 5 – Structured findings with evidence chains for each root cause

As shown in Figure 5, AWS DevOps Agent produces distinct findings for each causal chain: root volume exhaustion blocking all SSM commands, an application service writing unbounded logs without rotation (LOG-4421), and apt sources reconfigured to reference an unresolvable internal mirror (INFRA-2847). The investigation concludes with a consolidated root cause determination.

Figure 6 – Root cause determination — two independent misconfigurations identified

As shown in Figure 6, in this example, the investigation completed in under six minutes, concluding that the failures are two distinct misconfigurations rather than a shared patch-management fault.

Production considerations

Keep the following in mind when deploying to production:

• Cost: The solution uses serverless, pay-per-use components (Lambda, DynamoDB on-demand, EventBridge). Steady-state infrastructure costs are driven primarily by one AWS KMS key and one Secrets Manager secret per deployment region. For typical patch volumes, compute and storage costs are negligible. The per-command throttle (default: 5 investigations per command) prevents runaway costs during mass failures. For pricing details, refer pricing pages of AWS KMS, Secrets Manager, AWS DevOps Agent.
• Telemetry: Investigation quality improves when instances ship OS logs to Amazon CloudWatch Logs and patch operations are configured with Amazon S3 output logging (OutputS3BucketName). The agent reads full, untruncated command output from S3 during investigation. Sample CloudWatch Agent configurations are provided in the GitHub repository.
• Private VPC environments: For instances without direct internet access, enable Amazon Virtual Private Cloud (Amazon VPC) Flow Logs and CloudTrail S3 data events. These are prerequisites for the agent to diagnose network connectivity failures such as missing or misconfigured S3 VPC endpoints. Ensure the Agent Space role has permission to read VPC configuration (ec2:DescribeVpcEndpoints, ec2:DescribeRouteTables).
• Patch deployment compatibility: The solution captures failures from any method that executes AWS-RunPatchBaseline: Quick Setup Patch Policy, State Manager associations, Maintenance Windows, and manual Run Command.
• Continuous learning: Each investigation this pipeline triggers contributes to the Agent Space knowledge base. AWS DevOps Agent automatically analyzes completed investigations and extracts recurring root cause patterns into per-monitor memories. Over time, the agent recalls known failure patterns and prioritizes validation over exploratory diagnosis – reducing mean time to root cause for recurring failure modes. You can accelerate this by seeding the directives memory store with environment-specific context such as patching schedules and infrastructure conventions. For more information, see DevOps Agent Memories in the AWS DevOps Agent User Guide.
• Complementing with custom agents: This solution provides real-time, event-driven RCA within seconds of failure. For periodic operational reporting, such as daily patch compliance summaries or weekly failure trend analysis, consider creating a custom agent with a scheduled trigger. Custom agents can produce visual artifacts including charts and tables, making them well-suited for recurring health reports that complement the instant response this pipeline provides. For more information, see Custom Agents in the AWS DevOps Agent User Guide.
• Template hardening: The CloudFormation templates in this walkthrough are simplified for clarity. For production deployments, consider adding AWS KMS customer managed key encryption and point-in-time recovery for DynamoDB, a dead-letter queue for Lambda, and converting inline IAM policies to managed policies. The Lambda function runs outside a VPC by default for simplicity. If your security requirements mandate VPC-attached Lambda functions, deploy the function in a private subnet with a NAT gateway or configure VPC endpoints for the required AWS services (Secrets Manager, DynamoDB, STS, SSM, EC2) and ensure outbound HTTPS access to the DevOps Agent webhook endpoint.
• This solution responds to patch failure events emitted by Systems Manager. Instances in Connection Lost state (SSM agent unreachable) receive an Undeliverable status, which does not emit an EventBridge event and therefore does not trigger an investigation. For proactive detection of unmanaged instances, deploy the optional scheduled-health-check.yaml template from the companion repository, which periodically queries DescribeInstanceInformation for instances with PingStatus ConnectionLost.

Cleaning up

To avoid incurring future charges, delete the resources.

To delete the resources:

1. In the account where you deployed Stackset, choose StackSets in AWS CloudFormation console, select the Stackset you created, and choose Actions > Delete stacks from StackSet for all accounts and regions. Then delete the StackSet.
2. In the primary account, choose Stacks in AWS CloudFormation console, select the primary Stack you created, and choose Delete Stack.
3. In the primary account, select the Agent Space we created in AWS DevOps Agent console, and choose Actions > Delete Agent Space.

Note: When the primary stack is deleted, the AWS Secrets Manager secret enters a 7–30 day recovery window before permanent deletion. To delete immediately, use the AWS CLI command as per Delete an AWS Secrets Manager secret documentation.

Conclusion

In this post, you deployed an event-driven solution that automatically investigates AWS Systems Manager patch failures using AWS DevOps Agent. You configured cross-account event forwarding with Amazon EventBridge, implemented a Lambda function with deduplication and fleet-context enrichment, and validated that the agent autonomously links related failures, distinguishes independent root causes, and delivers structured findings – in this walkthrough, within minutes of a patch failure.

In this walkthrough, the solution reduced investigation time from hours to minutes. Each patch failure triggers an independent investigation, so investigation time per node remains consistent regardless of fleet size. The solution correlates failures across accounts and Regions, and scales from a single account to an entire AWS Organization via CloudFormation StackSets.

To get started, download the AWS CloudFormation templates from the companion GitHub repository and tag a few critical instances.

For more information, see the following resources:

• AWS DevOps Agent documentation
• AWS Systems Manager Patch Manager
• Amazon EventBridge User Guide