AWS Cloud Operations Blog

Centralized multi-account and multi-Region patching with AWS Systems Manager Automation

Update 01/2023: AWS Systems Manager announces Patch Policies, enabling cross account and cross Region patching. Patch Policies provide a user experience in a single console to easily define and enforce patch compliance across accounts and Regions with a few clicks. For more information, see Centrally deploy patching operations across your AWS Organization using Systems Manager Quick Setup.

AWS Systems Manager Automation now supports multi-account and multi-Region actions enabling you to centrally manage your AWS resources. This feature reduces the time and overhead needed for enterprise-wide configuration, operational actions, and compliance remediation.

In this blog post, I show you how to use AWS Systems Manager Automation to patch your managed instances across multiple AWS accounts and Regions. I’ll demonstrate how to leverage Resource Groups to organize instances for patching. For example, you can create Resource Groups for different environments such as development, test, and production. I’ll show you how to create a custom Automation Document which leverages Patch Manager and show you how to execute the custom Automation Document to install patches on your managed instances.

This post will walk through the four steps to set up your management account and target accounts for multi-account Automation patching.

Step 1: Create resource groups to logically group your managed instances

Step 2: Set up the required IAM roles

Step 3: Create an Automation Document to execute Patch Baseline Operations

Step 4: Execute Automation to patch multi-account resources

Step 1: Creating resource groups

A resource group is an optional means of organizing your AWS resources. Resource groups make it easier to manage and automate tasks on large numbers of resources at one time. Resource groups can be created based on a server function, for example, web servers and databases. Resource groups can help you avoid deploying patches to the wrong set of instances.

After you have opened the AWS Systems Manager console, select Find Resources from the left navigation pane. In the example that follows, the group of instances that we want to patch have been tagged with the tag:key Name:WebServers. A fleet of instances that have these tags can be patched using this approach.

On the Find Resources page, specify the resource type as AWS::EC2::Instance, the tag key as Name, and the tag value as WebServers and select View query results. After you have verified the returned queried results are appropriate, select Save query as group and provide a Group name and optional Group description.

CLI command:

aws resource-groups create-group --name WebServers --resource-query '{"Type":"TAG_FILTERS_1_0","Query":"{\"ResourceTypeFilters\":[\"AWS::EC2::Instance\"],\"TagFilters\":[{\"Key\":\"Name\",\"Values\":[\"WebServers\"]}]}"}'

Step 2: Set up the required IAM roles

First you need to pick any one account as your management account and Region as your management Region. From this management location (management account and Region), you can execute Automation tasks targeting other AWS accounts and Regions.

Management account

In the management account, provision the administrator automation IAM role, which can be created using the following AWS CloudFormation template, AWS-SystemsManager-AutomationAdministrationRole.zip. Or, if your accounts are managed by AWS Organizations, AWS-SystemsManager-AutomationAdministrationRole-org.zip.

For the purposes of this blog post, we will call the IAM role AWS-SystemsManager-AutomationAdministrationRole. Here is an example of the IAM policy that will be created for the AWS Organizations managed administration role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::*:role/AWS-SystemsManager-AutomationExecutionRole",
            "Effect": "Allow"
        },
        {
            "Action": [
                "organizations:ListAccountsForParent"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        }
    ]
}

Next, I’ll create an IAM user in my management account with the policy AmazonSSMAutomationRole attached. In addition to the AmazonSSMAutomationRole policy, I’ll add an inline policy to allow iam:PassRole and resource-groups:ListGroupResources actions in order to target Resource Groups. Here is an example inline policy that will be created:

Important: Make sure to replace <ManagementAccountId> in the policy below with the account ID of the management account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "resource-groups:ListGroupResources"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::<ManagementAccountId>:role/AWS-SystemsManager-AutomationAdministrationRole"
        }
    ]
}

Target account

In the account that I want to target, I will provision the target IAM role that can be created using the following CloudFormation template available, AWS-SystemsManager-AutomationExecutionRole.zip. Or, if your accounts are managed by AWS Organizations, AWS-SystemsManager-AutomationExecutionRole-org.zip.

For the purposes of this blog post, we will call the target automation IAM role AWS-SystemsManager-AutomationExecutionRole. Here is an example of the IAM policy that will be created for the AWS Organizations managed execution role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "resource-groups:ListGroupResources",
                "tag:GetResources",
                "ec2:DescribeInstances"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::<TargetAccountId>:role/AWS-SystemsManager-AutomationExecutionRole",
            "Effect": "Allow"
        }
    ]
}

Make sure to update the role ARN with the account ID of the target account. You will also need to make sure that the role has the management account ID and ssm.amazonaws.com is configured as a trusted entity for the IAM role as shown here:

{
    "Version": "2012-10-17",
    "Statement": [{
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ManagementAccountId>:role/AWS-SystemsManager-AutomationAdministrationRole"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-abcdefg012"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ssm.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Step 3: Create an Automation Document to execute patch baseline operations

In the management account, I will create a custom Automation Document, which will execute the Command Document AWS-RunPatchBaseline. From the AWS Systems Manager console of the management account, select Documents from the left navigation pane.

  1. Choose Create document.
  2. Type a descriptive name for the document. For this example I will use Automation-RunPatchBaseline.
  3. In the Document type list, choose the type Automation document.
  4. Delete the default content in the Content field, and then paste the following JSON content.
  5. Choose Create document to save the document.
{
    "outputs": [
        "runPatchBaseline.Output"
    ],
    "description": "Automation document to execute the Command document AWS-RunPatchBaseline",
    "schemaVersion": "0.3",
    "assumeRole": "{{AutomationAssumeRole}}",
    "parameters": {
        "AutomationAssumeRole": {
            "type": "String",
            "description": "(Optional) The ARN of the role that allows Automation to perform the actions on your behalf.",
            "default": ""
        },
        "Operation": {
            "allowedValues": [
                "Scan",
                "Install"
            ],
            "description": "(Required) The update or configuration to perform on the instance. The system checks if patches specified in the patch baseline are installed on the instance. The install operation installs patches missing from the baseline.",
            "type": "String"
        },
        "SnapshotId": {
            "default": "",
            "description": "(Optional) The snapshot ID to use to retrieve a patch baseline snapshot.",
            "type": "String",
            "allowedPattern": "(^$)|^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$"
        },
        "InstanceId": {
            "description": "(Required) EC2 InstanceId to which we apply the patch-baseline",
            "type": "String"
        },
        "InstallOverrideList": {
            "default": "",
            "description": "(Optional) An https URL or an Amazon S3 path-style URL to the list of patches to be installed. This patch installation list overrides the patches specified by the default patch baseline.",
            "type": "String",
            "allowedPattern": "(^$)|^https://.+$|^s3://([^/]+)/(.*?([^/]+))$"
        }
    },
    "mainSteps": [
        {
            "maxAttempts": 3,
            "inputs": {
                "Parameters": {
                    "SnapshotId": "{{SnapshotId}}",
                    "InstallOverrideList": "{{InstallOverrideList}}",
                    "Operation": "{{Operation}}"
                },
                "InstanceIds": [
                    "{{InstanceId}}"
                ],
                "DocumentName": "AWS-RunPatchBaseline"
            },
            "name": "runPatchBaseline",
            "action": "aws:runCommand",
            "timeoutSeconds": 7200,
            "onFailure": "Abort"
        }
    ]
}

Step 4: Execute Automation to patch resources in target accounts

In the management account AWS Systems Manager console, select Automation from the left navigation pane and then choose Execute automation.

  1. In the Choose document section, select the custom Automation Document created in Step 3, Automation-RunPatchBaseline and then click Next. 
  2. Select Multi-account and Region tab.
  3. In the Target accounts and Regions section:
    1. Provide the account IDs of the target accounts and specify the Regions where the EC2 instances are launched.
    2. (Optional) Specify the number or percentage of locations (account-Region pairs) on which to execute the task at the same time.
    3.  (Optional) Specify an error threshold which will stop the task after the task fails on a specific number or percentage of locations.
  4. In the Targets section:
    1. Select Resource Group as the Targets.
    2. Select InstanceId as the Parameter.
    3. Select WebServers as the Resource group.
  5.  In the Input parameters section:
    1. For AutomationAssumeRole, provide the ARN for the AWS-SystemsManager-AutomationAdministrationRole IAM role you previously created in the section Step 2: Set up the required IAM roles.
    2. Specify Install as the Operation.
    3. (Optional) Specify an InstallOverrideList as a list of patches to be installed. This patch installation list overrides the patches specified by the default patch baseline.
    4. (Optional) Specify the snapshot ID to use to retrieve a patch baseline snapshot.
  6. In the Rate control section:
    1. Specify the number or percentage of target instances on which to execute the task at the same time.
    2. Specify an error threshold which will stop the task after the task fails on a specific number or percentage of target instances.

CLI command:

aws ssm start-automation-execution
--document-name Automation-RunPatchBaseline
--parameters AutomationAssumeRole=arn:aws:iam::<ManagementAccountId>:role/AWS-SystemsManager-AutomationAdministrationRole,Operation=Install
--target-parameter-name InstanceId
--targets Key=ResourceGroup,Values=WebServers
--target-locations Accounts=<TargetAccountId>,Regions=<TargetAccountRegion>,ExecutionRoleName=AWS-SystemsManager-AutomationExecutionRole

Conclusion

In this blog post, I have shown you how to patch AWS Systems Manager Managed Instances across multiple AWS accounts using Automation. I started by showing you how to create the required IAM service roles used by Automation in the management and target accounts. After you created your IAM roles, I showed you how to create a custom Automation Document for executing patch baseline operations. Following this, I showed you how to execute Automation Documents that targeted your managed instances via resource groups in target accounts.

You can customize your workflow further by creating your own Automation Document based on the document AWS-PatchInstanceWithRollback. For more details, see Working with Automation Documents. After you have created your Document, you can write additional steps and add them to the workflow.

Example steps include:

  • Updating an ALB Target Group to remove the target during patching operations and add the target back to the group following successful patching (aws:executeAwsApi action type)
  • Validating your OS-level application states using Run Command with the AWS-RunPowerShellScript or AWS-RunShellScript Documents (aws:runCommand action type)
  • Creating an AMI of your patched managed EC2 instance (aws:createImage action type)

About the Author

Erik Weber is a Cloud Support Engineer in AWS Premium Support. He specializes in AWS Systems Manager, Amazon EC2 Windows, and Microsoft PowerShell. Outside of work, Erik has a passion for hiking, cooking, and fanatically dancing at concerts he attends.