Automate vulnerability management and remediation in AWS using Amazon Inspector and AWS Systems Manager – Part 1

Update 11/2022 – In September 2022, Amazon Inspector added support for Windows operating systems for continual software vulnerability scanning. This post supports remediating vulnerabilities only on Linux operating systems supported by the Systems Manager agent.

AWS recently launched the new Amazon Inspector for performing continuous vulnerability scans on Amazon Elastic Compute Cloud (Amazon EC2) instances and container images stored in Amazon Elastic Container Registry (Amazon ECR). These scans assess software vulnerabilities and unintended network exposure. The new Amazon Inspector uses the Systems Manager (SSM) agent to collect software application inventory of the Amazon EC2 instances. Then, Inspector scans this data and identifies software vulnerabilities, a crucial step in vulnerability management.

You should perform regular patching operations for resolving vulnerabilities identified by Amazon Inspector based on the severity of the vulnerabilities. You can use AWS Systems Manager Patch Manager to automate the process of patching nodes managed by Systems Manager using the SSM agent. There may be zero-day or other high and critical severity vulnerabilities where patches are available. However, you may not want to wait for the regular patching schedule to remediate them. In these cases, on-demand mechanisms for patching should exist.

This series provides two methods for remediating Amazon Inspector software vulnerability findings on-demand using a Systems Manager Automation runbook. An Automation runbook defines the AWS Systems Manager’s actions on your managed instances and other AWS resources when an automation runs. It contains one or more steps that run in sequential order or branches based on preceding steps. You can prepare both methods in your AWS account to be ready for use when speed is critical to remediate software package vulnerabilities identified by Inspector.

In Part 1 of this series, you’ll learn how to remediate Inspector findings for a specific vulnerability affecting multiple EC2 instances. You will use AWS Security Hub custom actions to trigger a Systems Manager Automation runbook for on-demand vulnerability patching on the selected EC2 instances.

In Part 2, you’ll learn how to directly invoke the Systems Manager Automation runbook to remediate all Amazon Inspector findings for EC2 instances using resource tags and Amazon Inspector finding severity. You’ll use the Systems Manager Automation runbook direct invocation method if you want to attempt patching all software vulnerabilities identified by Inspector. This approach effectively manages the EC2 instance fleet using tag-based targeting across your AWS Organization.

These methods work in multiple regions and accounts managed by an AWS Organization. They can help you reduce the operational complexity of manual updates, reduce errors, and increase the remediation speed.

Solution overview

Remediate using Security Hub custom action

This method lets you remediate selected Amazon Inspector findings from Security Hub using custom actions. The custom action enables an Automation runbook for selected findings. Then, the Automation runbook will use the affected packages’ information from the Amazon Inspector findings details and use Systems Manager Patch manager to update only these packages to the latest version. Enable Security Hub to use this method.

For example, there are affected packages in the following figure, curl and libcurl, for CVE 2017-8816. When the user invokes the custom action for this finding, Systems Manager Patch Manager updates both packages on the target managed instance. Amazon Inspector will automatically close all of the findings associated with these affected packages shortly after the vulnerability is patched.

Figure 1: Finding details in the Amazon Inspector Console.

The Security Hub delegated administrator account can trigger the automation runbook and target Amazon EC2 instances in member accounts in a multi-account, multi-region scenario. This post provides two custom actions that update the affected packages and either perform a reboot or defer the instance reboot. You can choose to reboot the Amazon EC2 instance after installing updates to benefit immediately from the updates. Furthermore, you can delay the reboot to your maintenance window by selecting the no reboot option.

With this custom action, we recommend searching Amazon Inspector findings with a specific CVE. You can use the Title filter in Security Hub, filter match type as starts with, and enter the CVE as shown using arrows in the following figure. Use the Resource Type filter with AwsEc2Instance to only show findings associated with Amazon EC2 instances. Once these filters are selected, Security Hub will show you all Amazon EC2 instances affected by the CVE. Then, you can choose all the findings displayed with this filter associated with Amazon EC2 instances and run the custom action.

The following figure illustrates that you can run the custom action on multiple findings associated with different Amazon EC2 instances. However, you shouldn’t select numerous Amazon Inspector findings for the same Amazon EC2 instance. To resolve multiple Amazon Inspector findings for the same Amazon EC2 instances, see Part 2 of this series.

Figure 2: Selecting multiple Amazon Inspector findings for remediation in Security Hub.

Architecture overview

The automation workflow enables the resolveInspectorFindingsRunbook custom Automation runbook provided in this post. Security Hub Custom action initiates this runbook, as described in the solution overview section. Let’s review the process as shown in the following figure:

Figure 3: Automation process in multiple accounts

1. When using the Security Hub custom action, you select the Amazon Inspector finding in Security Hub and invoke the custom action. An event is sent to Amazon EventBridge with the Amazon Inspector finding details. The resolveInspectorFindings EventBridge rule matches this event pattern.
2. The resolveInspectorFindings EventBridge rule parses the Amazon EC2 instance ID and Amazon Inspector finding ID to the Automation runbook as parameters. The reboot option is selected based on the custom action being used.
3. The resolveInspectorFindingsRunbook creates a multi-account and multi-Region StartAutomationExecution API call to initiate the patch workflow in the target account and Region of the Amazon EC2 instance.
4. A Python script gathers the affected packages for the Amazon Inspector finding, as shown in Figure 1. Then, the script creates a patch install override list from the affected packages gathered. The delegated administrator account stores the list in an Amazon Simple Storage Service (Amazon S3) bucket. The AWS CloudFormation template provided with this post creates this bucket .
5. A Run Command task for the document AWS-RunPatchBaseline is run on the Amazon EC2 instance using the install override list created in Step 4. The reboot option is passed based on the Security Hub custom action selected. If an existing patching operation is running on the target Amazon EC2 instance, the Automation runbook doesn’t proceed. If an update is available, the AWS-RunPatchBaseline install operation updates the affected packages identified in the patch install override list to the latest version available.
6. Once the update completes, the Automation runbook queries the Systems Manager for the State Manager association applied to the Amazon EC2 instance to gather software inventory using the AWS-GatherSoftwareInventory document.
7. The association for AWS-GatherSoftwareInventory immediately initiates to apply on the Amazon EC2 instance to refresh inventory.
8. The Amazon EC2 instance sends refreshed inventory information to AWS Systems Manager.

Amazon Inspector sets the remediated finding status to CLOSED if new patches are installed successfully. Security Hub’s corresponding finding record state sets to ARCHIVED.

Prerequisites

The required prerequisites are as follows:

1. Systems Manager must manage the Amazon EC2 instances.
2. Enable Security Hub

Walkthrough

This walkthrough includes the following steps:

1. Create Security Hub custom actions for resolving Amazon Inspector Findings.
2. Deploy the CloudFormation template for the Automation runbook.
3. Create execution roles for multi-account and multi-Region using StackSets.
4. Update EC2 IAM instance roles with permissions to access the install override list S3 bucket.
5. Remediate vulnerabilities using the Security Hub custom action.

Step 1: Create Security Hub custom actions for resolving Amazon Inspector Findings

1. In the Security Hub delegated administrator account and Region, navigate to the Security Hub console. In the left-hand navigation pane, choose Settings, and select the Custom actions tab.
2. Choose Create custom action.
3. For remediating Amazon Inspector findings without rebooting the EC2 instance, perform the following steps:

1. 1. Set the Action name to Rem-Inspector-NoRBT.
   2. Set the Description of the custom action, such as Action that will remediate Amazon Inspector findings without restarting the instance.
   3. Set the Custom action ID to InspectorRemNoRBT.

Figure 4: Create Security Hub custom action for remediating findings without instance reboot

4. To create a second custom action for remediating Inspector findings and rebooting the EC2 instance, perform the following steps:

1. 1. Set the Action name to Rem-Inspector-RBT.
   2. Set the Description for the custom action, such as Action that will remediate Amazon Inspector findings and reboot the EC2 instance.
   3. Set the Custom action ID to InspectorRemRBT.

1. 5. Copy the custom action ARNs, this will be used in the next step.

Figure 5: Security Hub custom action ARNs.

Step 2: Deploy the CloudFormation template for automation runbook

The AWS CloudFormation template provided with this post deploys the resources necessary to enable Amazon Inspector to find resolution in your AWS account. You must deploy these resources in your Security Hub delegated administrator account. The CloudFormation should be deployed only in the aggregator region if using Security Hub cross-region aggregation.

Open the following GitHub page, and download the ResolveInspectorFindingsCFN.yaml file:

https://github.com/aws-samples/automate-vulnerability-management-and-remediation/blob/main/templates/resolveInspectorFindingsCFN.yaml

1. Navigate to the AWS CloudFormation console in the Security Hub delegated administrator account and Region. Create a stack using the ResolveInspectorFindingsCFN.yaml
   template file.
2. On the Specify stack details page, perform the following steps:
   1. Set the Stack name to a descriptive stack name.
   2. Set the RemediateInspectorFindingCustomActionNoRBTArn to the Security Hub custom action ARN copied in the previous section to remediate Amazon Inspector findings without rebooting the EC2 instance.
   3. Set the RemediateInspectorFindingCustomActionRBTArn to Security Hub custom action ARN copied in the previous section to remediate Amazon Inspector findings and reboot the EC2 instance.
   4. Set the OrganizationId to the Organization ID if deploying this template in the delegated administrator or management account. You can find your Organization ID by navigating to the AWS Organization console, as illustrated below. Leave this field empty if deploying this template in a standalone account.

Figure 6: Organization’s console.

      e. Choose Next

Figure 7: Stack details page.

3. On the Configure stack options page, optionally add required tags, and then choose Next.
4. Review the parameters. Select I acknowledge that AWS CloudFormation might create IAM resources with custom names. Next, choose Create Stack to submit your stack configuration.
5. After refreshing the page, the status of your stack should be CREATE_IN_PROGRESS. When the status changes to CREATE_COMPLETE, proceed to the next section. You can view the resources created in the resources tab of the CloudFormation console.
6. Copy the AutomationRunbookName and InstallOverrideListS3BucketName from the outputs tab of the CloudFormation console, as illustrated below. You will use these in the following steps.

Figure 8: CloudFormation outputs tab.

Step 3: Create execution roles for multi-account and multi-Region automation using StackSet

You must create the Automation execution cross-account role for triggering this automation (deployed in previous step) from a delegated administrator account targeting Amazon EC2 instances in member accounts. Use the following steps to deploy the Automation execution role in all member accounts with EC2 instances using CloudFormation StackSets.

If you want to use this automation in a single account only, you must create this role in the same AWS account used in Step 1.

Open the following GitHub page and download the automationExecutionRole.yml file.

https://github.com/aws-samples/automate-vulnerability-management-and-remediation/blob/main/templates/automationExecutionRole.yaml

1. Navigate to the AWS CloudFormation console in the Organization management account or CloudFormation delegated administrator.
2. From the navigation pane, choose StackSets.
3. At the top of the StackSets page, choose Create StackSet.
4. Under Prerequisite – Prepare template. Choose template is ready.
5. Under Specify template, select Upload a template file, choose file, choose the automationExecutionRole.yml, and select Next.
6. On the Specify stack details page, perform the following steps:

1. 1. Set the Stack name and the Description.
   2. Set the AutomationRunPatchBaselineRunbook to the runbook that you copied in point 6 of step 2.
   3. Set the DelegatedAdministratorAccountId to the Security Hub delegated administrator account ID. If deploying in a standalone account, specify the targeted account ID.
   4. Set the InstallOverrideListBucket to the S3 bucket name that you copied in point 6 of step 2.

Figure 9: CloudFormation StackSet details.

7. On the Configure stack options page, optionally add required tags, and then choose Next.
8. On the Set deployment options page, select the desired Region. Since you’re creating IAM resources, you only need to specify one region and then choose Next.
9. Review all of the information. Select I acknowledge that AWS CloudFormation might create IAM resources with custom names. Next, choose Submit to submit your stack configuration.
10. After refreshing the page, the status of your StackSet should be Running. When the status changes to Succeeded, proceed to the next section. You can view the outcome of the individual Stack instances under the Stack instances tab of the CloudFormation StackSet console.

Step 4: Update EC2 IAM instance roles with permissions to access the install override list S3 bucket

The Amazon EC2 instances in the target accounts must have IAM permission to the install override lists located in the S3 bucket in the Security Hub delegated administrator account. The CloudFormation template from Step 1 creates this bucket. The following section contains a sample IAM policy required for the EC2 Instance Profile definition. This policy is needed in addition to IAM policies for Systems Manager to manage the resource.

You must update the bucket’s name in the resource section of the sample policy with the install override list bucket’s name that you copied in step 2, point 6 previously. You must add this permissions policy to all of the EC2 instance roles for instances where you want the ability to remediate Amazon Inspector findings using automation.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::NameOfInstallOverrideListBucketHere/*"
    }]
}

Step 5: Remediate vulnerabilities using the Security Hub custom action

Let’s walk through using the Security Hub custom action to remediate an Amazon Inspector finding associated with an Amazon EC2 instance:

1. In the Security Hub Delegated administrator account, navigate to the Security Hub console in the Region where you deployed Step 1, and go to the findings page.

1. 1. Use the Title filter in Security Hub, filter match type as starts with, and enter the CVE you want to remediate.
   2. Use the Resource Type filter with AwsEc2Instance to only show findings associated with Amazon EC2 instances.
   3. Once these filters are selected, Security Hub will show you all of the Amazon EC2 instances affected by the CVE.
   4. Select findings that you want to remediate and run the custom action.
   5. Select Rem-Inspector-RBT if you want to reboot the Amazon EC2 instance after installing the patch. If you don’t want to reboot the Amazon EC2 instance, choose Rem-Inspector-NoRBT.

Figure 10: Security Hub findings page and custom action.

2. Once you select the custom action, you will get a notification banner confirming EventBridge receives the events.
3. You can monitor the progress of the Automation runbook for each of the findings selected in the Systems Manager Automation console under the Executions tab, as shown in the following figure.

Figure 11: Systems Manager Automation execution status.

4. After patching the packages, the Amazon Inspector findings update to the Closed status. You can view the closed Amazon Inspector findings on the All findings page on the Amazon Inspector console. Select Closed from the drop-down, as shown in the following figure.

Figure 12: Amazon Inspector console for closed finding.

The vim package-related and dependency-related findings for the Amazon EC2 instance will be remediated and closed.

5. The corresponding Security Hub finding record state is set to ARCHIVED, as shown in the following figure.

Figure 13: Security Hub finding record state set to Archived.

Notes

1. This method will only remediate Amazon Inspector Package Vulnerability findings associated with Amazon EC2 instances. It doesn’t remediate Amazon Inspector Network reachability findings or Amazon Inspector Package Vulnerability findings related to container images in Amazon ECR.
2. You can use the custom action to remediate one Amazon Inspector finding per Amazon EC2 instance at a time. However, suppose the affected package resulted in multiple findings with different CVEs for the same EC2 instance. In that case, all of the findings are remediated following a successful patching operation for the affected package. If you want to remediate inspector findings at scale, refer to part 2 of this blog series.
3. You can simultaneously trigger the custom action on multiple Amazon Inspector findings associated with different Amazon EC2 instances. We recommend filtering Amazon Inspector findings with a specific CVE in Security Hub and then running the custom action on every Amazon EC2 instance with the vulnerability, as shown in Figure 2.
4. The Automation runbook installs the latest version of the affected packages and is available in the package manager repositories.

Conclusion

This post discussed how the new Amazon Inspector can help you detect vulnerabilities in Amazon EC2 instances. We reviewed why on-demand patching methods are needed to remediate zero-day and critical vulnerabilities. Finally, we walked through an automation method using the Security Hub custom action to remediate Inspector findings for package vulnerabilities on selected Amazon EC2 instances affected by a specific CVE. This automation can help you reduce the operational complexity of manual updates, reduce errors, and increase the remediation speed.

You can use the Security Hub custom actions Rem-Inspector-RBT and Rem-Inspector-NoRBT to remediate the same vulnerability across multiple Amazon EC2 instances across an AWS Organization.

In Part 2, we’ll show you how to use the Systems Manager Automation runbook directly from the Automation console to remediate the Amazon Inspector findings for Amazon EC2 instances at scale across an AWS organization based on resource tag and Amazon Inspector finding severity.

Authors: