Implement AWS Config rule remediation with Systems Manager Change Manager

AWS Config enables you to assess, audit, and evaluate the configurations of your AWS resources. AWS Config rules are evaluated when changes are made in the AWS environment. By analyzing changes as they occur, you can catch compliance violations quickly and minimize the exposure to your organization. When you apply a remediation action to an AWS Config rule violation, automation is initiated when the rule violation is detected.

AWS Systems Manager Change Manager provides a tool that allows you to request, approve, implement, and report on operational changes to your application configuration and infrastructure. When AWS Config detects a change, it can trigger a change request to remediate the issue. The change request can be configured to make the change as soon as possible or within the next available maintenance window based on the change calendar. You can also configure change approvals to assure that the change is appropriate. In this blog post, I will share an example where an Amazon Elastic Compute Cloud (Amazon EC2) instance with a public IP address generates a change request, which once approved, will turn off the instance.

Figure 1: Automated compliance remediation with approval in Change Manager

AWS Config rules

Let’s say that your security organization does not allow open public-facing SSH ports in your network. Let’s assume that this is a hard rule—no exceptions. Your security team would like you to detect any open SSH ports and close them as quickly as possible.

To detect open SSH ports, create and configure an AWS Config rule.

1. From the left navigation pane of the AWS Config console, choose Rules, and then choose Add rule.
2. On Specify rule type, choose Add AWS managed rule.
3. Under AWS Managed Rules, search for and choose restricted-ssh, and then choose Next.

Figure 2:  Specify the rule type for restricted-ssh

4. On Configure rule, use the defaults, and then choose Next.
5. On Review and create, choose Next.

Now, to test the rule, create a security group with SSH open to the world.

1. In the left navigation pane of the EC2 console, choose Security Groups, and then choose Create Security Group.
2. On Create Security Group, enter the following information, and then choose Create Security Group.
   • For the security group name, enter PublicSSH.
   • Under Inbound rules, choose Add Rule. For Type, use SSH. For Source, use Anywhere.

After the security group has been created, make a note of the security group ID. You will need it later.

When you return to the AWS Config console after a few minutes, you can see the security group is out of compliance.

Figure 3: AWS Config rule with a noncompliant resource

For remediation, your security and compliance teams want to know the compliance violation has been remediated and they want tracking to show it happened.

AWS Config rule auto remediation

Let’s use an automation to remediate the issue.

First, create an IAM policy and role for the remediation. The following policy includes all the capabilities and permissions required to complete the steps in this blog post.

Create a policy

1. In the left navigation pane of the IAM console, choose Policies, and then choose Create policy.
2. Choose the JSON tab and then paste the following policy. Replace 111111111111 with your AWS account ID.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ChangeAutomation1",
            "Effect": "Allow",
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ssm:StartChangeRequestExecution",
                "ec2:StopInstances",
                "ec2:DescribeInstanceStatus"
            ],
            "Resource": "*"
        },
        {
            "Sid": " ChangeAutomation2",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::111111111111:role/ConfigAutoRemediation"
        }
    ]
} 

3. Choose Next: Tags.
4. Leave the defaults, and then choose Next: Review.
5. For the policy name, enter ConfigAutoRemediation, and then choose Create policy.

Create a role

1. In the left navigation pane, choose Roles, and then choose Create role.
2. For Select type of trusted entity, choose AWS service, and then choose Systems Manager.
3. Under Select your use case, choose Systems Manager.
4. Choose Next: Permissions.
5. Search for ConfigAutoRemediation, and select the checkbox next to the ConfigAutoRemediation
6. Choose Next:Tags.
7. Choose Next: Review.
8. For the role name, enter ConfigAutoRemediation, and then choose Create role.

After the role is created, choose the link to view the role details. Make a note of the role ARN. You will need it later.

Now run Systems Manager Automation to disable the security group.

1. In the left navigation pane of the Systems Manager console, choose Automation, and then choose Execute Automation.
2. Search for and choose the AWS-DisablePublicAccessForSecurityGroup document, and then choose Next.
3. For the type of execution, choose Simple execution.
4. Enter the security group ID. For AutomationAssumeRole, choose ConfigAutoRemediation, and then choose Execute.
5. Watch the execution. The steps for IPv6 and RDP might fail, but that’s OK because the DisableSSHFromIpV4 step is successful.
6. To check the result of the automation, go back to the EC2 console and validate that the security group no longer has the public SSH rule.

Now that you have a way to detect and repair a noncompliant security group, connect the two using an auto remediation in AWS Config.

Configure the auto remediation in AWS Config

Now configure the automation that disables the public SSH access to the AWS Config rule that identifies the noncompliant resource.

1. In the left navigation pane of the AWS Config console, choose Rules, and then choose restricted-ssh.
2. From Actions, choose Manage remediation.
3. On Edit: Remediation Action, choose Automatic Remediation, and then choose the AWS-DisablePublicAccessForSecurityGroup
4. In Parameters, choose GroupID for the resource ID parameter. This maps the security group that has changed to the GroupID parameter in the Automation document.
5. For the AutomationAssumeRole parameter, use the ARN of the ConfigAutoRemediation role. It will have the following format: arn:aws:iam::111111111111:role/ConfigAutoRemediation where 111111111111 is your AWS account ID.
6. You can leave IpAddressToBlock
7. Choose Save changes.

To test the automation to create an SSH ingress rule.

1. Go to the EC2 console, select Security Groups, and select the ‘Create security group’ button.
2. Enter the Security group name: OpenSSHRuleTest
3. Under Inbound rules, choose Add Rule.
4. For Type, choose SSH.
5. For Source, choose Anywhere.
6. Choose Create security group

Now go to the AWS Config console to track the compliance of the restricted-ssh rule. In a matter of minutes, the rule will initially be marked as compliant, then switch to noncompliant, and then return to compliant.

Return to the security group to validate that the SSH ingress rule has been removed.

Routing for approval with AWS Systems Manager Change Manager

Not every remediation should be run automatically. Imagine a scenario where the remediation needs to be reviewed and deployed within a change window. Using the earlier example, your security team has a rule that no EC2 instances should have a public IP address. The security team wants to review any instances that are in violation before they are turned off in case there is an impact to organization services. AWS Systems Manager Change Manager provides a framework to approve, track, and schedule releases.

Create a change template

If you haven’t configured Change Manager, you can use AWS Identity and Access Management (IAM) to configure your users.

In AWS IAM, create the following users:

• TemplateEditor
• TemplateApprover
• ChangeRequester
• ChangeApprover

Attach the following permissions policy to each user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
	"Sid": "SSMandEC2Interactions",
            "Effect": "Allow",
            "Action": [
                "iam:List*",
                "iam:Get*",
                "ssm:SendAutomationSignal",
                "ssm:Get*",
                "ssm:CreateDocument",
                "ssm:UpdateDocumentMetadata",
                "ssm:StartChangeRequestExecution",
                "ssm:Describe*",
                "ssm:List*",
                "ec2:StartInstances",
                "ec2:Describe*",
                "cloudwatch:Describe*",
                "organizations:List*",
                "organizations:Describe*"
            ],
            "Resource": "*"
        },
    {
        "Sid": "PolicyStatementToAllowUserToPassOneSpecificRole",
        "Effect": "Allow",
        "Action": [ "iam:PassRole" ],
      "Resource":"arn:aws:iam::111111111111:role/ConfigAutoRemediation"
    }

    ]
}

This policy grants the permissions required to execute the Change Manager functions. Change Manager requires read-only access to IAM users to route approvals, PassRole to pass a role to an AWS Systems Manager change request, and read-only access to Systems Manager. The policy grants read-only access to EC2 and StartInstance permissions to allow the user to view the created EC2 instances and restart them. Additionally, in AWS Accounts governed by AWS Organizations, the user requires read access to AWS Organizations to configure the AWS Systems Manager Change Manager. The user needs this read access to view the Change Manager data.

Although all four users have the same permissions, Systems Manager Change Manager provides controls outside of the IAM policy for users to approve templates and change requests.

Create an AWS Config rule

Now create an AWS Config rule that detects instances with a public IP address.

1. In the left navigation pane of the AWS Config console, choose Rules, and then choose Add rule.
2. On Specify rule type, choose Add AWS managed rule.
3. Under AWS Managed Rules, search for and choose ec2-instance-no-public-ip, and then choose Next.
4. On Configure rule, use the defaults, and then choose Next.
5. On Review and create, use the defaults, and then choose Add Rule.
6. Go to the EC2 console and create an instance with a public IP address. Be sure to set Auto-assign public IP to Enable. After you create the instance, make a note of the instance ID. You will need it later.

Wait a few minutes and then check the compliance status of the ec2-instance-no-public-ip rule. You’ll see the noncompliant resource.

AWS Systems Manager Change Manager uses a set of templates that define the actions required to execute the change request. A template creator will define the template, at which point it will be awaiting approval. A different user who has been configured to be the template approver must approve the template before others can use it.

Create a Change Manager template

Create a change template that will address noncompliant item.

1. Using the TemplateEditor user, sign in to the AWS Management Console, and then open the Systems Manager console.
2. In the left navigation pane, choose Change Manager. If you haven’t used Change Manager before, you will see a Set up Change Manager Choose it. For more information, see Setting up Change Manager in the AWS Systems Manager User Guide. During setup, configure the TemplateReviewer user as a template reviewer.
3. On the Change Manager page, choose Create template.
4. For Name, enter fix-ec2-instance-no-public-ip.
5. For Description, enter Fixes ec2 instances with public IP.
6. For Change template type, choose Standard change template.
7. For Runbook options, choose Select a single runbook.
8. For Runbook, enter AWS-StopEC2Instance.
9. For Template usage, use the default.
10. For Add Approver, choose Template-Specific, and then choose ChangeApprover from the list.
11. Leave the other fields at their defaults, and then choose Save and preview.
12. On Template details, choose Submit for approval.
13. Sign out of the AWS Management Console and then use the TemplateApprover user to sign in.
14. In the left navigation pane of the Systems Manager console, choose Change Manager, and then choose Templates.
15. Choose fix-ec2-instance-no-public-ip, and then choose Approve.

Create the change request

Now create the change request that will stop the noncompliant instance.

1. Using the ChangeRequester user, sign in to the AWS Management Console, and then open the Systems Manager console.
2. In the left navigation pane, choose Change Manager, and then choose Templates.
3. Choose fix-ec2-instance-no-public-ip, and then choose Create change.
4. For Name, enter Stop-instance-id.
5. For Change request information, use the default.
6. For Workflow start time, choose Run the operation as soon as possible after approval, and then choose Next.
7. For IAM role, choose ConfigAutoRemediation.
8. For Deployment location, choose Apply change to this account.
9. For Deployment targets, choose Single resource.
10. For Runbook parameters, For the ‘instance-id’ parameter, provide the instance-id for the noncompliant EC2 instance, choose Next, and then choose Submit for approval.

Approve the change request

1. Using the ChangeApprover user, sign in to the AWS Management Console, and then open the Systems Manager console.
2. In the left navigation pane, choose Change Manager, choose Requests, choose Stop instance-id, and then choose Approve.
3. Using the ChangeRequester user, sign in to the Systems Manager console to track the progress of the automation. You can follow the progress from the Task tab of the request details page.
4. Verify the AWS Config rule has remediated the noncompliant resources.

Automate the end-to-end process

Now create an automation that creates the change request in Change Manager.

1. In the left navigation pane of the AWS Systems Manager console, choose Documents, and then choose Create document.
2. Choose Automation.
3. For Name, enter CreateStopInstanceChange.
4. Choose the Editor tab, and then choose Edit.
5. Paste the following code into Document Editor text box, and then choose Create automation.

description: |-
  *This automation will create a AWS Change Manager request.*  
schemaVersion: '0.3'
assumeRole: "{{ AutomationAssumeRole }}"
parameters:
  InstanceId:
    type: String
    description: (Required) The ID of the EC2 instance on which you want to stop.
  AutomationAssumeRole:
    default: ''
    type: String
    description: (Optional) The Amazon Resource Name (ARN) of the role that allows SSM Automation to perform the actions on your behalf.
mainSteps:
  - name: StopInstancesChange
    action: 'aws:executeScript'
    inputs:
      Runtime: python3.6
      Handler: script_handler
      InputPayload:
        InstanceId: '{{InstanceId}}'
        AutomationAssumeRole: '{{AutomationAssumeRole}}'
      Script: |-
        import json

        def script_handler(event, context):
            import boto3
            ssm_client = boto3.client('ssm')
            instanceId = event["InstanceId"]
            roleArn = event['AutomationAssumeRole'] 
            
            response = ssm_client.start_change_request_execution(
                DocumentName='fix-ec2-instance-no-public-ip',
                Runbooks=[
                    {
                        'DocumentName': 'AWS-StopEC2Instance',
                        'Parameters': {
                            'InstanceId': [instanceId],
                            'AutomationAssumeRole': [roleArn]
                        }
                    }
                ]
            )
            
    description: Start a change that stops an instance

Configure the auto remediation

The next step is to connect the automation that creates the change request to the AWS Config rule auto remediation configuration. Create an automated remediation for AWS Config.

1. Using an account that has AWS Config administrator permissions, sign in to the AWS Management Console, and then open the AWS Config console.
2. In the left navigation pane, choose Rules, and then choose the ec2-instance-no-ip
3. From Actions, choose Manage remediation.
4. Choose Automatic Remediation, and then choose the CreateStopInstanceChange remediation action.
5. For Resource ID, enter the text ‘InstanceId’.
6. For AutomationAssumeRole, use the ConfigAutoRemediation role ARN, which you created in an earlier step.

The AWS Config rule has now been configured with auto remediation. This will create a change request that will be reviewed and approved by the ChangeApprover user, which was configured in the change template in AWS Systems Manager Change Manager. After the change request is approved, the EC2 instance will be stopped.

1. Sign in to the EC2 console and from the left navigation pane, choose Instances.
2. Choose the instance you created earlier, choose Instance State, and then choose Start Instance.
3. In the left navigation pane of the AWS Config console, choose Rules.
4. Refresh the page until the ec2-instance-no-ip rule is noncompliant.

Now that the AWS Config rule is noncompliant, AWS Config initiates a change request to shut down the instance will be created and wait for approval.

1. Using the ChangeApprover user, sign in to the AWS Management Console, and then open the Systems Manager console.
2. In the left navigation pane, choose Change Manager.
3. On the Requests tab, you’ll see a list of change requests. Choose the pending request, and then choose Approve to execute the remediation.

Figure 4: Pending change requests in AWS Systems Manager Change Manager

4. Wait for the change automation to run, and then check the compliance status of the rule. Now validate that the instance has been shut down.

Cleanup

To stop continuing AWS charges, terminate the EC2 instance and delete the restricted-ssh and ec2-instance-no-public-ip rules.

1. Go to the AWS Console and select EC2.
2. On the left hand nav bar, select Instances.
3. Select the EC2 instance created in the earlier step and click the Instance state dropdown button above the list of instances and select Terminate Instance.
4. On the left hand nav bar, select Security Groups.
5. Select the security group created earlier and click Actions and Delete security groups.
6. On the top nav bar, select Services and select Config.
7. Select Rules in the left-hand nav bar.
8. Select ec2-instance-no-public-ip and Actions and Delete rule.
9. Select restricted-ssh and Actions and Delete rule.
10. In AWS Console, select AWS Systems Manager, and select Change Manager. Under the Templates tab, select ‘fix-ec2-instance-no-public-ip.’
11. On the template detail page, select Actions and Delete template.
12. On the left-hand nav bar, select Documents.
13. Click the Owned by me tab, and select CreateStopInstanceChange.
14. On the document detail page, click the Delete button.
15. In the AWS Service List, select Identity and Access Management (IAM)
16. On the left-hand nav bar, select ‘Users’.
17. Delete the following users:
    1. TemplateEditor
    2. TemplateApprover
    3. ChangeRequester
    4. ChangeApprover

Conclusion

In this blog post, I demonstrated how the AWS Config auto remediation feature can be used to automatically disable security groups that open SSH ports to the internet. I also showed how you can use AWS Systems Manager Change Manager to route a change order to remediate a public-facing EC2 instance and keep a history of change requests. By combining these two capabilities, your organization can improve its response time and streamline the workflow for mitigating noncompliant configurations.

About the author