Automate AWS Config data visualization with AWS Systems Manager

Earlier this year we published a blog, Visualizing AWS Config data using Amazon Athena and Amazon QuickSight. It outlines the steps for setting up AWS Config with Amazon Athena and Amazon QuickSight. We received great feedback from that post. To further help our customers adopt these tools we are happy to announce the availability of a sample AWS Systems Manager Automation runbook that simplifies the process of setting up different components required for the visualization solution.

Solution overview

We want to provide customers with an automated way of setting up every requirement to properly visualize AWS Config data. The sample solution described in this post simplifies this process by using Systems Manager Automation to configure the steps involved within minutes, as opposed to manually running through each step. This demonstrates the art of what’s possible with SSM Automation. Furthermore, it provides an excellent example of simplifying complex tasks to manage infrastructure, thereby enabling customers to create visualization dashboards from the data recorded by AWS Config across their organization.

Prerequisites

This sample solution requires the following:

1. Enable Amazon Simple Storage Service (Amazon S3) configuration snapshot delivery in AWS Config.
2. Retrieve the Amazon S3 bucket name used with AWS Config.
3. Sign up for an Amazon QuickSight subscription in the same AWS account where you run the automation runbook.
4. Authorize QuickSight to connect to Athena and the Amazon S3 Bucket used by AWS Config.
5. Obtain your QuickSight username.

Deploy the AWS CloudFormation template

The AWS CloudFormation template deploys an SSM automation runbook called Config-QuickSight-Visualization. It can be utilized to setup AWS Config for use with Athena and configure QuickSight to create visualize dashboards.

1. Download and save the CloudFormation template Config-QuickSight-Visualization-SSM-Automation.yaml.
2. Open the AWS CloudFormation console.
3. Choose Create stack.
4. For Specify template, choose Upload a template file, choose the file you saved locally, Config-QuickSight-Visualization-SSM-Automation.yaml, and choose Next.
5. For Stack name, enter Config-QuickSight-Visualization-SSM-Automation, and choose Next.
6. On the Configure stack options page, leave the defaults and choose Next.
7. On the Review page, check the box I acknowledge that AWS CloudFormation might create IAM resources with custom names, and choose Create stack.

Gathering details to run the Config-QuickSight-Visualization automation runbook

1. Running the following AWS Command Line Interface (AWS CLI) command outputs the Amazon S3 bucket name and the name of your AWS Config delivery channel currently in use:

   aws configservice describe-delivery-channels

2. Note the Amazon S3 bucket name and the delivery channel name, this will be needed when running the SSM automation runbook.
3. In QuickSight, open your profile menu in the top right and click Username.

Figure 1: Amazon QuickSight Profile Menu

4. Select and note your QuickSight Username. This will be needed when running the SSM automation runbook.

Figure 2: Amazon QuickSight Username

Running the Config-QuickSight-Visualization automation runbook

1. Open the AWS Systems Manager Documents console.
2. Click the Owned by me
3. Click on the Config-QuickSight-Visualization.
4. Click Execute automation.
5. Enter in the parameters below required by the automation runbook.

Figure 3: AWS SSM Automation Runbook Input Parameters Dialog Box

• ConfigDeliveryChannelName: (Required) Name of your AWS Config delivery channel. The default is set to the default value. The preceding section shows how to acquire this information.
• ConfigS3BucketLocation: (Required) AWS Config S3 Bucket Name, which is the name of your S3 Bucket currently utilized for AWS Config (i.e., config-bucket-1234567891). The preceding section shows how to acquire this information.
• QuickSightUserName: (Required) The Amazon QuickSight Username. The previous section shows how to acquire this information.
• AutomationAssumeRole: (Optional) The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role that allows SSM automation to conduct the actions on your behalf. This role needs access in order to conduct every action required to create the visualization.

6. Click Execute.

Creating visuals in Amazon QuickSight

The Config-QuickSight-Visualization automation runbook creates the following views and datasets within Athena and QuickSight. You can then utilize these to create your visualization dashboard.

• v_config_rules_compliance
• v_config_resource_compliance
• v_config_rds_dbinstances
• v_config_iam_resources
• v_config_ec2_vpcs
• v_config_ec2_instances
• v_config_resources

Creating your analyses in Amazon QuickSight

1. From the QuickSight console, choose New analysis.

Figure 4: Amazon QuickSight New analysis

2. On the Datasets page, choose the v_config_resource_compliance dataset, and then choose Create Analysis.

Figure 5: Amazon QuickSight Create analysis

Create a visual by using AutoGraph

1. Create a visual by using AutoGraph, which is selected by default.
2. On the analysis page, choose resourcetype and compliancetype in the Fields list pane.
3. QuickSight creates a Horizontal bar chart using this data.

Figure 6: Horizontal bar chart

Adding Additional Datasets to your Analyses

1. Add more data sets to the analysis to create more visuals.
2. From within the analysis, click the Add,edit,replace and remove datasets icon.

Figure 7: Add, edit, replace, and remove datasets

3. Click Add Datasets.
4. Select the v_config_rules_compliance and click Select
5. On the analysis page, choose configrulename and compliancetype in the Fields list
6. In the Visual types pane, choose the Donut Chart
7. Drag the field compliancetype to the Value field under the Field wells

Figure 8: Field wells section

8. You can create a filter on any field in the currently selected visual. When you create a filter, by default it applies only to the currently selected visual.
9. Click on the Filter icon within the QuickSight side bar.
10. Under the Filters section, click “Create one…“, and then select the compliancetype
11. Click on the compliancetype field and uncheck the Select all check box.
12. Select NON_COMPLIANT and click Apply.
13. Click on the Visualize button on the side bar to return to make changes to your visual.

Figure 9: Donut Chart

Create a dashboard

1. In the analysis, choose Share in the application bar in the upper-right, and then choose Publish dashboard.
2. In the Publish dashboard page that opens, choose Publish new dashboard as, and enter the name Config Dashboard.
3. Click Publish dashboard.
4. On the Share dashboard page that opens, choose the X icon to close it.

Figure 10: Amazon QuickSight Dashboard

Cleanup

If you like to remove the resources created by the Config-QuickSight-Visualization automation runbook, you can do the following steps:

1. Open the AWS Systems Manager Documents console.
2. Click the Owned by me
3. Click on the Config-QuickSight-Visualization.
4. Click Execute automation.
5. Set the DeleteConfigVisualization to true.
6. Set the other parameters as you did previously.
7. Click Execute.
8. Open the AWS CloudFormation console
9. Choose the stack Config-QuickSight-Visualization-SSM-Automation and click Delete.
10. Click Delete Stack.

Conclusion

This post demonstrates how to use AWS Systems Manager automation runbook in order to simplify the process of integrating AWS Config data with Amazon Athena and Amazon QuickSight. It reduces the configuration time to few minutes instead of manual steps. It provides an excellent example of how to simplify complex tasks in order to manage infrastructure, and it will enable customers to create visualization dashboards from the data recorded by AWS Config across their organization. To learn more on AWS Systems Manager automation runbook see Working with runbooks for AWS Systems Manager.

About the author