Track AWS Resource Configurations With AWS Config

One of the coolest aspects of the Cloud is its dynamic nature. Resources can be created, attached, configured, used, detached, and destroyed in a matter of minutes. Some of these changes are triggered by a direct human action; others have their origins in AWS CloudFormation templates or take place in response to Auto Scaling triggers. The resources themselves, as well as their connections, settings, and other attributes, change over time.

With all of this change happening, organizations of all sizes face some new challenges when it comes to asset tracking, inventory management, change management, and governance in the Cloud. They need to know what was changed, when it happened, and how the change might affect other AWS resources. This need might arise due to an unexpected configuration change, a suspected system failure, a compliance audit, or a possible security incident. Regardless of the cause, having access to the right data enables a deep, data-driven forensic analysis.

Traditional configuration management tools were built in an era where resources and the relationships between them changed infrequently. These tools were costly, complex, and required some care and feeding.

Introducing AWS Config
We aim to address these challenges with AWS Config. This new AWS service captures the initial state of your AWS resources (EC2 instances and related items to start, with others planned) and the relationships between them, and then tracks creations, deletions, and property changes for analysis, visualization, and archiving.

You can enable AWS Config with two clicks! Once enabled, it discovers resources and records their current configurations and any changes to them. This configuration data can be viewed in timeline fashion in the AWS Management Console. AWS Config also delivers these CIs to you. Configuration changes are streamed to an Amazon Simple Notification Service (Amazon SNS) topic of your choice and are also snapshotted to an Amazon Simple Storage Service (Amazon S3) S3 bucket (also of your choice) every 6 hours. You can also process this data using tools from our partners (see below) or on your own.

AWS Config understands and tracks the relationships between your AWS resources. It knows that an EBS volumes can be mounted to an EC2 instance, and that the instance can be associated with (among other things) Security Groups, Elastic IP Addresses, VPCs, and Elastic Network Interfaces

With AWS Config, you get full visibility in to the state of your AWS resources. You can watch them change over time, and you can view the full history of configuration changes for a resource. You can see the connections between resources and determine how a change to one resource could potentially affect other resources. AWS Config gives you the information that you need to have in order to work productively in an environment that is subject to constant change!

You can discover all of your AWS resources and determine which resources are outside of policy for your organization. For example, you might want to track down all resources that are not within a production VPC. You might want to see which instances a particular Elastic IP address has been associated with over the course of the last two weeks. Or, you might need to know the state of a resource as of a particular date.

Using AWS Config
AWS Config is enabled on a per-account, per-Region basis. It is accessible from the AWS Management Console, the AWS Command Line Interface (AWS CLI), and also provides a basic lookup API.

I start by enabling AWS Config for my account (within a particular Region). I can create a new SNS topic and S3 bucket, use a topic and bucket of my own, or I can use a topic and a bucket that belongs to a different AWS account (with proper permission):

I need to provide AWS Config with access to my AWS resources. This is done using an IAM role:

Data will begin to appear in the bucket and change notifications will be sent to the SNS topic. Here’s what the bucket looks like:

Unless you are building your own tools for AWS Config, you will probably not spend any time looking at the bucket or the data (scroll down to Inside the AWS Config Data if you want to know more). Instead, you will use the Console or a third-party tool. The Console lets you select a resource and then view configuration changes on a timeline:

Partner Support
Members of the AWS Partner Network (APN) have been working with AWS Config in order to address a variety of customer use cases.

Launch partners for AWS Config include:

1. 2^nd Watch
2. CloudCheckr
3. CloudNexa
4. Evident.IO
5. Red Hat Cloud Forms
6. RedSeal Networks
7. Splunk

Here’s what they have to offer, in their own words and screen shots!

2^nd Watch enterprise tools will allow users to visually see changes as they occur in their environment both in real-time and playback mode. The integration with AWS Config events also includes integration with New Relic application alerts, Amazon CloudWatch alarms and AWS CloudTrail events to simplify workload management. Customers have a visual tool to simplify event management and incident resolution.

AWS Config offers users the ability to create and maintain an audit history for their environment. The logs present an invaluable aid for security and compliance. The dynamic nature of the cloud, however, presents challenges for properly leveraging the logs. CloudCheckr‘s compliance policy engine already converts AWS CloudWatch metrics and CloudTrail logs into actionable information. AWS Config represents a natural extension further into this area.

Cloudnexa integrates with AWS Config to get a snapshot of resources in the AWS account, and for audit of historical configuration changes. This capability makes it unnecessary for Cloudnexa to design, build and maintain software and infrastructure to get these features.

AWS Config allows Red Hat CloudForms customers to enforce policies and ensure compliance for workloads running in Amazon Web Services. This extends the same level of control that CloudForms customers already enjoyed for virtualization and private cloud workloads to the public cloud.

AWS Config enables customers to track and store the history of Amazon VPC configurations and configuration changes in Amazon S3. With AWS Config, RedSeal customers get even more information so they can strengthen the defenses on their AWS-based networks.

Splunk provides software and cloud services that enable you to collect, index and harness machine data generated by the applications, servers, networks, sensors and other systems that power your business. The Splunk App for AWS, integrated with AWS Config, enables you to gain real-time and historical visibility into the configuration of AWS resources and how these resources relate to one another. You can also use the app to correlate data from AWS Config and AWS CloudTrail in order to gain a comprehensive view into security and compliance in your AWS account.

Inside the AWS Config Data (Developers Only)
Let’s take an inside look at the data generated by AWS Config. Here is a small portion of the snapshot data associated with a single EC2 instance. As you can see it includes complete identifying information, lists the set of tags on the instance, and describes the relationships that the instance has with a security group and an EBS volume:

{
  "configurationItemVersion" : "1.0",
  "configurationItemCaptureTime":"2014-10-28T02:30:36.989Z",
  "configurationStateId":2,
  "relatedEvents":["f8cdf490-3ddc-41ac-9cfd-9e1268dfba93"],

  "awsAccountId":"448164394201",
  "configurationItemStatus":"OK",
  "resourceId":"i-7053641e",
  "ARN":"arn:aws:ec2:us-east-1:348414629041:instance/i-7053641e",
  "awsRegion":"us-east-1",
  "availabilityZone":"us-east-1b",
  "configurationStateMd5Hash":"6ae267fafa03d87827137290c8b303e2",
  "resourceType":"AWS::EC2::Instance",
  "resourceCreationTime":"2013-04-26T19:36:06.000Z",

  "tags":{
   "UserTagDemo":"TemporaryTag",
   "Name":"RoadTripBlogServer"
  },

  "relationships":[
  {
    "resourceId":"sg-6e371c06",
    "resourceType":"AWS::EC2::SecurityGroup",
    "name":"Is associated with SecurityGroup"
  },

  {
    "resourceId":"vol-357a5f6c",
    "resourceType":"AWS::EC2::Volume",
    "name":"Is attached to Volume"
  }
  ]
}

AWS Config will send a notification to the given SNS topic each time it detects a change. The body of the notification contains detailed information about the change:

{
  "configurationItemDiff":{
    "changedProperties":{
    },
    "changeType":"CREATE"
  },

  "configurationItem":{
    "configurationItemVersion":"1.0",
    "configurationItemCaptureTime":"2014-11-04T02:28:33.146Z",
    "configurationStateId":1,
    "relatedEvents":[
       "f8cdf490-3ddc-41ac-9cfd-9e1268dfba93"
    ],

    "awsAccountId":"448164394201",
    "configurationItemStatus":"ResourceDiscovered",
    "resourceId":"vol-02fecb4d",
    "ARN":"arn:aws:ec2:us-east-1:348414629041:volume/vol-02fecb4d",
    "awsRegion":"us-east-1",
    "availabilityZone":"us-east-1a",
    "configurationStateMd5Hash":"16772ac8f8ccc7ed493a878f3bd88f8c",
    "resourceType":"AWS::EC2::Volume",
    "resourceCreationTime":"2014-11-04T02:25:10.281Z",
    "tags":{ },
    "relationships":[ ],

   "configuration":{
      "volumeId":"vol-02fecb4d",
      "size":2,
      "snapshotId":"",
      "availabilityZone":"us-east-1a",
      "state":"available",
      "createTime":"2014-11-04T02:25:10.281Z",
      "attachments":[ ],
      "tags":[ ],
      "volumeType":"gp2",
      "iops":6,
      "encrypted":false
    }
  },
  "notificationCreationTime":"2014-11-04T02:28:33.345Z",
  "messageType":"ConfigurationItemChangeNotification",
  "recordVersion":"1.2"
}

AWS Config will also send an SNS notification each time it stores a new snapshot of the current configuration.

AWS Config APIs
AWS Config provides two APIs that allow you to retrieve the resource configuration information:

• GetResourceConfigHistory – Look up configurations for a given resource within a given historical time range.
• DeliverConfigSnapshot – Trigger the creation of a full snapshot of your resources for delivery to S3.

Pricing and Availability
AWS Config is available in limited preview form and you can start using it today in the US East (N. Virginia) Region. We plan to make it available in all public AWS Regions

With AWS Config, you are charged based on the number of resources and configuration changes recorded for supported resources in your AWS account (Configuration Items). There is no up-front commitment and you can stop recording Configuration Items at any time.

You will be charged $3.00 per 1000 Configuration Items recorded per month. Standard S3 rates apply for the storage of Configuration snapshots and Configuration history files. Standard rates also apply to any notifications delivered via SNS.

If you generate 10,000 Configuration Items per month, you can expect to pay less than $0.13 per month in S3 storage charges. The AWS Free Tier provides you will 1 million SNS notifications per month (you’ll get about 10,000 notifications if you have 10,000 Configuration Items).

— Jeff;