View CloudFormation Logs in the Console

Background

AWS CloudFormation simplifies provisioning and management on AWS. You can templatize the service and application architectures you want, and have CloudFormation use the templates for quick and reliable provisioning of the services or applications as ‘stacks’. You can also easily update or replicate the stacks as needed. Inside the CloudFormation templates, you can configure the method to install and bootstrap applications (or services) running on the stacks. You can choose from a range of options depending on how quickly you want the application to be up and running, and how flexible you need to be in terms of deploying new versions of the application. The options are: Pre-install your application into a custom Amazon Machine Image (AMI) Customize an EC2 instance at launch time using Cloud-Init Use AWS::CloudFormation::Init and the helper scripts Use other tools such as Jenkins, Chef, or Puppet, optionally triggering them from inside a CloudFormation template

Regardless of which option you use, you may need to access the application bootstrapping logs created on the EC2 instances. The logs are useful for debugging provisioning failures and as future reference. Earlier, for each instance you wanted to debug, you had to put a KeyPair on the instance at launch, and later, SSH into the instance to access the logs.

With the advent of Amazon CloudWatch Logs, it is now possible to access the application bootstrapping logs (or any other logs from an EC2 instance) right inside the AWS Management Console.

Viewing Logs

Here is a sample template  which streams the application bootstrapping logs from a CloudFormation stack (cloud-init.log, cfn-init.log, cfn-hup.log, and cfn-wire.log files) to CloudWatch Logs. The logs can then be viewed inside the CloudWatch Logs Console, simplifying the debugging of provisioning failures.

The sample template spins up a simple PHP application stack. To hold the application bootstrapping logs, the template creates a Log Group in CloudWatch Logs:

If you are creating the Log Group inside the stack that you are debugging (as this sample template does), then remember to disable the automatic rollback while kicking-off stack creation. This way, the logs are retained even when stack creation fails. Alternatively, you can create a Log Group separately and optionally use it for multiple stacks.

This sample template utilizes AWS::CloudFormation::Init to configure 1) cfn scripts including cfn-hup – the CloudFormation daemon to automatically apply subsequent configuration updates, 2) ClouwWatch logs, and 3) the simple PHP application, on an EC2 instance. The template utilizes Configsets for the desired installation order:

To configure the logs, the template 1) creates a CloudWatch Logs configuration file (cfn-logs.conf) on the instance, and 2) downloads the awslogs package and 3) starts the awslogs daemon:

You have the option to remotely store a common CloudWatch Logs configuration file and use it across multiple stacks.

To kick off the configuration specified in AWS::CloudFormation::Init and to signal back the result, the sample template runs a couple of commands in the User Data section of the EC2 instance:

You can now view the CloudFormation application bootstrapping logs in the CloudWatch Logs Console:

CloudWatch Logs is currently available in US East (Northern Virginia) Region.

Try this now: sample template .

— Chetan Dandekar, Senior Product Manager, Amazon Web Services.