AWS DevOps & Developer Productivity Blog

Authenticated File Downloads with CloudFormation

In this guest post, AWS Solution Architect Grace Mollison (@grapesfrog) discusses options for authenticated file downloads when using CloudFormation.

Note: This post assumes familiarity with cfn-init and CloudFormation Metadata to bootstrap the EC2 instances in your CloudFormation stack. To learn more, see Automate LAMP Installation Using CloudFormation, Bootstrapping AWS CloudFormation Windows Stacks, or the cfn-init documentation which includes several sample templates.

If you’ve ever used cfn-init and Cloudformation Metadata to bootstrap your EC2 instances, you know that you’re able to download files and sources from S3 and other URIs to EC2 instances in your stack. If the files or sources being downloaded require authentication, you can use the AWS::CloudFormation::Authentication type to declare authentication credentials that cfn-init will use to download these files. As we’ll see, authentication is supported for both S3 and other HTTP sources (e.g., GitHub).

Use Cases for Authenticated File Downloads

In the examples below we’ll illustrate downloading an HTML file from a private location, but the concept extends to any private content you need to bootstrap your EC2 instances. This might include credentials to a 3rd party API, an app configuration file that includes database connection info, etc.

Authenticated File and Source Downloads from S3

Let’s spend a few moments discussing the AWS::CloudFormation::Authentication resource before we look at template samples. The authentication resource has several properties (outlined in the documentation), but you’ll always start by defining the type property. Valid values for type are S3 or basic.

If you choose S3 as the authentication type, you can indicate the IAM Role (check out Jeff’s blog post for an introduction to IAM Roles) that will be used to authenticate the file download by specifying the roleName property. Additionally, you may use the buckets property to indicate which bucket(s) your AWS::CloudFormation::Authentication resource applies to (i.e., you can have different credentials for different buckets).
Let’s assume you want to download the file “index.html” from a bucket named your-bucket that requires authentication. First define the necessary resources to create an IAM role with GetObject access to that bucket and object: 
{
  ...
   "Resources":{
      "InstanceRole":{
         "Type":"AWS::IAM::Role",
         "Properties":{
            "AssumeRolePolicyDocument":{
               "Statement":[
                  {
                     "Effect":"Allow",
                     "Principal":{
                        "Service":[
                           "ec2.amazonaws.com"
                        ]
                     },
                     "Action":[
                        "sts:AssumeRole"
                     ]
                  }
               ]
            },
            "Path":"/"
         }
      },
      "RolePolicies":{
         "Type":"AWS::IAM::Policy",
         "Properties":{
            "PolicyName":"S3Download",
            "PolicyDocument":{
               "Statement":[
                  {
                     "Action":[
                        "s3:GetObject"
                     ],
                     "Effect":"Allow",
                     "Resource":"arn:aws:s3:::your-bucket/index.html"
                  }
               ]
            },
            "Roles":[
               {
                  "Ref":"InstanceRole"
               }
            ]
         }
      },
      "InstanceProfile":{
         "Type":"AWS::IAM::InstanceProfile",
         "Properties":{
            "Path":"/",
            "Roles":[
               {
                  "Ref":"InstanceRole"
               }
            ]
         }
      }
   }
  ...
}

Now that the role is defined, we associate it to our EC2 instance via an InstanceProfile and reference it in the roleName property of the S3AccessCreds declaration.

{
    ...
    "Resources": {
        "InstanceRole": {...},
        "RolePolicies": {...},
        "InstanceProfile": {...},
        "YourInstance": {
            "Type": "AWS::EC2::Instance",
            "Metadata": {
                "AWS::CloudFormation::Authentication": {
                    "S3AccessCreds": {
                        "type": "S3",
                        "roleName": {
                            "Ref": "InstanceRole"
                        }
                    }
                },
                "AWS::CloudFormation::Init": {
                    "config": {
                        "files": {
                            "/var/www/html/index.html": {
                                "source": "http://your-bucket.s3.amazonaws.com/index.html",
                                "mode": "000400",
                                "owner": "apache",
                                "group": "apache",
                                "authentication": "S3AccessCreds"
                            }
                        }
                    }
                }
            },
            "Properties": {
                "IamInstanceProfile": {
                    "Ref": "InstanceProfile"
                }
            }
        }
    }
    ...
}

Notice that we explicitly declare that the “/var/www/html/index.html” file should be downloaded using the S3AccessCreds resource by specifying it in the authentication property. We could have excluded this property, and instead declared which bucket(s) the credentials applied to by setting the buckets property of the S3AccessCreds resource:

"AWS::CloudFormation::Authentication": {
  "S3AccessCreds": {
    "type": "S3",
    "roleName": { "Ref" : "InstanceRole"}
    "buckets" : ["your-bucket"]
  }
}

Whichever method you choose, cfn-init will now use the role to authenticate to S3 when it downloads the file.

Downloading authenticated sources:

Downloading authenticated sources from S3 is very similar to files as described above with one important exception: sources does not support declaring the authentication property. You must specify the buckets key in the AWS::CloudFormation::Authentication declaration:

"YourInstance": {
    "Type": "AWS::EC2::Instance",
    "Metadata": {
        "AWS::CloudFormation::Authentication": {
            "S3AccessCreds": {
                "type": "S3",
                "roleName": { "Ref" : "InstanceRole"},
                "buckets" : ["your-bucket"]
            }
        },
        "AWS::CloudFormation::Init": {
            "config": {
                "sources": {
                    "/var/www/html/": "https://your-bucket.s3.amazonaws.com/mywebsite.tar"
                }
            }
        }
    }
    ...
}

Authenticated File and Source Downloads from Other URIs

You can also use cfn-init for authenticated file and sources download for other (non-S3) URIs. To do this, set the type of your AWS::CloudFormation::Authentication resource to “basic” and include the username, password, and uris properties. For example, specifying authentication credentials for downloading a file from GitHub might look like:

"AWS::CloudFormation::Authentication": {
	"GithubCredentials": {
		"type": "basic",
		"username": { "Ref": "GithubLogin" },
		"password": { "Ref": "GithubPassword" },
		"uris": ["github.com"]
	}
}

And the file download declaration:

"files": {
	"/var/www/html/index.html": {
		"source": "https://github.com/yourgithubaccount/mywebsite/myfrontpage.html",
		"mode": "000400",
		"owner": "apache",
		"group": "apache",
		"authentication": "GithubCredentials"
	}
}