Best practices for deploying EC2 instances with AWS CloudFormation

Deploying an Amazon Elastic Compute Cloud (Amazon EC2) instance with AWS CloudFormation is a relatively simple process. The challenges arise as you begin to customize the instance to suit your environment. What you deploy in a private subnet will have different security constraints compared with what you deploy in a public subnet.

In this post, I want to give you some guidance to ensure that no matter how you deploy an instance, you apply what I consider to be the best practices for bootstrapping an EC2 instance.

When building an instance for your infrastructure, you must consider several aspects of security and access control, and configuration.

Security and access control

To illustrate today’s topic, I have chosen one of our most used Quick Starts: The Linux bastion hosts Quick Start.

Because a bastion host is on the public side of the DMZ and is exposed to attack, it is a “hardened” server. As such, it’s a good example for highlighting instance-deployment best practices.

When I build security around an instance, I like to approach it in layers. I work my way in from the perimeter of the network to the server. Then I work in from the server to the various resources that the server will access.

During this process, I consider the following:

• Security groups
• AWS Identity and Access Management (IAM) on EC2 instances, including the following:
  • Amazon CloudWatch
  • Amazon Simple Storage Service (Amazon S3)
  • Instance profiles

Security groups

Security groups help to ensure that only those with the proper access can traverse the bastion host into your network. Use security groups inside your network to restrict access between resources.

Because the bastion host is essentially a gateway into your network, you must ensure that the appropriate port is open and that you restrict access to a specific IP address or IP range.

The port restriction will depend on what kind of instance you are deploying. It’s up to you, as the orchestrator, to determine which port or ports must be allowed input into your instance.

Because this is a bastion host and you will access the server by using Secure Shell (SSH), allow input only on port 22.

Important: Inbound rules should never specify 0.0.0.0/0. This value should be set as needed. This example relies on a parameter that is provided at runtime (!Ref RemoteAccessCIDR) and the only allowed port is port 22.

BastionSecurityGroup:  
    Type: 'AWS::EC2::SecurityGroup'  
    Properties:  
      GroupDescription: Enables SSH Access to Bastion Hosts  
      VpcId: !Ref VPCID  
      SecurityGroupIngress:  
        - IpProtocol: tcp  
          FromPort: 22  
          ToPort: 22  
               CidrIp: !Ref RemoteAccessCIDR
             - IpProtocol: icmp  
               FromPort: -1  
               ToPort: -1  
               CidrIp: !Ref RemoteAccessCIDR

The following diagram shows the external user accessing the bastion host.

1. Specify inbound access only for the ports necessary.
2. Grant inbound access only to the IP address or IP range needed. In this case, I allow the IP range 10.0.0.1/28.

You have established inbound controls for your instance by using security groups. Now I’ll show you how to use IAM to further secure your EC2 instance.

IAM on EC2 instances

Now that your instance has been secured from the outside, let’s look at the access your host needs to other resources within your environment. For a bastion host, typically the access is to other EC2 instances that have no inbound access from outside the network. Your bastion host must present valid credentials to the other instances before they in turn will grant it access. To achieve this, you can use AWS Security Token Service (AWS STS). The AWS STS service issues temporary credentials to your bastion host.

BastionHostRole:  
  Condition: CreateIAMRole  
  Type: 'AWS::IAM::Role'  
  Properties:  
    Path: /  
    AssumeRolePolicyDocument:  
      Statement:  
        - Action:  
            - 'sts:AssumeRole'  
               Principal:  
                 Service:  
                   - ec2.amazonaws.com  
               Effect: Allow  
           Version: 2012-10-17  
         ManagedPolicyArns:  
           - 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM'

The IAM role grants the bastion host the needed AWS STS credentials to access other EC2 instances. You can also use AWS STS to request access to different AWS services.

CloudWatch

To monitor your instances and log various events and activities, you can use CloudWatch.

As you bootstrap your bastion host, you must first create a log group. The log group is a container for the logging activity that is originated from the instance.

BastionMainLogGroup:  
  Type: 'AWS::Logs::LogGroup'  

Now that you have created the log group, you must grant your bastion host access to act upon that log group.

- Action:  
    - 'logs:CreateLogStream'  
    - 'logs:GetLogEvents'  
    - 'logs:PutLogEvents'  
    - 'logs:DescribeLogGroups'  
    - 'logs:DescribeLogStreams'  
    - 'logs:PutRetentionPolicy'  
    - 'logs:PutMetricFilter'  
    - 'logs:CreateLogGroup'  
       Resource: !Sub   
         - arn:${Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${BastionMainLogGroup}:*  
         - Partition: !If   
             - GovCloudCondition  
             - aws-us-gov  
             - aws  
       Effect: Allow

Under Action, you specify the different the activities in CloudWatch that you want manage.

Under Resource, you restrict these activities to a specific log group.

Under Effect, you define whether to allow the activities to be performed.

Amazon S3

When bootstrapping instances, you often need to access to Amazon S3 to retrieve additional software or to store the results of tasks performed on the instance. It is best to make sure that access is to S3 is scoped as tightly as possible. For the bastion host, you will need access to retrieve data from S3.

- Action:  
    - 's3:GetObject'  
  Resource: !Sub   
    - 'arn:${Partition}:s3:::${QSS3BucketName}/${QSS3KeyPrefix}*'  
    - Partition: !If   
        - GovCloudCondition  
        - aws-us-gov  
        - aws  
  Effect: Allow

Under Action, specify the activity in Amazon S3 that you want to manage.

Under Resource, restrict the activity to a specific S3 bucket.

Under Effect, define whether to allow the activity to be performed.

Instance profiles

To conclude the IAM section, associate your roles and policies with your instance by using the instance profile.

BastionHostProfile:  
  DependsOn: BastionHostPolicy  
  Type: 'AWS::IAM::InstanceProfile'  
  Properties:  
    Roles:  
      - !If   
        - CreateIAMRole  
        - !Ref BastionHostRole  
        - !Ref AlternativeIAMRole  
         Path: /

The BastionHostProfile relies on the BastionHostPolicy and either the BastionHostRole that is created as part of the CloudFormation stack or a role provided at runtime (AlternativeIAMRole).

Configuration

For configuration of your EC2 instance, you can use user data and AWS::CloudFormation::Init with CloudFormation helper scripts.

When an EC2 instance is launched, the user data prepares the instance for use by executing scripts on the instance at startup. This is where you execute the tasks that you require for your instance build.

For a bastion host, the user data section is extensive, comprising calls to install different packages and perform updates. This is where the helper scripts come in. The helper scripts are Python scripts that assist in the buildup of your instance. These helper scripts are:

• cfn-init
• cfn-signal
• cfn-get-metadata
• cfn-hup

For this post, I want to focus on two of the helper scripts: cfn-init and cfn-signal.

cfn-init

The cfn-init script directs your build to execute the configuration as defined in the init section of your CloudFormation template.

- 'cfn-init -v --stack '  
- !Ref 'AWS::StackName'  
- ' --resource BastionLaunchConfiguration --region '  
- !Ref 'AWS::Region'

The cfn-init script invokes the AWS::CloudFormation::Init type. Typically, the CloudFormation Init contains multiple config sets, and the cfn-init script will invoke these in the desired order of execution. However, when there is only a single config set inside the CloudFormation Init, there is no need to specify the config set.

For this application, the command:

cfn-init -v --stack !Ref 'AWS::StackName’ --resource BastionLaunchConfiguration --region !Ref 'AWS::Region’ 

is the same as

cfn-init -v --stack !Ref 'AWS::StackName’ --resource BastionLaunchConfiguration --region !Ref 'AWS::Region’ --configsets ‘config’

'AWS::CloudFormation::Init':  
  config:  
    files:  
      /tmp/bastion_bootstrap.sh:  
        source: !If   
          - UseAlternativeInitialization  
          - !Ref AlternativeInitializationScript  
          - !Sub   
            - https://${QSS3BucketName}.${QSS3Region}.amazonaws.com/${QSS3KeyPrefix}scripts/bastion_bootstrap.sh  
                 - QSS3Region: !If   
                     - GovCloudCondition  
                     - s3-us-gov-west-1  
                     - s3  
             mode: '000550'  
             owner: root  
             group: root  
             authentication: S3AccessCreds  
         commands:  
           b-bootstrap:  
             command: !Join   
               - ''  
               - - ./tmp/bastion_bootstrap.sh  
                 - ' --banner '  
                 - !Ref BastionBanner  
                 - ' --enable '  
                 - !Ref EnableBanner  
                 - ' --tcp-forwarding '  
                 - !Ref EnableTCPForwarding  
                 - ' --x11-forwarding '  
                 - !Ref EnableX11Forwarding

The cfn-init directs the execution of the config sets inside of the CloudFormation Init.

The single config set (config) in this example includes two sections: files and commands. The files section retrieves a file from Amazon S3. The commands section executes the file and passes along relevant parameters based on previous inputs.

Now that the tasks are executing, you must notify AWS CloudFormation when the tasks are complete and whether they are successful. For this, you can use cfn-signal.

cfn-signal

The signaling process in a CloudFormation template is composed of two parts:

• A creation policy that tells the CloudFormation template how many signals to expect
• The signal itself

When the CloudFormation template execution comes across a creation policy, it halts execution of its tasks until the number of signals specified in the creation policy is received or a timeout occurs.

If the signal is a failure or a timeout occurs, the stack launch fails and, if appropriate, the stack is rolled back.

BastionAutoScalingGroup:  
  Type: 'AWS::AutoScaling::AutoScalingGroup'  
  Properties:  
    LaunchConfigurationName: !Ref BastionLaunchConfiguration  
    VPCZoneIdentifier:  
      - !Ref PublicSubnet1ID  
      - !Ref PublicSubnet2ID  
    MinSize: !Ref NumBastionHosts  
    MaxSize: !Ref NumBastionHosts  
         Cooldown: '300'  
         DesiredCapacity: !Ref NumBastionHosts  
         Tags:  
           - Key: Name  
             Value: LinuxBastion  
             PropagateAtLaunch: true  
       CreationPolicy:  
         ResourceSignal:  
           Count: !Ref NumBastionHosts  
           Timeout: PT30M  

In this example, the creation policy indicates to AWS CloudFormation to wait for a signal from each of the bastion hosts being deployed, or to time out after 30 minutes.

- 'cfn-signal -e $? --stack '  
- !Ref 'AWS::StackName'  
- ' --resource BastionAutoScalingGroup --region '  
- !Ref 'AWS::Region'  

1. The creation policy in the Auto Scaling group notifies AWS CloudFormation of the number of signals and timeouts to expect.
2. When an instance comes online, a signal is sent to AWS CloudFormation.
3. AWS CloudFormation halts execution of subsequent tasks until the number of signals expected is received or a timeout occurs.

Conclusion

In this overview of best practices for deploying EC2 instances by using AWS CloudFormation, I covered the following:

• Protecting your outer perimeter with security groups
• Controlling access to other account resources by using IAM
• Using various aspects of user data and cfn-init scripts to perform bootstrapping operations on your instances
• Enabling signaling within your stack to let AWS CloudFormation know if the bootstrapping operations completed successfully

The key takeaway from this post is to always consider what resources and services your instance needs to access, and to scope your permissions tightly around those requirements.

I hope that seeing these bootstrapping best practices in context within an existing Quick Start will help you understand how the pieces come together to form a production-level deployment.

If you liked this post, I encourage you to visit our Quick Starts GitHub repository or the Quick Start Contributor’s Guide.