Best practices for using Amazon S3 endpoints with AWS CloudFormation

If you create AWS CloudFormation templates, you can access Amazon Simple Storage Service (Amazon S3) objects using either path-style or virtual-hosted-style endpoints. This post helps you understand what endpoint patterns are, how they’ve evolved, best practices for using each, and why I recommend that you adopt virtual-hosted-style endpoints as your overall best practice.

First, I explain the difference between path-style and virtual-hosted-style endpoints. Then, I walk you through testing the endpoint styles by creating an entry for each in the AWS Systems Manager Parameter Store. Finally, I review the AWS CloudFormation master template, which includes all three patterns.

Prerequisites

This blog post assumes that you’re familiar with AWS CloudFormation templates, AWS Command Line Interface (AWS CLI), GitHub, and Git commands.

For this post, you need the following:

• An AWS account
• Access to AWS Management Console

Amazon S3 bucket endpoints

When you connect to an AWS service programmatically, you use an AWS service endpoint. To access a resource in an S3 bucket, in particular, you specify the object’s address using a RESTful API. Common use cases:  

• Reference a template URL
• Use nested stacks
• Retrieve AWS Lambda source code
• Bootstrap Amazon Elastic Compute Cloud (Amazon EC2) instances using the cfn-init helper script

Path-style S3 endpoints

Path-style S3 endpoints, which are commonly used, may fall into either of two subdomains:

• s3.amazonaws.com
• s3.<AWS Region>.amazonaws.com

Only one subdomain includes the AWS Region, and neither includes the S3 bucket name. The bucket name is part of the path. This means that these two subdomains handle all requests across AWS that use path-style endpoints.

The first path-style pattern, shown here, is sometimes called a global endpoint because it includes no Region. This pattern consists of the service name (s3) and the AWS suffix (amazonaws.com) followed by the bucket name (awsdoc-example-bucket) and key name (foo):

In this pattern, requests made to the endpoint are routed by default to the US East (N. Virginia) Region (us-east-1). If the bucket is not located in us-east-1, and if the Region where the bucket is located was launched before March 20, 2019, the request is redirected to the correct Region automatically. If the Region hosting the bucket was created after March 20, 2019, the request is not redirected, and a Bad Request error is returned.

The second path-style pattern, a type of Regional endpoint, addresses this issue by including the Region between the service name (S3) and the AWS suffix (amazonaws.com):

Virtual-hosted-style S3 endpoints

Going beyond both path styles, virtual-hosted-style S3 endpoints include both the Region and the S3 bucket name in the subdomain. Since these endpoints route requests directly to the bucket where the objects reside, they never return a Bad Request error or a redirect. In fact, according to a post that announced AWS plans to shift toward this model, this bucket-specific subdomain is “the key that opens the door to many important improvements to S3.”

The virtual-hosted style has the following pattern: bucket name (awsdoc-example-bucket), service name (s3), Region where the bucket is hosted (us-west-2), AWS suffix (amazonaws.com), and key name (foo):

Test the three endpoint patterns

Now that I’ve covered the three endpoint patterns, put your knowledge into practice by testing all three patterns. This diagram gives an overview of the two steps that I walk you through.

Step 1: Create the test S3 bucket, and prepare to test the three endpoint patterns

Clone the example repository from GitHub

1. Open your command-line application (PowerShell, Terminal, etc.).
2. Navigate to the location where you want to store the repository.
3. Clone the example repository by executing this command:

   git clone git@github.com:aws-quickstart/s3-endpoints-and-cfn.git

Create the test S3 bucket

1. In the command line, execute the following AWS CLI command, replacing <awsexamplebucket1-us-west-2> with the name of your bucket.

   aws s3 mb s3://<awsexamplebucket1-us-west-2> --region=us-west-2

2. Copy the project folder to your S3 bucket with the following command, replacing <awsexamplebucket1-us-west-2> with the name of your bucket.

   aws s3 cp --recursive s3-endpoints-and-cfn s3://<awsexamplebucket1-us-west-2>/s3-endpoints-and-cfn/

Configure parameters

1. In the command line, navigate to the parameters folder:

   cd s3-endpoints-and-cfn/parameters/

2. Use a text editor to edit each file that contains execution parameters. Replace <awsexamplebucket1-us-west-2> with the name of your bucket.

Step 2: Test each pattern using the AWS CloudFormation master template

Test pattern 1: pathstyle1

1. In the command line, execute the template with this command:

   aws cloudformation create-stack --stack-name pathstyle1 --template-body file://s3-endpoints-and-cfn/templates/master.template.yaml —parameters file://s3-endpoints-and-cfn/parameters/path-style-1.json

   This creates an entry for this style in the AWS Systems Manager Parameter Store. The command line returns output similar to the following:

   {
       "StackId": "arn:aws:cloudformation:us-east-1:000000000000:stack/pathstyle1/00000000-llll-jjjj-kkkk-000000000000"
   }

2. Enter q to exit the output screen.
3. Execute the following command:

   aws cloudformation describe-stacks --stack-name pathstyle1

   The command line returns output similar to the following, which shows the results of the stack execution.

   {
       "Stacks": [
           {
               "StackId": "arn:aws:cloudformation:us-east-1:503759272940:stack/pathstyle1/85ae8110-7dda-11ea-bc0f-0e43f2110cc5",
               "StackName": "pathstyle1",
               "Description": "Workload Template using Sigv2 S3 URLS (qs-1qkt7uig1)",
               "Parameters": [
                   {
                       "ParameterKey": "BucketName",
                       "ParameterValue": "awsexamplebucket1-us-west-2"
                   },
                   {
                       "ParameterKey": "BucketRegion",
                       "ParameterValue": "us-west-2"
                   },
                   {
                       "ParameterKey": "ChildTemplate",
                       "ParameterValue": "PathStyle-1"
                   },
                   {
                       "ParameterKey": "BucketPrefix",
                       "ParameterValue": "s3-endpoints-and-cfn/"
                   },
                   {
                       "ParameterKey": "SSMParameterValue",
                       "ParameterValue": "ps1"
                   }
               ],
               "CreationTime": "2020-04-13T23:00:08.776000+00:00",
               "RollbackConfiguration": {},
               "StackStatus": "CREATE_COMPLETE",
               "DisableRollback": false,
               "NotificationARNs": [],
               "Tags": [],
               "EnableTerminationProtection": false,
               "DriftInformation": {
                   "StackDriftStatus": "NOT_CHECKED"
               }
           }
       ]
   }

4. In the output, look for the line with the stack status ("StackStatus": "CREATE_COMPLETE"). This indicates that there were no issues with the stack execution.

Test pattern 2: pathstyle2

Repeat the same steps as for testing pattern 1, replacing all references to pathstyle1 with pathstyle2. For example, execute the template with this command:

aws cloudformation create-stack --stack-name pathstyle2 --template-body file://s3-endpoints-and-cfn/templates/master.template.yaml —parameters file://s3-endpoints-and-cfn/parameters/path-style-2.json

Test pattern 3: virtualhostedstyle

Repeat the same steps as for testing pattern 1, replacing all references to pathstyle1 with virtualhostedstyle. For example, in the command line, execute the template with this command:

aws cloudformation create-stack --stack-name virtualhostedstyle --template-body file://s3-endpoints-and-cfn/templates/master.template.yaml —parameters file://s3-endpoints-and-cfn/parameters/virtual-hosted-style.json

So far, I’ve reviewed what the three S3 endpoint patterns are, and I’ve walked through testing them. Next, I describe how an AWS CloudFormation template accesses them.

How an AWS CloudFormation template accesses the S3 endpoints

To follow along with the descriptions in this section, open the AWS CloudFormation master template:

1. Navigate to the templates folder of the repository: cd s3-endpoints-and-cfn/templates.
2. Open the master.template.yaml file using your favorite editor.
3. Review the conditions and resources sections of the AWS CloudFormation master template, as detailed in the following sections.

Conditions in the AWS CloudFormation master template

The conditions section of the master template contains elements that are required for executing endpoint patterns and determining which endpoint pattern to use.

GovCloudCondition: !Equals
    - !Ref AWS::Region
    - us-gov-west-1  
UsingDefaultBucket: !Equals 
    - !Ref BucketName
    - "mybucket"
ChildTemplate1Condition: !Equals
    - !Ref ChildTemplate
    - "PathStyle-1"
ChildTemplate2Condition: !Equals
    - !Ref ChildTemplate
    - "PathStyle-2"
ChildTemplate3Condition: !Equals
    - !Ref ChildTemplate
    - "VirtualHostedStyle-1"

• GovCloudCondition—This condition, which applies only to PathStyleStack2, is for use cases that involve the AWS GovCloud Region.
• UsingDefaultBucket—This condition, which applies only to VirtualHostedStyleStack3, is used when implementing virtual-hosted-style endpoints.
• ChildTemplate<1/2/3>Condition—These conditions indicate which pattern to use when launching the child template.

Resources in the AWS CloudFormation master template

The resources section of the master template contains calls to execute three stacks—PathStyleStack1, PathStyleStack2, and VirtualHostedStyleStack3—which represent the three endpoint patterns described earlier. As you create new AWS CloudFormation templates and update existing ones, I recommend that you use the pattern in VirtualHostedStyleStack3.

  PathStyleStack1:
    # Path-Style endpoint no region specified
    Condition: ChildTemplate1Condition
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub "https://s3.amazonaws.com/${BucketName}/${BucketPrefix}templates/template.yaml"
      Parameters:
        SSMParameterValue: !Ref "SSMParameterValue"
  PathStyleStack2:
    # Path-Style endpoint with specified region
    Condition: ChildTemplate2Condition
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub
        - "https://${S3Region}.amazonaws.com/${BucketName}/${BucketPrefix}templates/template.yaml"
        - S3Region: !If [GovCloudCondition, s3-us-gov-west-1, s3.us-west-2]
      Parameters:
        SSMParameterValue: !Ref "SSMParameterValue"
  VirtualHostedStyleStack3:
    # Virtual Hosted-Style
    Condition: ChildTemplate3Condition
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub
        - https://${BucketName}.s3.${S3Region}.${AWS::URLSuffix}/${BucketPrefix}templates/template.yaml
        - S3Region:
            !If [UsingDefaultBucket, !Ref "AWS::Region", !Ref BucketRegion]
      Parameters:
        SSMParameterValue: !Ref "SSMParameterValue"

All three stacks call the same template.yaml file. The difference is in how each stack calls the template. Let’s review each stack and examine how to implement the pattern.

PathStyleStack1

The first stack to call, PathStyleStack1, implements the first path-style pattern: a global S3 endpoint. This endpoint makes no reference to the AWS Region.

PathStyleStack1:
    # Path-Style endpoint no region specified
    Condition: ChildTemplate1Condition
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub "https://s3.amazonaws.com/${BucketName}/${BucketPrefix}templates/template.yaml"
      Parameters:
        SSMParameterValue: !Ref "SSMParameterValue"

Notice the TemplateURL construction:

To implement this first pattern, you must include the following elements: service name (s3), AWS URL suffix (amazonaws.com), and key name (templates/template.yaml). Instead of a bucket-name element, there’s a reference to the BucketName parameter (${BucketName}). A new element—the s3 prefix/folder—refers to the BucketPrefix parameter (${BucketPrefix}). The BucketPrefix, which I commonly treat as a folder structure, refers to the file structure of your S3 bucket. In this case, the path to the template appears as https://s3.amazonaws.com/awsexamplebucket1-us-west-2/s3-endpoints-and-cfn/templates/template.yaml where BucketName is awsexamplebucket1-us-west-2 and BucketPrefix is s3-endpoints-and-cfn/.

PathStyleStack2

The next stack to call, PathStyleStack1, implements the second path-style pattern: a Regional endpoint. This implementation uses the AWS GovCloud condition mentioned earlier as well as the AWS Region.

PathStyleStack2:
    # Path-Style endpoint with specified region
    Condition: ChildTemplate2Condition
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub
        - "https://${S3Region}.amazonaws.com/${BucketName}/${BucketPrefix}templates/template.yaml"
        - S3Region: !If [GovCloudCondition, s3-us-gov-west-1, s3.us-west-2]
      Parameters:
        SSMParameterValue: !Ref "SSMParameterValue"

Notice the TemplateURL construction:

Instead of separate service and Region elements, there’s an {S3Region} variable, which resolves based on GovCloudCondition:

- S3Region: !If [GovCloudCondition, s3-us-gov-west-1, s3.us-west-2]

GovCloudCondition in the preceding statement is evaluated as follows:

• If the stack is deployed in GovCloudRegion, the value of the S3Region variable is set to s3-us-gov-west-1.
• If the stack is not deployed in GovCloudRegion, the value of the S3Region variable is set to s3.us-west-2.

VirtualHostedStyleStack3

The final stack implements the virtual-hosted-style endpoint. Again, as you create new AWS CloudFormation templates and evaluate existing ones, I recommend that you use this pattern.

  VirtualHostedStyleStack3:
    Condition: ChildTemplate3Condition
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub
        - https://${BucketName}.s3.${S3Region}.${AWS::URLSuffix}/${BucketPrefix}templates/template.yaml
        - S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref BucketRegion]
      Parameters:
        SSMParameterValue: !Ref "SSMParameterValue"

Notice the TemplateURL construction:

The most important difference is that the S3 bucket name moved to the beginning of the URL. This means that your bucket has its own subdomain. The next element is the service name (s3) followed by the Region. The Region is not joined with the service name as in the previous pattern. The {S3Region} variable resolves based on the UsingDefaultBucket condition.

- S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref BucketRegion]

The UsingDefaultBucket condition in the previous statement is evaluated as follows:

• If the stack uses the default S3 bucket, the Region element is set by the pseudo parameter AWS::Region, which resolves to the Region where the AWS CloudFormation template is executed. For more on pseudo parameters, see Pseudo parameters reference.
• If the stack uses a different S3 bucket, the Region element is set by the parameter BucketRegion.

The next element in the URL is the AWS URL suffix. Whereas the other patterns use the hardcoded value amazonaws.com, this value uses another pseudo parameter: AWS::URLSuffix. This pseudo parameter allows for deployment of the AWS CloudFormation templates in special Regions that don’t use amazonaws.com, such as the China Region where the suffix is amazonaws.com.cn.

The remaining elements of the URL are the same as in the other patterns.

Cleanup

To avoid incurring future charges, delete resources as follows.

1. From the command line, delete the AWS CloudFormation stacks by executing these commands:

   aws cloudformation delete-stack --stack-name pathstyle1
   aws cloudformation delete-stack --stack-name pathstyle2
   aws cloudformation delete-stack --stack-name virtualhostedstyle

2. Delete the S3 bucket by executing this command:

   aws s3 rb s3://awsexamplebucket1-us-west-2.s3-us-west-2 --force

Conclusion

In this post, I described the three S3 endpoint patterns and the best practices for using each pattern. I walked you through testing each endpoint pattern by creating an entry for each in the AWS Systems Manager Parameter Store. Finally, I reviewed the AWS CloudFormation master template, which includes all three patterns. If you haven’t already done so, I encourage you to switch from path-style to virtual-hosted style endpoints as you create new AWS CloudFormation templates and update existing ones.

To learn more about the Amazon S3 service in general, see the AWS documentation on Amazon S3. To learn more about our recommended approach to S3 endpoints, see Virtual hosting of buckets.

Let me know your thoughts in the comments.