AWS Developer Tools Blog

New: Improved flexibility when configuring endpoint URLs with the AWS SDKs and tools

The AWS SDKs and Tools team is excited to announce improvements for configuration of the endpoint URL used for API service requests through the shared SDK configuration file and environment variables with the AWS SDKs and Tools.

Previously, you could specify the endpoint URL used for AWS requests by setting the --endpoint-url command line parameter when invoking the AWS CLI or supplying an endpoint to an SDK client constructor. When switching between testing and production environments like Amazon DynamoDB local and the Amazon DynamoDB web service, you must provide different endpoints directly. Now, you can configure the endpoint URL using the same mechanisms as other AWS CLI or SDK configuration parameters. In addition, you can specify the endpoint to be used for all AWS services or for a specific AWS service.

This post demonstrates how to configure the endpoint URL when using the local implementations of Amazon DynamoDB local and AWS Step Functions Local through the AWS Command Line Interface (AWS CLI).

Prerequisites

In order to use this feature with the AWS CLI, you must have AWS CLI 1.29.0 or greater, or 2.13.0 or greater, installed. To install the AWS CLI, follow the instructions in the “Install or update the latest version of the AWS CLI” page of the AWS CLI User Guide.

This post makes use of Docker container images for the local implementations of Amazon DynamoDB local and AWS Step Functions Local. This post assumes that you can run these Docker container images. See the respective reference documentation pages for each of these services to learn more about how to run these services locally.

Configure the endpoint URL for all API requests

First, we demonstrate how to set the endpoint URL that will be used for all API requests made with the AWS CLI. The Amazon DynamoDB local implementation can be started in the following way to open port 8080 to accept API requests and direct them to port 8000 inside of the Docker container:

$ docker run --rm -p 8080:8000 amazon/dynamodb-local

Note: the --rm flag indicates to clean up the container when the command exits. By default, a container’s file system persists even after the container exits.

You can connect to this port using the URL http://localhost:8080. You can use this endpoint URL with the AWS CLI’s aws dynamodb commands to send requests to the Amazon DynamoDB local service running in the Docker container.

Previously, you would only be able to use this endpoint URL with the --endpoint-url parameter for the AWS CLI. Let’s create a table in our local implementation using the --endpoint-url parameter:

$ aws --endpoint-url http://localhost:8080/ dynamodb create-table \
  --table-name MyLocalTable \
  --attribute-definitions AttributeName=ColumnA,AttributeType=S AttributeName=ColumnB,AttributeType=S \
  --key-schema AttributeName=ColumnA,KeyType=HASH AttributeName=ColumnB,KeyType=RANGE \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Using environment variables

With the new feature, you can set the endpoint via the environment variable AWS_ENDPOINT_URL, and now you can list your newly created MyLocalTable table:

$ export AWS_ENDPOINT_URL=http://localhost:8080/
$ aws dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

To stop using the AWS_ENDPOINT_URL configured endpoint URL, you must unset it in your environment:

$ unset AWS_ENDPOINT_URL

Using the SDK shared configuration file

This new feature also allows you to set the endpoint URL in a profile in the SDK shared configuration file. Add the following profile to your shared configuration file (by default, this file is at ~/.aws/config):

[profile local]
endpoint_url=http://localhost:8080/

Now you can use the profile local in the AWS CLI command and put an item in the MyLocalTable table:

$ aws --profile local dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

Configure the endpoint URL for specific services

A limitation of setting the endpoint URL through the AWS_ENDPOINT_URL environment variable or endpoint_url profile property is that all API requests, including those to other AWS services will be directed to this endpoint URL. This might be convenient if you are only using a single AWS service, or if you are using a third-party local development environment like LocalStack.

However, you might want more control over which AWS services use a different endpoint URL. To only send DynamoDB API requests to the endpoint, you can configure service-specific endpoint URLs through environment variables and the SDK shared configuration file.

Using environment variables

To set a service-specific endpoint URL for Amazon DynamoDB API requests, use the AWS_ENDPOINT_URL_DYNAMODB environment variable and retrieve the item we previously inserted:

$ AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8080/ aws dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

This technique is applicable to any AWS service. A list of the available can be found in the “Identifiers for service-specific endpoints“ documentation.

Using the SDK shared configuration file

You can also now configure a service-specific endpoint URL in a profile in the SDK shared configuration file. To do this, you will use a services section and associate it with a profile. A services section contains sub-sections for the AWS service(s) that you want to configure. The services section is referenced in a profile through the services profile parameter.

This technique is applicable to any AWS service. A complete list of the available service section names can be found in the “Identifiers for service-specific endpoints“ documentation.

For example, to configure the AWS DynamoDB endpoint to the local Docker container, add the following profile and services section to your shared configuration file:

[profile production]
region=us-west-2

[profile local]
region=us-west-2
services=local-services

[services local-services]
dynamodb =
  endpoint_url=http://localhost:8080/

Now you can use the profile local in the AWS CLI command to make requests to the Amazon DynamoDB local installation:

$ aws --profile local dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

Then, with no other configuration changes, you can specify the production profile to make requests to the Amazon DynamoDB web service:

$ aws --profile production dynamodb list-tables

Note: the output of this command depends if you have created any tables in the Amazon DynamoDB web service.

This enables you to keep the configuration that determines which endpoint URL to use outside of your scripts and in an officially supported configuration location.

For details on the syntax of the shared configuration file when specifying a custom endpoint URL, see the section “Configure endpoints using the shared config file” in the “Service-specific endpoints“ documentation.

Configure the endpoint URL for multiple services

You can also use the AWS CLI with multiple AWS services. This new feature allows you to configure separate custom endpoint URLs for each service at the same time. A list of the environment variable and service section names can be found on the “Identifiers for service-specific endpoints” documentation page. For example, to use AWS Step Functions local at the same time as Amazon DynamoDB local, you can use the environment variable AWS_ENDPOINT_URL_SFN or services section sfn to set the endpoint when using aws stepfunctions commands.

The AWS Step Functions local implementation can be started with the following command:

$ docker run --rm -p 8083:8083 amazon/aws-stepfunctions-local

Similar to the example for Amazon DynamoDB local, you can connect to this port using the URL http://localhost:8083. You can use this endpoint URL with the AWS CLI’s aws stepfunctions commands to send requests to the AWS Step Functions local service running in the Docker container.

To use both local implementations, you can set both environment variable at the same time to their respective endpoint URLs:

$ export AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8080/
$ export AWS_ENDPOINT_URL_SFN=http://localhost:8083/

Then create an AWS Step Functions state machine using our local implementation:

$ aws stepfunctions create-state-machine \
  --name HelloWorldLocal \
  --definition '{"StartAt": "HelloWorld", "States": {"HelloWorld": {"Type": "Pass", "Result": "Hello World", "End": true}}}' 
  --role-arn "arn:aws:iam::123456789012:role/HelloWorldLocalRole"
{
    "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldLocal",
    "creationDate": "2023-07-12T13:48:26.507000-07:00"
}

Now, list the tables and state machines:

$ aws dynamodb list-tables  # Uses the endpoint from AWS_ENDPOINT_URL_DYNAMODB
{
    "TableNames": [
        "MyLocalTable"
    ]
}
$ aws stepfunctions list-state-machines  # Uses the endpoint from AWS_ENDPOINT_URL_SFN
{
    "stateMachines": [
         {
            "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldLocal",
            "name": "HelloWorldLocal",
            "type": "STANDARD",
            "creationDate": "2023-07-12T13:48:26.507000-07:00"
        }
    ]
}

As before, unset the environment variables to stop using the configured endpoint URLs:

$ unset AWS_ENDPOINT_URL_DYNAMODB
$ unset AWS_ENDPOINT_URL_SFN

You can also define both endpoints in the SDK shared configuration file and specify the profile when calling the AWS CLI. Add the following profile and services section to your shared configuration file:

[profile local]
region=us-west-2
services=local-services

[services local-services]
dynamodb =
  endpoint_url=http://localhost:8080/
sfn =
  endpoint_url=http://localhost:8083/

Invoking the AWS CLI with --profile local will use the specified Amazon DynamoDB and AWS Step Functions endpoint URLs for aws dynamodb and aws stepfunctions commands, respectively:

$ aws dynamodb list-tables --profile local
{
    "TableNames": [
        "MyLocalTable"
    ]
}
$ aws stepfunctions list-state-machines --profile local
{
    "stateMachines": [
        {
            "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldLocal",
            "name": "HelloWorldLocal",
            "type": "STANDARD",
            "creationDate": "2023-07-12T13:48:26.507000-07:00"
        }
    ]
}

Order of precedence for configuring the endpoint URL

You can configure the endpoint URL using any of the above methods at the same time. The endpoint URL used is determined by the AWS CLI with the following order of precedence:

  1. The value provided from the --endpoint-url command line parameter.
  2. The value provided by a service-specific environment variable (like AWS_ENDPOINT_URL_DYNAMODB).
  3. The value provided by the global endpoint environment variable (AWS_ENDPOINT_URL).
  4. The value provided by the endpoint_url parameter from a services definition section in the shared configuration file.
  5. The value provided by the endpoint_url parameter from a profile in the shared configuration file.

If an endpoint URL is not configured in any of these ways, the AWS CLI uses the default endpoint URL for the respective AWS service.

For more information about the order of precedence for AWS SDK parameters, see the section “Precedence of settings” on the “Service-specific endpoints” documentation page.

Ignoring configured endpoint URLs

If you want to ignore configured endpoint URLs from the SDK shared configuration file or environment variables, you can use the profile parameter ignore_configured_endpoint_urls=true or the environment variable AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true. Note that setting either of these parameters will not prevent the use of endpoints provided to the --endpoint-url command line parameter. To read more about how to disable this feature, see the “Service-specific Endpoints” documentation page.

Cleaning up

If you’ve started any Docker containers, be sure to shut them down by terminating the process or closing the terminal where they were running when you’re done working through the examples in this post. Also be sure to either close the terminal window you were using or unset any environment variables used in this post:

$ unset AWS_ENDPOINT_URL
$ unset AWS_ENDPOINT_URL_DYNAMODB
$ unset AWS_ENDPOINT_URL_SFN

Next steps

In this post, we have demonstrated how to simplify the configuration of endpoint URLs when using the AWS CLI. These examples only describe one way to use this feature. Other use cases are to configure AWS Security Token Service (AWS STS) VPC endpoints, S3-compatible object stores, or third-party local AWS development environments like LocalStack.

The AWS SDKs and tools that currently support this feature are the SDK for .NET, the SDK for Python, the SDK for Ruby, the AWS CLI v1, the AWS CLI v2, and the AWS Tools for Powershell. To learn when other SDKs support this feature, see the “Compatibility with AWS SDKs” section on the the “Service-specific endpoints” SDK documentation page.

For more information, see the “Service-specific endpoints” SDK documentation page. For any issues or feature requests, please file an issue on our GitHub repository. We look forward to hearing your feedback.

About the author:

Kenneth Daily

Kenneth Daily

Kenneth is a software development engineer working on the AWS CLI. You can find him on GitHub @kdaily.