How to use the AWS Secrets Manager Agent

September 18, 2024: This post has been updated to include instructions for building the Secrets Manager Agent.

AWS Secrets Manager is a service that helps you manage, retrieve, and rotate database credentials, application credentials, API keys, and other secrets throughout their lifecycles. You can use Secrets Manager to replace hard-coded credentials in application source code with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them. Storing the credentials in Secrets Manager helps to avoid unintended access by anyone who inspects your application’s source code, configuration, or components.

In this blog post, we introduce a new feature, the Secrets Manager Agent, and walk through how you can use it to retrieve Secretes Manager secrets.

New approach: Secrets Manager Agent

Previously, if you had an application that used Secrets Manager and needed to retrieve secrets, you had to use the AWS SDK or one of our existing caching libraries. Both these options are specific to a certain coding language and allow only limited scope for customization.

The Secrets Manager Agent is a client-side agent that allows you to standardize consumption of secrets from Secrets Manager across your AWS compute environments. (AWS has published the code for the agent as open source code.) Secrets Manager Agent pulls and caches secrets in your compute environment and allows your applications to consume secrets directly from the in-memory cache. The Secrets Manager Agent opens a localhost port inside your application environment. With this port, you fetch the secret value from the local agent instead of making network calls to the service. This allows you to improve the overall availability of your application while reducing your API calls. Because the Secrets Manager Agent is language agnostic, you can install the binary file of the agent on many types of AWS compute environments.

Although you can use this feature to retrieve and cache secrets in your application’s compute environment, the access controls for Secrets Manager secrets remain unchanged. This means that AWS Identity and Access Management (IAM) principals need the same permissions as if they were to retrieve each of the secrets. You will need to provide GetSecretValue and DescribeSecret permissions to the secrets that you want to consume by using the Secrets Manager Agent.

The Secrets Manager Agent offers protection against server-side request forgery (SSRF). When you install the Secrets Manager Agent, the script generates a random SSRF token on startup and stores it in the file /var/run/awssmatoken. The token is readable by the awssmatokenreader group that the install script creates. The Secrets Manager Agent denies requests that don’t have an SSRF token in the header or that have an invalid SSRF token.

Solution overview

The Secrets Manager Agent provides a language-agnostic way to consume secrets in your application code. It supports various AWS compute services, such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and AWS Lambda functions. In this solution, we share how you can install the Secrets Manager Agent on an EC2 machine and retrieve secrets in your application code by using CURL commands. See the AWS Secrets Manager Agent documentation to learn how you can use this agent with other types of compute services.

Prerequisites

You need to have the following:

An AWS account
The AWS Command Line Interface (AWS CLI) version 2
jq

Follow the steps on the Install or update to the latest version of the AWS CLI page to install the AWS CLI and the Configure the AWS CLI page to configure it.

Create the secret

The first step will be to create a secret in Secrets Manager by using the AWS CLI.

To create a secret

Enter the following command in a terminal to create a secret:

aws secretsmanager create-secret --name MySecret --description "My Secret" \
  --secret-string "{\"user\": \"my_user\", \"password:\": \"my-password\"}"

You will see an output like the following:

% aws secretsmanager create-secret —name MySecret —description "My Secret" \
 —secret-string "{\"user\": \"my_user\", \"password:\": \"my-password\"}"
{
 "ARN": "arn:aws:secretsmanager:us-east-1:XXXXXXXXXXXX:secret:MySecret-LrBlpm",
 "Name": "MySecret",
 "VersionId": "b5e73e9b-6ec5-4144-a176-3648304b2d60"
}

Record the secret ARN as <SECRET_ARN>, because you will use it in the next section.

Create the IAM role

The Lambda function, the EC2 instance, and the ECS task definition need an IAM role that grants permission to retrieve the secret you just created.

To create the IAM role

Using an editor, create a file named ec2_iam_policy.json with the following content:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        } 
    ]
}

Type the following command in a terminal to create the IAM role:

aws iam create-role --role-name ec2-secret-execution-role \
  --assume-role-policy-document file://ec2_iam_policy.json

Create a file named iam_permission.json with the following content, replacing <SECRET_ARN> with the secret ARN you noted earlier:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "<SECRET_ARN>"
        }
    ]
}

Type the following command to create a policy:
```
aws iam create-policy \
  --policy-name get-secret-policy \
  --policy-document file://iam_permission.json
```
Record the Arn as <POLICY_ARN>, because you will need that value next.
Type the following command to add this policy to the IAM role, replacing <POLICY_ARN> with the value you just noted:
```
aws iam attach-role-policy \
  --role-name ec2-secret-execution-role \
  --policy-arn <POLICY_ARN>
```

Type the following command to add the AWS Systems Manager policy to the role:

aws iam attach-role-policy \
  --role-name ec2-secret-execution-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Launch an EC2 instance

Use the steps in this section to launch an EC2 instance.

To create an instance profile

Type the following command to create an instance profile:

aws iam create-instance-profile --instance-profile-name secret-profile

Type the following command to associate this instance profile with the role you just created:

aws iam add-role-to-instance-profile --instance-profile-name secret-profile \
  --role-name ec2-secret-execution-role

To create a security group

Type the following command to create a security group:
```
aws ec2 create-security-group --group-name secret-security-group \
  --description "Secrets Manager Security Group"
```
Record the group ID as <GROUP_ID>, because you will need this value in the next step.

To launch an EC2 instance

Run the following command to launch an EC2 instance, replacing <GROUP_ID> with the security group ID:

aws ec2 run-instances \
  --image-id resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 \
  --instance-type t3.micro \
  --security-group-ids <GROUP_ID> \
  --iam-instance-profile Name=secret-profile \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=secret-instance}]'

Record the InstanceId value as <INSTANCE_ID>.

Check the status of this launch by running the following command:

aws ec2 describe-instances --filters Name=tag:Name,Values=secret-instance | \
  jq ".Reservations[0].Instances[0].State"

You will see a response like the following, which shows that the instance is running:

% aws ec2 describe-instances —filters Name=tag:Name,Values=secret-instance | jq ".Reservations[0].Instances[0].State"
{
 "Code": 16,
 "Name": "running"
}

After the instance is in running state, type the following command to connect to the EC2 instance, replacing <INSTANCE_ID> with the value you noted earlier:
```
aws ssm start-session --target <INSTANCE_ID>
```

Leave the session open, because you will use it in the next step.

Install the Secrets Manager Agent to the EC2 instance

Use the steps in this section to install the Secrets Manager Agent in the EC2 instance. You will run these commands in the EC2 instance you created earlier.

To download the Secrets Manager Agent code

Type the following command to install git in the EC2 instance:
```
sudo yum install -y git 
```
Type the following command to download the Secrets Manager Agent code:
```
cd ~;git clone https://github.com/aws/aws-secretsmanager-agent
```

To build the Secrets Manager Agent

Type the following command to install the development tools:
```
sudo yum -y groupinstall "Development Tools"
```

Type the following commands to install the Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
	. "$HOME/.cargo/env"

Type the following commands to build the agent:

cd aws-secretsmanager-agent
	cargo build --release

To install the Secrets Manager Agent

Type the following command to install the Secrets Manager Agent:

cd aws-secretsmanager-agent/aws_secretsmanager_agent/configuration
sudo ./install

To grant permission to read the token file

Type the following command to copy the token file and grant permission for the current user (ec2-user) to read it:
```
sudo cp /var/run/awssmatoken /tmp
sudo chown ssm-user /tmp/awssmatoken
```

Retrieve the secret

Now you can use the local web server to retrieve the agent. Processes running in this EC2 instance can retrieve the secret with a REST API call from the web server.

To retrieve a secret

Retrieving a secret is now possible for the process in this EC2 instance, thanks to the local agent.

Run the following command to retrieve the secret:

curl -H "X-Aws-Parameters-Secrets-Token: $(</tmp/awssmatoken)” localhost:2773/secretsmanager/get?secretId=MySecret

You will see the following output:

$ curl -H "X-Aws-Parameters-Secrets-Token: $(</tmp/awssmatoken)" localhost:2773/secretsmanager/get?secretId=MySecret
{"ARN":"arn:aws:secretsmanager:us-east-1:XXXXXXXXXXXX:secret:MySecret-3z00LH","Name":"MySecret","VersionId":"e7b07d00-a0e8-41b9-b76e-45bdd8daca4f","SecretString":"{\"user\": \"my_user\", \"password:\": \"my-password\"}","VersionStages":["AWSCURRENT"],"CreatedDate":"1716912317.961"}

Exit from the EC2 instance by typing exit.

Clean up

Follow the steps in this section to clean up the resources created by the solution.

To terminate the EC2 instance and associated resources

Type the following command to stop the EC2 instance, replacing <INSTANCE_ID> with the EC2 InstanceId received at the time of instance launch:
```
aws ec2 terminate-instances --instance-ids <INSTANCE_ID>
```

Run the following command to delete the security group:

aws ec2 delete-security-group --group-name secret-security-group

Run the following command to delete the IAM role from the instance profile:

aws iam remove-role-from-instance-profile --instance-profile-name secret-profile \
  --role-name ec2-secret-execution-role

Run these commands to delete the instance profile:

aws iam delete-instance-profile --instance-profile-name secret-profile

To clean up the IAM role

Run the following command to delete the policy role, replacing <POLICY_ARN> with the value you noted earlier:
```
aws iam detach-role-policy --role-name ec2-secret-execution-role \
  --policy-arn <POLICY_ARN>
```

Run the following command to detach the policy from the role:

aws iam detach-role-policy --role-name ec2-secret-execution-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Run the following command to delete the IAM role:

aws iam delete-role --role-name ec2-secret-execution-role

To clean up the secret

Run the following command to delete the secret:

aws secretsmanager delete-secret --secret-id MySecret

Conclusion

In this post, we introduced the Secrets Manager Agent and showed how to install it in an EC2 instance, allowing the retrieval of secrets from Secrets Manager. An application can call this web server to retrieve secrets without using the AWS SDK. See the AWS Secrets Manager Agent documentation to learn more about how you can use this Secrets Manager Agent in other compute environments.

To learn more about AWS Secrets Manager, see the AWS Secrets Manager documentation.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

AWS Security Blog