How to securely provide database credentials to Lambda functions by using AWS Secrets Manager

April 29, 2022: This post has been updated based on working backwards from a customer need to securely allow access and use of Amazon RDS database credentials from a AWS Lambda function.

In this blog post, we will show you how to use AWS Secrets Manager to secure your database credentials and send them to Lambda functions that will use them to connect and query the backend database service Amazon RDS—without hardcoding the secrets in code or passing them through environment variables. This approach will help you secure last-mile secrets and protect your backend databases. Long living credentials need to be managed and regularly rotated to keep access into critical systems secure, so it’s a security best practice to periodically reset your passwords. Manually changing the passwords would be cumbersome, but AWS Secrets Manager helps by managing and rotating the RDS database passwords.

Solution overview

This is sample code: you’ll use an AWS CloudFormation template to deploy the following components to test the API endpoint from your browser:

An RDS MySQL database instance on a db.t3.micro instance
Two Lambda functions with necessary IAM roles and IAM policies, including access to AWS Secrets Manager:
- LambdaRDSCFNInit: This Lambda function will execute immediately after the CloudFormation stack creation. It will create an “Employees” table in the database, where it will insert three sample records.
- LambdaRDSTest: This function will query the Employees table and return the record count in an HTML string format
RESTful API with “GET” method on AWS API Gateway

Here’s the high level setup of the AWS services that will be created from the CloudFormation stack deployment:

Figure 1: Architecture diagram

Clients call the RESTful API hosted on AWS API Gateway
The API Gateway executes the Lambda function
The Lambda function retrieves the database secrets using the Secrets Manager API
The Lambda function connects to the RDS database using database secrets from Secrets Manager and returns the query results

You can access the source code for the sample used in this post here: https://github.com/aws-samples/aws-secrets-manager-secure-database-credentials

Deploying the sample solution

By default, the stack will be deployed in the us-east-1 region. If you want to deploy this stack in any other region, download the code from the above source code link, build a zip file using the readme file, place the Lambda code zip file in a region-specific S3 bucket and make the necessary changes in the CloudFormation template to point to the right S3 bucket. (Please refer to the AWS CloudFormation User Guide for additional details on how to create stacks using the AWS CloudFormation console.)

Next, follow these steps to execute the stack:

After login go to CloudFormation service in the AWS Console. Select on the Stacks option and then select Create stack option.

Figure 2. Search for CloudFormation in the search bar

Figure 3. Create Stack in CloudFormation
Select Template is ready and then select Upload a template file. Next click on the Choose file button to choose the template file (secretsmanager_IaC.yml) from your desktop. Click Next.

Figure 4. Upload the CloudFormation template file
On the Specify stack details page, you’ll see the parameters pre-populated. These parameters include the name of the database and the database user name. Enter a name in the Stack name field. Select Next on this screen.

Figure 5. Parameters on the “Specify stack details” page
On the Configure stack options page, select the Next button.
On the Review screen, select all three check boxes, then select the Create change set button:

Figure 6. Select the check boxes and “Create Change Set”
After the change set creation is completed, choose the Execute button to launch the stack.
Stack creation will take between 10 – 15 minutes. After the stack is created successfully, select the Outputs tab of the stack, then select the link.

Figure 7. Select the link on the “Outputs” tab

This action will trigger the code in the Lambda function, which will query the “Employee” table in the MySQL database and will return the results count back to the API. You’ll see the following screen as output from the RESTful API endpoint:

Figure 8. Output from the RESTful API endpoint

At this point, you’ve successfully deployed and tested the API endpoint with a backend Lambda function and RDS resources. The Lambda function is able to successfully query the MySQL RDS database and is able to return the results through the API endpoint.

What’s happening in the background?

The CloudFormation stack deployed a MySQL RDS database with a randomly generated password using a secret resource. Now that the secret resource with randomly generated password has been created, the CloudFormation stack will use dynamic reference to resolve the value of the password from Secrets Manager in order to create the RDS instance resource. Dynamic references provide a compact, powerful way for you to specify external values that are stored and managed in other AWS services, such as Secrets Manager. The dynamic reference guarantees that CloudFormation will not log or persist the resolved value, keeping the database password safe. The CloudFormation template also creates a Lambda function to do automatic rotation of the password for the MySQL RDS database every 30 days. Native credential rotation can improve security posture, as it eliminates the need to manually handle database passwords through the lifecycle process.

Below is a reference CloudFormation code that covers these details (please make sure to use your version of the CloudFormation template as described in the sample code readme file):


#This is a Secret resource with a randomly generated password in its SecretString JSON.
MyRDSInstanceRotationSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
    Description: 'This is my rds instance secret'
    GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${!Ref RDSUserName}"}'
        GenerateStringKey: 'password'
        PasswordLength: 16
        ExcludeCharacters: '"@/\'
    Tags:
    -
        Key: AppNam
        Value: MyApp

#This is a RDS instance resource. Its master username and password use dynamic references to resolve values from
#SecretsManager. The dynamic reference guarantees that CloudFormation will not log or persist the resolved value
#We use a ref to the Secret resource logical id in order to construct the dynamic reference, since the Secret name is being
#generated by CloudFormation
MyDBInstance2:
    Type: AWS::RDS::DBInstance
    Properties:
    AllocatedStorage: 20
    DBInstanceClass: db.t2.micro
    DBName: !Ref RDSDBName
    Engine: mysql
    MasterUsername: !Ref RDSUserName
    MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', !Ref MyRDSInstanceRotationSecret, ':SecretString:password}}' ]]
    MultiAZ: False
    PubliclyAccessible: False      
    StorageType: gp2
    DBSubnetGroupName: !Ref myDBSubnetGroup
    VPCSecurityGroups:
    - !Ref RDSSecurityGroup
    BackupRetentionPeriod: 0
    DBInstanceIdentifier: 'rotation-instance'

#This is a SecretTargetAttachment resource which updates the referenced Secret resource with properties about
#the referenced RDS instance
SecretRDSInstanceAttachment:
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
    SecretId: !Ref MyRDSInstanceRotationSecret
    TargetId: !Ref MyDBInstance2
    TargetType: AWS::RDS::DBInstance
#This is a RotationSchedule resource. It configures rotation of password for the referenced secret using a rotation lambda
#The first rotation happens at resource creation time, with subsequent rotations scheduled according to the rotation rules
#We explicitly depend on the SecretTargetAttachment resource being created to ensure that the secret contains all the
#information necessary for rotation to succeed
MySecretRotationSchedule:
    Type: AWS::SecretsManager::RotationSchedule
    DependsOn: SecretRDSInstanceAttachment
    Properties:
    SecretId: !Ref MyRDSInstanceRotationSecret
    RotationLambdaARN: !GetAtt MyRotationLambda.Arn
    RotationRules:
        AutomaticallyAfterDays: 30

#This is a lambda Function resource. We will use this lambda to rotate secrets
#For details about rotation lambdas, see https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html     https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html
#The below example assumes that the lambda code has been uploaded to a S3 bucket, and that it will rotate a mysql database password
MyRotationLambda:
    Type: AWS::Serverless::Function
    Properties:
    Runtime: python3.9
    Role: !GetAtt MyLambdaExecutionRole.Arn
    Handler: mysql_secret_rotation.lambda_handler
    Description: 'This is a lambda to rotate MySql user passwd'
    FunctionName: 'cfn-rotation-lambda'
    CodeUri: 's3://devsecopsblog/code.zip'      
    Environment:
        Variables:
        SECRETS_MANAGER_ENDPOINT: !Sub 'https://secretsmanager.${AWS::Region}.amazonaws.com'

Verifying the solution

To be certain that everything is set up properly, you can look at the Lambda code that’s querying the database table by following the below steps:

Go to the AWS Lambda service page
From the list of Lambda functions, click on the function with the name scm2-LambdaRDSTest-…

You can see the environment variables at the bottom of the Lambda Configuration details screen. Notice that there should be no database password supplied as part of these environment variables:

Figure 9. Environment variables


    import sys
    import pymysql
    import boto3
    import botocore
    import json
    import random
    import time
    import os
    from botocore.exceptions import ClientError
    
    # rds settings
    rds_host = os.environ['RDS_HOST']
    name = os.environ['RDS_USERNAME']
    db_name = os.environ['RDS_DB_NAME']
    helperFunctionARN = os.environ['HELPER_FUNCTION_ARN']
    
    secret_name = os.environ['SECRET_NAME']
    my_session = boto3.session.Session()
    region_name = my_session.region_name
    conn = None
    
    # Get the service resource.
    lambdaClient = boto3.client('lambda')
    
    
    def invokeConnCountManager(incrementCounter):
        # return True
        response = lambdaClient.invoke(
            FunctionName=helperFunctionARN,
            InvocationType='RequestResponse',
            Payload='{"incrementCounter":' + str.lower(str(incrementCounter)) + ',"RDBMSName": "Prod_MySQL"}'
        )
        retVal = response['Payload']
        retVal1 = retVal.read()
        return retVal1
    
    
    def openConnection():
        print("In Open connection")
        global conn
        password = "None"
        # Create a Secrets Manager client
        session = boto3.session.Session()
        client = session.client(
            service_name='secretsmanager',
            region_name=region_name
        )
        
        # In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
        # See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
        # We rethrow the exception by default.
        
        try:
            get_secret_value_response = client.get_secret_value(
                SecretId=secret_name
            )
            print(get_secret_value_response)
        except ClientError as e:
            print(e)
            if e.response['Error']['Code'] == 'DecryptionFailureException':
                # Secrets Manager can't decrypt the protected secret text using the provided KMS key.
                # Deal with the exception here, and/or rethrow at your discretion.
                raise e
            elif e.response['Error']['Code'] == 'InternalServiceErrorException':
                # An error occurred on the server side.
                # Deal with the exception here, and/or rethrow at your discretion.
                raise e
            elif e.response['Error']['Code'] == 'InvalidParameterException':
                # You provided an invalid value for a parameter.
                # Deal with the exception here, and/or rethrow at your discretion.
                raise e
            elif e.response['Error']['Code'] == 'InvalidRequestException':
                # You provided a parameter value that is not valid for the current state of the resource.
                # Deal with the exception here, and/or rethrow at your discretion.
                raise e
            elif e.response['Error']['Code'] == 'ResourceNotFoundException':
                # We can't find the resource that you asked for.
                # Deal with the exception here, and/or rethrow at your discretion.
                raise e
        else:
            # Decrypts secret using the associated KMS CMK.
            # Depending on whether the secret is a string or binary, one of these fields will be populated.
            if 'SecretString' in get_secret_value_response:
                secret = get_secret_value_response['SecretString']
                j = json.loads(secret)
                password = j['password']
            else:
                decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary'])
                print("password binary:" + decoded_binary_secret)
                password = decoded_binary_secret.password    
        
        try:
            if(conn is None):
                conn = pymysql.connect(
                    rds_host, user=name, passwd=password, db=db_name, connect_timeout=5)
            elif (not conn.open):
                # print(conn.open)
                conn = pymysql.connect(
                    rds_host, user=name, passwd=password, db=db_name, connect_timeout=5)
    
        except Exception as e:
            print (e)
            print("ERROR: Unexpected error: Could not connect to MySql instance.")
            raise e
    
    
    def lambda_handler(event, context):
        if invokeConnCountManager(True) == "false":
            print ("Not enough Connections available.")
            return False
    
        item_count = 0
        try:
            openConnection()
            # Introducing artificial random delay to mimic actual DB query time. Remove this code for actual use.
            time.sleep(random.randint(1, 3))
            with conn.cursor() as cur:
                cur.execute("select * from Employees")
                for row in cur:
                    item_count += 1
                    print(row)
                    # print(row)
        except Exception as e:
            # Error while opening connection or processing
            print(e)
        finally:
            print("Closing Connection")
            if(conn is not None and conn.open):
                conn.close()
            invokeConnCountManager(False)
    
        content =  "Selected %d items from RDS MySQL table" % (item_count)
        response = {
            "statusCode": 200,
            "body": content,
            "headers": {
                'Content-Type': 'text/html',
            }
        }
        return response

In the AWS Secrets Manager console, you can also look at the new secret that was created from CloudFormation execution by following the below steps:

Go to theAWS Secret Manager service page with appropriate IAM permissions
From the list of secrets, click on the latest secret with the name MyRDSInstanceRotationSecret-…
You will see the secret details and rotation information on the screen, as shown in the following screenshot:

Figure 10. Secret details

Figure 11. Secret rotation details

Conclusion

In this post, we showed you how to manage database secrets using AWS Secrets Manager and how to leverage Secrets Manager’s API to retrieve the secrets into a Lambda execution environment to improve database security and protect sensitive data. Secrets Manager helps you protect access to your applications, services, and IT resources without the upfront investment and ongoing maintenance costs of operating your own secrets management infrastructure. To get started, visit the Secrets Manager console. To learn more, visit Secrets Manager documentation.

If you have feedback about this post, add it to the Comments section below. If you have questions about implementing the example used in this post, open a thread on the Secrets Manager Forum.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

AWS Security Blog