Access Amazon RDS across AWS accounts using AWS PrivateLink, Network Load Balancer, and Amazon RDS Proxy

Amazon RDS Proxy is a fully managed, highly available database proxy for Amazon Relational Database Service (Amazon RDS) and Amazon Aurora that makes applications more scalable, secure, and resilient to database failures. With RDS Proxy, you can handle unpredictable surges in database traffic that might otherwise cause issues due to using all available connections or creating new connections at a fast rate. One of the main benefits of RDS Proxy is that it can improve application recovery time while efficiently and automatically handling database failovers, which is valid with both planned and unplanned failovers. For more details, see Improving application availability with Amazon RDS Proxy. For common use cases where RDS Proxy is beneficial, refer to Amazon RDS Proxy FAQs.

In this post, we show how to securely and efficiently connect applications in different AWS accounts to an RDS database instance or Aurora cluster using RDS Proxy, AWS PrivateLink, and Network Load Balancer.

Solution overview

The following diagram illustrates the high-level architecture of the solution:

In this solution, we make use of PrivateLink, NLB, and RDS Proxy for cross-account access to the RDS instance in the database account. The high-level steps to set up this architecture are as follows:

1. Create an Aurora cluster or RDS database and associate RDS Proxy with its corresponding endpoint.
2. Use a Lambda function to get the corresponding IP addresses from the RDS Proxy endpoint and register them in the NLB target group.
3. Create a VPC endpoint service using PrivateLink to enable cross-account access.

We provide an AWS CloudFormation template to implement these steps, along with the deployment steps of the required networking infrastructure in the database account.

As part of this template, the Lambda function runs automatically when you deploy the CloudFormation template for the first time, or if there are any changes in the template. We discuss this specific implementation later in this post.

Prerequisites

To represent the cross-account environment, we use two AWS accounts:

• Database account – The account with RDS Proxy, NLB, and PrivateLink
• Application account – The account to test the cross-account access to the database account

You can use the provided CloudFormation template in the following section to set up the infrastructure for the database account. For information on the availability of the resources deployed, refer to Service endpoints and quotas.

For this post, we use Amazon Aurora PostgreSQL-Compatible Edition. RDS Proxy also supports Amazon Aurora MySQL-Compatible Edition, Amazon RDS for MySQL, Amazon RDS for PostgreSQL, and Amazon RDS for MariaDB. This means that this solution is suitable for MySQL and PostgreSQL engines on Amazon RDS or Aurora. For more information about supported versions, refer to the specific versions of Aurora supported by RDS Proxy or the versions of Amazon RDS for MySQL and PostgreSQL supported by RDS Proxy.

For more information on how to create an AWS account, refer to How do I create and activate a new AWS account.

Set up the infrastructure using AWS CloudFormation

You can deploy the infrastructure using the following CloudFormation template. This template deploys the following resources needed for this solution in the database account:

• Networking resources – A VPC with two private subnets and security groups
• Database resources – An Aurora database cluster with two instances (reader and writer), RDS Proxy, and integration with AWS Secrets Manager to manage the database credentials
• VPC endpoints – An Elastic Load Balancing interface endpoint and Amazon Simple Storage Service (Amazon S3) gateway endpoint
• NLB – Network Load Balancer with its target group and listener
• Lambda function – A function to register the IP addresses from the RDS Proxy endpoint in the NLB target group
• PrivateLink endpoint – A PrivateLink endpoint service to allow cross-account access from the application account to the database account

After you deploy the template, navigate to the Amazon RDS console and confirm that the RDS database has been created. If you used the default template parameters, the engine of the database will be Amazon Aurora PostgreSQL, and it will have two Multi-AZ instances (reader and writer) located in the private VPC also created in this template.

On the RDS Proxy console, you can observe that it is deployed in the same VPC and subnets as the RDS instances.

Lastly, you can observe that the RDS Proxy target group has been associated with the previous database and the authentication is integrated with Secrets Manager.

Register the RDS Proxy endpoint IP addresses in the NLB target group

For this solution, we enable cross-account access using PrivateLink connected to a Network Load Balancer. Because the NLB target group only accepts IP addresses and the RDS Proxy endpoint has a DNS name associated, we need to obtain the corresponding IP addresses for the endpoint.

We use a Lambda function to obtain these IP addresses (one IP for each subnet where RDS Proxy is located). The high-level steps of this function are:

1. Get the IP addresses from the RDS Proxy endpoint.
2. Register the IP addresses in the NLB target group.

Because the RDS Proxy endpoint name is immutable, this function will be run when the CloudFormation template is first deployed, or if there is any change in the template. For this we use a Lambda-backed custom resource, which ensures the function is invoked whenever the custom resource is created, updated, or deleted.

Navigate to the Load balancers on the Amazon EC2 console and observe that the NLB target group has been successfully created. On the Listeners tab of the NLB created, confirm that the port and protocol are the correct ones for the database engine you selected (for example, TCP protocol and 5432 port for PostgreSQL). Choose the link for the NLB target group under Forward to target group.

On the Target groups page, notice the RDS Proxy endpoint IP addresses are registered as the NLB targets (using the corresponding port for the engine you selected). You can also check the health status of these targets. For more information, refer to Create a target group for your Network Load Balancer and Health checks for your target groups.

Create a PrivateLink endpoint service

The last step to perform in the database account is to create a PrivateLink endpoint service to allow access from the application account. Consider the following when configuring this endpoint:

• Endpoint services require a load balancer that receives requests from service consumers and routes them to your service. In this case, you create the endpoint service using the NLB you previously created. For more information, refer to Create an endpoint service.
• You can allow specific AWS principals to create a VPC interface endpoint to connect to your endpoint service. In this case, the application AWS account is the principal. However, you can narrow down the access to a specific role or user from the application account. For more information, refer to Manage permissions.
• You can configure your endpoint service to accept connection requests automatically or manually. In this case, the endpoint service will accept connections automatically. For more information on how to configure a manual approval, refer to Accept or reject connection requests.

On the Endpoint services page on the Amazon VPC console, confirm that the load balancers for the endpoint service are the correct ones. If you followed the VPC and subnet configuration from the CloudFormation template, the endpoint service will be associated to the load balancers located in two subnets in different Availability Zones.

After verifying the configuration of the endpoint service, you can start testing the cross-account access to the database account. Before moving to the application account, note down the endpoint service name and the database credentials located in Secrets Manager.

Create a VPC endpoint

In the application account, we need to create a VPC endpoint before testing the database connection from the instance.

1. On the Amazon VPC console, choose Endpoints in the navigation pane.
2. Choose Create endpoint.
3. For Endpoint settings, enter a name (for instance, Cross-Account-RDS-Endpoint).
4. For Service category, select Other endpoint services.
5. For Service settings, copy the VPC endpoint service name created in the database account and verify the service.
6. For VPC, choose the VPC you used for the previous instance in the application account. Make sure you choose at least two subnets.
7. For Security group, choose the two security groups previously selected for the instance.
8. Verify your settings and choose Create endpoint.

Wait until the endpoint status changes from Pending to Available.

If you selected a manual approval process when creating the endpoint service in the database account, you need to accept the endpoint connection request from the database account. It can take a few seconds for the VPC endpoint status to become Available.

Take note of the VPC endpoint DNS name, which is the hostname used to establish the connection from the PostgreSQL client in the following section.

Create an application instance

Testing the cross-account connection requires a client configured in the application account. For this scope, it is sufficient to create an environment in AWS Cloud9 in the same AWS Region where you deployed the resources in the database account. Alternatively, the connection can also be tested from an EC2 instance.

You must use the same VPC for the VPC endpoint and instance. Also, modify the security group configuration of the Cloud9 EC2 instance to include the default security group of the selected VPC. To do so, on the Amazon EC2 console, open the corresponding Cloud9 EC2 instance details page and change the security group configuration, as shown in the following screenshot.

Install the PostgreSQL client

You can now configure the PostgreSQL client following the procedure described in this section.

In this post, we describe how to quickly create and configure a PostgreSQL client in a Cloud9 environment, with the intention of demonstrating the cross-account connectivity to the database. In your case, you may want to test it with a client EC2 instance or your application.

For more information, refer to Creating and connecting to a PostgreSQL DB instance.

1. Connect to the Cloud9 environment and update all the installed packages to their latest versions that are available in the configured yum repositories:

$ sudo yum update

Let the yum finish the update of the system, which may require explicit confirmation and some time to complete.

2. Add the PostgreSQL 14 repository to the yum configuration:

$ sudo tee /etc/yum.repos.d/pgdg.repo<<EOF
[pgdg14]
name=PostgreSQL 14 for RHEL/CentOS 7 - x86_64
baseurl=https://download.postgresql.org/pub/repos/yum/14/redhat/rhel-7-x86_64
enabled=1
gpgcheck=0
EOF

3. Run the yum update command again after you add the PostgreSQL 14 repository:

$ sudo yum update
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
pgdg14                                                                               | 3.6 kB  00:00:00     
(1/2): pgdg14/group_gz                                                               |  244 B  00:00:00     
(2/2): pgdg14/primary_db                                                             | 223 kB  00:00:00     
244 packages excluded due to repository priority protections
No packages marked for update

4. Install the PostgreSQL 14 client:

$ sudo yum install postgresql14

5. Check the psql client works as expected:

$ psql -V
psql (PostgreSQL) 14.9

Test the connection from the application instance

When the PostgreSQL client installation is complete, before testing the connection, make sure you have the following:

• The database password – You can retrieve its value from the Secrets Manager console in the database account. The secret name prefix is AuroraSecret-.

• The database user – If you didn’t change the original CloudFormation template, the user will be sampleuser.
• The VPC endpoint DNS name from the application account – You created this in the previous section.

Now you can connect to the Aurora database from the client configured in the application account:

$ psql -p 5432 -d postgres -U sampleuser -h vpce-xxxxxxxxxxxxxxxxxx-yyyyyyyy.vpce-svc-zzzzzzzzzzzzzzzzzzz.eu-west-1.vpce.amazonaws.com
Password for user sampleuser: 
psql (14.9, server 14.6)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> select version();
version  
-------------------------------------------------------------------------------------------------------------
PostgreSQL 14.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.5.0, 64-bit
(1 row)

postgres=> select aurora_version();
aurora_version 
----------------
14.6.4
(1 row)

The Aurora cluster running in the database account is configured with a writer and a reader instance. We tested the connection to the writer instance, and were able to create a new table and insert some data:

postgres=> create table blog_table (col1 int, col2 varchar);
CREATE TABLE

postgres=> insert into blog_table values (1, 'Some text');
INSERT 0 1

postgres=> select * from blog_table;
 col1 |   col2
------+-----------
    1 | Some text
(1 row)

Leave the psql client connected to the database to verify that, after the failover, your connection is still active, because the RDS Proxy masks the failures of the underlying Aurora cluster preserving the client connections.

Test a failover event

Let’s now open the RDS console on the database account and select Databases. The writer instance is currently running in the eu-west-1b Availability Zone. Let’s trigger a manual failover and check the cross-account connectivity again. To do so, select the writer instance on the Amazon RDS console and choose Failover on the Actions menu.

In a few moments, the failover is complete and the new Aurora cluster configuration is changed with the writer instance, this time in the eu-west-1a Availability Zone.

Let’s verify that the client connection from psql is still alive:

postgres=> select * from blog_table;
 col1 |   col2    
------+-----------
    1 | Some text
(1 row)

postgres=> insert into blog_table values (2, 'Some additional text');
INSERT 0 1

postgres=> select * from blog_table;
 col1 |         col2         
------+----------------------
    1 | Some text
    2 | Some additional text
(2 rows)

We are still able to read and write our data hosted in the Aurora cluster database in the database account. This demonstrates that RDS Proxy is able to automatically handle the failover event of the Aurora cluster, preserving and redirecting the client connections from the application account to the new writer instance of the Aurora cluster in the database account.

Limitations

This solution works across AWS accounts and VPCs within the same Region. To make this solution work across Regions, use PrivateLink access over VPC peering and refer to Working with Amazon VPC across AWS Regions. Additionally, this solution won’t work when connecting to RDS for PostgreSQL database instances or Aurora PostgreSQL clusters using SSL certificates with sslmode set to verify-full or when connecting to RDS for MySQL database instances or Aurora MySQL clusters using SSL certificates with sslmode set to verify_identity. For more information, refer to Determining whether a client requires certificate verification in order to connect.

Clean up

When you’re finished experimenting with this solution, complete the following steps to clean up your resources:

1. Delete those resources you created in the application account, for instance, the EC2 instance and the VPC interface endpoint, before deleting the CloudFormation stack in the database account.
2. Delete the CloudFormation stack from the database account. For additional information refer to Deleting a stack.

You can’t delete a stack that has termination protection enabled. In case of issues related to the Lambda-backed custom resource during the stack deletion, refer to the following re:Post article for some guidance on how to resolve it.

Pricing

Refer to the following links for pricing details of the main AWS services deployed as part of this solution:

• Amazon Aurora pricing
• Amazon RDS Proxy pricing
• Elastic Load Balancing pricing
• AWS PrivateLink pricing
• AWS Lambda pricing

Alternative solutions

In the scenario described in this post you can also use VPC peering or AWS Transit Gateway to implement the networking between the database and application accounts, but each option has potential drawbacks. First, you may want to limit access to your RDS database instance only rather than the entire VPC where it is hosted. Second, both VPC peering and Transit Gateway are not a possibility in the case of overlapping CIDR blocks.

A possible alternative solution in this scenario is to implement a cross-account connection using PrivateLink and Network Load Balancer (NLB) in the database account, and a VPC endpoint configured in the application account. This approach is described in Access Amazon RDS across VPCs using AWS PrivateLink and Network Load Balancer, where the database failover event is handled with an AWS Lambda function triggered by an Amazon Simple Notification Service (Amazon SNS) topic and an RDS event subscription. This mechanism is needed because Amazon RDS automatically changes the Domain Name System (DNS) record of the writer instance in the Amazon Route 53 configuration in case of a failover. Therefore, the NLB target group configuration needs to be updated with the new IP of the primary database instance and the failover time depends on DNS TTL and TCP keepalive parameters settings.

A second alternative solution, described in Use Amazon RDS Proxy to provide access to RDS databases across AWS accounts, uses RDS Proxy and AWS Resource Access Manager (AWS RAM). With this approach, you create a new endpoint for the proxy using the subnet resource shared with the account where the RDS instance or Aurora cluster resides. The proxy resides in the database account, and the VPC endpoint resides in the application account, along with the other resources, such as the Amazon Elastic Compute Cloud (Amazon EC2) instances, which host the application required to connect to the database. One advantage of this solution is that RDS Proxy uses its internal mechanism to detect the failover and redirect the client request to the new writer instance, bypassing the DNS TTL mechanism. This results in a faster failover handling. A possible limitation of this approach is that the resource sharing through AWS RAM might not be permitted in all organizations.

In this post, we offered a different solution for those cases where the previously mentioned alternatives are not suitable, offering a robust and secure architecture to enable cross-account and cross-VPC access of an RDS database instance.

Conclusion

RDS Proxy is a highly available database proxy service that makes your applications more scalable and resilient to database failover, and doesn’t require any additional mechanisms for handling RDS failover.

In this post, you learned how to implement a robust and secure custom solution to connect to an RDS database instance or an Aurora cluster hosted in a different AWS account or VPC. For this, we used RDS Proxy, PrivateLink, and NLB, which avoided the implementation of complex mechanisms to handle the database failover events.

Leave your feedback in the comments section to further improve this post.

About the Authors

Domenico di Salvia is a Senior Database Specialist Solutions Architect at AWS. In his role, Domenico works with customers in the EMEA region to provide guidance and technical assistance on database projects, helping them improve the value of their solutions when using or migrating to AWS, designing scalable, secure, performant, sustainable, cost-effective, and robust database architectures in the AWS Cloud.

Itziar Molina Fernandez is an AI/ML Consultant in the AWS Professional Services team, where she helps customers build their data and machine learning platforms.