Using Amazon IPAM to enhance AWS Control Tower governance for Networking resources

Many of our customers are scaling their AWS environment to meet the growing needs of their businesses. They are building multi-VPC environments and interconnecting them with on-premises through AWS Site-to-Site VPN connection or AWS Direct Connect. As the number of projects grow customers have also adopted multi-account strategy using AWS Control Tower. This has helped them with better isolation of workloads, environments and improved governance.They utilize the Account Factory to let their teams create accounts, however the networking infrastructure required to interconnect these workloads and environments had to rely on network administrators to perform their own IP address management (IPAM). Network admins typically planned IP addresses using their homegrown or spreadsheet-based tools that were time-consuming to maintain as they required manual updates and also required synchronization across inter-connected networks. Amazon IPAM helps to plan, allocate and monitor IP addresses across multiple accounts and Regions. In such cases, network administrators can benefit from integrating IPAM into Account Factory such that IPAM pools are sub-divided and provisioned for use for each of these accounts from the top level IP space.

This post presents a solution using which you can integrate Amazon IPAM within AWS Control Tower through the use of Lifecycle Events. It presents the architecture view and shows how this solution extends your AWS Control Tower environment with Amazon IPAM to allow teams to access IPAM pools for their workload accounts.

Architecture overview

Figure 1 shows a multi-account structure created by AWS Control Tower:

Figure 1: AWS Control Tower multi-account environment

AWS Control Tower is set up in the management account. Based on the landing zone best practice, a networking account is created to host all the centralized networking resources that support the requirements of the multi-account, multi-VPC architecture.  In this example, the company has several business units for e.g. Consumer Services BU and Marketing BU with their own accounts to host their applications and workloads. They have adopted the multi-account strategy best practice of having a Infrastructure OU and Workloads OU. Given that most companies have different policy requirements for production workloads, some OUs can have nested OUs for non-production (SDLC) and production (prod). The accounts of the BUs are part of these OUs. Production accounts under the Prod OU and Dev/Staging environments and accounts in SDLC OU.

Teams within the Business Units use the Account Factory product to create their own accounts for creating their environment for their projects and initiatives. Each of the Business Units have their own applications in Virtual Private Clouds (VPC) and talk to other applications from other business units over a shared network topology created through AWS Transit Gateway.  The workloads also span multiple Regions. In this architecture, IPAM is setup using this solution described in the blog to achieve the following goals.

• Avoid IP address conflicts across the multi-account environment.
• Ensure that Prod and Non-prod environments have separate IP blocks to allow for better isolation
• Ability to allocate IP resources across the AWS accounts such that interconnecting networks can use summarized prefixes in their security groups and route tables. For e.g. Summarizing all non-prod CIDR blocks allows us to create route table entry to allow routing to only non-prod networks and a security group rule to block access to prod networks.

Amazon IPAM pool is a collection of contiguous IP address ranges or Classless Inter-Domain Routing (CIDR) blocks. IPAM pools enable you to organize your IP addresses according to your routing and security needs. You can have multiple pools within a top-level pool. Within IPAM pools, you allocate IP CIDRs to AWS resources. An allocation is a CIDR assignment from an IPAM pool to another resource or IPAM pool. When you create a VPC and choose an IPAM pool for the VPC’s CIDR, the CIDR is allocated from the CIDR provisioned to the IPAM pool. You can monitor and manage the allocation with IPAM.

The IP pool structure created as part of this solution is shown below. The solution creates IPAM pool hierarchy for multiple AWS Regions within a top-level pool. Each AWS Regional pool has multiple IPAM pools within it, one per environment. These are referred to in this solution as environment level pools. For example, in figure below we have created one pool for non-production environments to support SDLC (Dev,Test, QA) workloads and one pool for production environment.

Figure 2: IPAM Pool Structure for Prod and Non-Prod workloads

The solution shares only the environment level pools with member accounts of AWS Control Tower selectively to ensure only those specific pools can be used by a particular member account. For e.g. when a member account is created in Prod OU only the Prod level environment pools from the specified Regions is shared with the member account.

When this pool structure is shared with member accounts of AWS Control Tower it allows network administrators to configure Transit Gateway route tables using summarized routes to other Regions for reaching corresponding production and non-production environments. Additionally security groups in Production VPCs can specify summary prefixes corresponding to prod networks across accounts, there by reducing the number of rules to be maintained as number of accounts scale.

This is depicted in the figure below:

Figure 3: Multi-region VPC network topology with simplified routing setup

When the solution is deployed, the following components are created as part of the architecture, as shown in Figure 1:

• IPAM instance in the Networking account with a top-level pool and child pools created within IPAM scope.
• Custom CloudFormation resource backed by a Lambda function in the Networking account which creates the IPAM pool hierarchy.
• The Lambda function in the management account that listens for events from AWS Control Tower Lifecycle event “CreateManagedAccount”

How it works

Figure 4 shows the sequence of events when the CloudFormation is deployed, cloud administrator creates a new account in AWS Control Tower and an application developer creates a VPC.

Figure 4: Sequence of Events

IPAM pool creation in the Networking Account

1. When the CloudFormation is deployed in the AWS Control Tower management account it uses Stacksets to deploy resources in the Networking account.
2. This in turn creates a stack and a CloudFormation Custom Resource in the Networking account.
3. The Custom Resource is backed by a Lambda function which creates the IPAM pool structures comprising of the top-level pool, Regional pools for the Regions specified and environment pools for each Region.

A new account is created in AWS Control Tower

When you create an account through Account Factory:

4. The CreateManagedAccount lifecycle event triggers the CTIPAMLifeCycleLambda Lambda function.
5. The Lambda function assumes the AWSControlTowerExecution role in the networking account and invokes AWS Resource Access Manager (RAM)
6. RAM shares only the IPAM pools that matches the allocation tag of the pool with the member account’s OU name.
7. In the member account, you can now create VPCs with the shared IPAM pools applicable for that Region.

Optionally, you can use a Service Control Policy (SCP) to ensure that all member accounts use IPAM pools for address allocation and not allocate CIDR blocks manually. This can be achieved using some of the example SCPs here.

Deploy the solution

Prerequisites

• This solution requires a AWS Control Tower landing zone setup in your account. Refer to this guide to setup your AWS Control Tower.
• In your AWS Control Tower environment, identify the networking account. If it does not exist, create a Networking account which will hold your networking resources. For this solution, Networking account will be setup as a delegated administrator for IPAM. The parent OU of Networking account can be as per your existing structure or as per your choice but consider following multi-account best practices and create it under Infrastructure OU. Make a note of the Networking account id.
• Create an OU structure for your Workload Accounts to represent your environments. Example shown here talk about 2 environments Production and SDLC and Figure 1 shows 2 OUs Prod OU and SDLC OU. Make note of these OU names in your environment.

Deployment Steps

1. In the AWS Control Tower management account, go to the Amazon VPC IP Address Manager console, select settings and click on edit to specify the Networking account id under the delegated administrator account.

The networking account would be delegated as the IPAM administrator.

Figure 5. Delegating the Networking Account as IPAM Administrator

Figure 6. Verify that the Networking Account is delegated as the IPAM Administrator

2. Login to the AWS Control Tower management account, go to the AWS Resource Access Manager console. On the Settings page, select the Enable sharing with AWS Organizations check box.

Figure 7. On the Settings page, the Enable sharing with AWS Organizations check box is selected.

3. Login to the networking account, go to Amazon VPC IP Address Manager console . Select Create IPAM and then Select the check box to Allow Amazon VPC IP Address Manager to replicate data from the member account(s) into the Amazon VPC IP Address Manager delegate account.

Figure 8. Creating IPAM Pool

4. Provide Name, Description and Operating Regions for the Amazon VPC IP Address Manager.

Figure 9. Populating the IPAM Pool details

5. When you create IPAM instance, Amazon automatically does the following:

1. 1. Returns a globally unique resource ID (IpamId) for the IPAM.
   2. Creates a default public scope (PublicDefaultScopeId) and a default private scope (PrivateDefaultScopeId).

6. In the Amazon VPC IP Address Manager console, go to Scopes and make note of the  PrivateDefaultScopeId

Figure 10. Verifying the IPAM Scopes

7. In the AWS Control Tower management account, go to the CloudFormation console, click on Stacks in the navigation pane, click on Create Stack and select “With new resources” and specify this CloudFormation template to deploy a solution that provides AWS Control Tower integration with Amazon VPC IP Address Manager. In the next screen, The CloudFormation stack creation presents with the screen shown below requiring input parameters for the deployment. CloudFormation stacks are deployed using the console as explained in the documentation through console or CLI.

Figure 11. Deploying the CloudFormation Stack in the AWS Control Tower management account.

The input parameters required are explained below

Centralized Networking resources

Lambda Source Repository

Other parameters

8. After the CloudFormation is deployed, login to the networking account and to go the VPC IP Address Manager Console to verify that the top level pool, Regional pools and Environment level pools along with the Allocation tags(Prod OU and SLDC).

Figure 12. Verifying the pools created in the Networking Account after the CloudFormation deployment.

9. Under the top level pool (ipam-pool-0e37f71bc40741216), 2 Regional pools are created in us-west-2 (ipam-pool-0027f2d7cec2a6474) and us-east-1 (ipam-pool-0ae80c2062300cbf8). Under each Region, you will find 2 Environmental pools created.
   Environment pools (ipam-pool-0284d5e9f938354d9 and ipam-pool-0da932e60b3d5deb0) in Region us-west-2 and Environment pools (ipam-pool-069a9bbceeb3b2ba7 and ipam-pool-0c333d93e55f5f922 ) in us-east-1.
10. You can verify the Allocation tag of each Environment Pool. Click on the Environment pool and go to the compliancy tab. Under the Resource tag compliancy section, verify the allocation tag.

Figure 13. Verifying the Allocation tag of each Environment Pool.

Testing the Solution

1. You can test the solution by creating a new member account using AWS Control Tower account factory in one of the OUs. Make sure you create the account under an OU which maps to a particular environment. Select the Organizational unit for the member account : Prod OU or SDLC OU.

Figure 14. Creating new member Account using Control Tower Account Factory.

2. The example here shows an account CTTEST created under Prod OU.

Figure 15. Verify the new member Account is created under AWS Control Tower

3. Once the member account is created, you can login into to the member account. Go to the Amazon VPC IP Address Manager console, and select Pools. This will show you the list of IPAM pools that match the allocation tag with the member account’s OU name.

Figure 16. Verify the IPAM Pools shared with the new member Account in the Amazon VPC IP Address Manager.

4. You can verify the Allocation tag of each shared Environment Pool to check whether it matches the OU name of the member account. Click on the shared Environment pool and go to the compliancy tab. Under the Resource tag compliancy section, verify the allocation tag.
   The allocation tag matches the Organization unit name which in this case is Prod OU.

Figure 17. Verify the Allocation tag of each shared Environment Pool in IPAM Pools shared with the new member Account.

Figure 18. Verify the Allocation tag of each shared Environment Pool in IPAM Pools shared with the new member Account.

5. Additionally, you can go the Resource Access Manager in the member account to confirm the pools shared with the member account. From the resource share, you can verify the Owner of the share which is the AWS Control Tower management account.

Figure 19. Verify the IPAM Pools shared with the new member Account in the Amazon Resource Access Manager.

6. In the member account, Now go the VPC console and try creating VPC in one of the Regions specified in the CloudFormation template. You would now be able to select the IPAM-allocated IPv4 CIDR block.

Figure 20. Creating VPC in the new member account and verifying that the IPAM pools are shared.

Cleanup

Follow these steps to remove the resources deployed by this solution. These steps will remove the VPCs created by this solution.

1. In the member accounts, remove any VPC that are using IPAM assigned IP addresses.
2. In the networking account, de-provision the CIDR blocks provisioned to the IPAM pools
3. In the networking account, delete the CloudFormation stack with name starting with StackSet-CTNetworkingStackSet- that was deployed as part of this solution.
4. In the AWS Control Tower management account, delete the main CloudFormation stack that was deployed for this solution.

Considerations

Keep in mind the following considerations with respect to this solution.

• While the solution shares environment pools from all Regions to the member account, user can only use the environment pool from a particular Region where the VPC is being created.
• The solution presented here covers the scenario of private IP addressing within your environment. This can be modified to support your IPAM pool plan requirements for public IP addressing. Refer to this tutorial to see how you can use your own public IP space with IPAM.
• While this solution implements a IPAM pool hierarchy based on environment, customers can choose to implement their own IPAM hierarchy based on their needs. Refer to other example IPAM pool plans.
• While the solution will allow for creating summarized prefixes in route tables and security groups, as your environment scales you may still have large number of entries to connect your evolving networks. Consider using Managed Prefix lists to simplify route table and security group configurations.
• You are charged hourly for each active IP address that IPAM monitors. An active IP address is defined as an IP address assigned to a resource such as an EC2 instance or an Elastic Network Interface (ENI). Refer to IPAM pricing

Summary

This blog presented an approach to integrating Amazon IPAM with a multi-account environment of AWS Control Tower. As your environment scales and your teams create new accounts, this solution will help you to define only the right set of IPAM pools will be shared with those accounts. This reduces the manual effort required by network administrators to perform pool allocations and manage IP addressing. It allows the development teams to be more agile by taking away the request approval workflow and effort in ensuring unique IP block allocations. This automation helps organizations to extend the AWS Control Tower benefits of governance and business agility to networking and IP address management.

About the authors: