Simplify global hybrid connectivity with AWS Cloud WAN and AWS Direct Connect integration

In this post, we review how you can build hybrid connectivity architectures using the AWS Cloud WAN built-in support for AWS Direct Connect attachments. We share best practices and considerations for designing global hybrid networks on AWS that help you enable seamless connectivity between your on-premises environments and the AWS Cloud.

Now, AWS Cloud WAN offers built-in support for AWS Direct Connect gateway attachments. This integration provides you with greater flexibility in configuring global hybrid networks, without the need to use intermediary AWS Transit Gateways. It helps you simplify routing management by enabling AWS Direct Connect gateways to advertise routes directly between your AWS Cloud WAN segments and on-premises environments. You can attach AWS Direct Connect Gateways to your core network, and achieve routing behaviors specific to AWS Regions and Cloud WAN segments.

In the first section of this post, we review how this new feature works, along with three common scenarios for greenfield AWS Direct Connect integration with your AWS Cloud WAN global network. Then, we review the recommended migration steps for customers currently using AWS Cloud WAN with AWS Direct Connect through an AWS Transit Gateway.

Prerequisites

We assume that you are familiar with networking constructs on AWS, including Amazon Virtual Private Cloud (Amazon VPC), AWS Direct Connect, AWS Transit Gateway, and AWS Cloud WAN. We won’t focus on defining these services, but we do outline their capabilities and how you can use them for the highlighted routing scenarios. For additional background on the migration best practices described in this post, we recommend to review the previous post Advanced hybrid routing scenarios with AWS Cloud WAN and AWS Direct Connect, which highlights the hybrid routing architectures that include Transit Gateways. We do not focus in the post on AWS Cloud WAN segmentation strategies, or how to create and manage core network segments. We recommend to review the documentation for details on AWS Cloud WAN components.

Baseline Architecture

For simplicity, we use a baseline architecture (Figure 1) where AWS Cloud resources are deployed across three AWS Regions: ap-southeast-2, us-west-2, and us-east-1. In each AWS Region of the core network, AWS Cloud WAN deploys and manages a core network edge (CNE). We configured three segments for workloads: Production, Development, and Hybrid. The architectures in this post apply to environments with different number of segments, attachment types, and number of AWS Regions. We assume you created the AWS Cloud WAN core network as shown, in your testing environment. In the documentation, you can find a policy example for a core network with segments similar to the architecture in Figure 1.

Figure 1: Baseline architecture showing an AWS Cloud WAN core network across three AWS Regions

We’ll review three scenarios for configuring AWS Cloud WAN Direct Connect attachments. For simplicity, in each scenario we use a separate segment on the Cloud WAN core network:

• In the first two scenarios, we attach two different AWS Direct Connect gateways to the Development and Production segments respectively, and achieve end-to-end segmentation between AWS and on-premises environments.
• For the third scenario, we use the Hybrid segment and configure global routing segmentation using two new Direct Connect gateways. In this example, the Hybrid segment is dedicated for partner connectivity from on-premises, and does not communicate directly with Production and Development segments.

How it works

You can attach one or more Direct Connect gateways to your AWS Cloud WAN core network, across all or a subset of your core network Regions. When you associate a Direct Connect gateway attachment to only a subset of your core network Regions, the Direct Connect gateway receives only the routes that are local to those associated Regions. With this integration, your on-premises routers now receive the full BGP AS_PATH attribute for routes advertised by the core network through the Direct Connect gateway. This improves the visibility across your global network routing, and helps you make granular routing decisions in your on-premises environment.

Let’s review the step-by-step configuration of the Cloud WAN Direct Connect integration. Starting from the baseline architecture, we first create a new Direct Connect gateway, and attach it to the Development segment, across all three Cloud WAN Regions.

Step 1: Navigate to the Direct Connect Console, select Direct Connect gateways, and click Create. Configure a Name and an Autonomous System Number (ASN) for the Direct Connect gateway, as shown in Figure 2. The Direct Connect gateway ASN must be unique, and must not overlap with the ASN range of your core network, or with your on-premises environments. Click Create Direct Connect gateway.

Figure 2: Create Direct Connect gateway Console

Step 2: Navigate to the AWS Network Manager Console and select your Global Network. Navigate to your Core Network and select Attachments. Click on Create Attachment. For the attachment settings configure:

• (Optional) Name: Choose a name for your attachment.
• Attachment type: Direct Connect gateway.
• Edge locations: Select all or a subset of AWS Regions of your Core Network for the attachment. In this example, we select all AWS Regions where the core network is configured. We share best practices for choosing the AWS Regions in the following “Things to know” section of this post.
• Direct Connect gateway: Select the Direct Connect gateway created in Step 1.
• Tags: Set the right Key and Value pair for the tag, matching your Cloud WAN attachment policy.

Figure 3: Settings for the Direct Connect gateway attachment to Cloud WAN Development segment

If your Cloud WAN policy specifies require-attachment-acceptance: true for your segment, proceed to accept the attachment, and verify the correct segment association. Once the attachment state becomes Available, you can verify the routes received from the Direct Connect gateway in the segment route tables, in each AWS Region. On your on-premises routers, you will automatically receive the routes from the Cloud WAN segment that are local to the specific AWS Regions you selected.

In this post we do not show the Direct Connect Virtual interfaces configuration steps. For details, refer to the AWS Direct Connect documentation.

Scenario 1: A single on-premises location connected to the AWS Cloud WAN global network Development segment

Figure 4 shows the architecture after the configuration steps in the preceding “How it works” section. We attached the Direct Connect gateway to the Development segment, and associated it with all three CNEs. The sample architecture relies on two Direct Connect transit virtual interfaces (VIFs), distributed across two physical Direct Connect connections. We configured a primary transit VIF (VIF-1) on one of the physical connections, and a secondary transit VIF (VIF-2) on the second physical connection. We also configured Direct Connect BGP community tags to achieve an active/passive Direct Connect setup. For simplicity, we do not use a highly resilient architecture for AWS Direct Connect. We recommend to review the details on creating active/passive BGP connections over AWS Direct Connect, and the AWS Direct Connect Resiliency Recommendations.

Figure 4: Scenario 1 – Single on-premises location with two AWS Direct Connect connections, and two transit Virtual interfaces

1. Route advertisements from on-premises to AWS

Figure 5 shows the BGP routes advertised from on-premises to AWS:

Figure 5: Scenario 1 – On-premises to AWS route propagation

• (A), (B) – On-premises routers advertise IPv4 and IPv6 prefixes to the Direct Connect gateway on both transit VIFs. We use BGP community tags to influence the BGP local preference for the Direct Connect gateway, and achieve active/passive routing on the two VIFs.
• (C) – The Direct Connect gateway updates the AS_PATH for all routes with its own ASN, and advertises all prefixes to the associated CNEs. The Direct Connect gateway maintains BGP communities unchanged. However, these BGP communities are not relevant to AWS Cloud WAN. Each CNE installs the best path in its route table for the Development segment, with next hop Direct Connect gateway Development.
• (D) – Each CNE adds its ASN in the AS_PATH for the IPv4 and IPv6 routes received from the Direct Connect gateway, and advertises them to all other CNEs in remote Regions. CNEs do not prefer on-premises routes from other CNEs, since these have longer AS_PATH attributes compared to the routes received directly from the Direct Connect gateway.

2. Route advertisement from AWS to on-premises

Figure 6 shows the routes propagated from Cloud WAN to on-premises through the Direct Connect gateway. With the built-in Direct Connect attachment to AWS Cloud WAN, routes are dynamically propagated in both directions, without requiring allowed prefixes list configurations.

Figure 6: Scenario 1 – Route propagation from AWS to on-premises

• (A) – All VPC prefixes (IPv4 and IPv6) are automatically propagated to the CNEs in the corresponding segment, based on the attachment policy configuration. You can have other attachment types on your core network, such as Connect and AWS Site-to-Site-VPN.
• (B) – CNEs advertise all local routes in a segment to each other, and to the associated Direct Connect gateway.
• (C) – The Direct Connect gateway adds its own ASN to the AS_PATH, and advertises all prefixes to all associated transit VIFs. On your on-premises routers, you must configure BGP attributes to achieve the active/passive routing behavior for the two transit VIFs (VIF-1 and VIF-2).

3. Traffic flows

Traffic flows between VPCs and on-premises are shown in Figure 7:

Figure 7: Scenario 1 – Traffic flows between AWS and on-premises

• AWS to on-premises traffic flows: VPCs forward traffic to the CNE in their local Region. CNEs send traffic to the local Direct Connect gateway attachment. If the Direct Connect gateway is not associated with all Regions, the CNE will randomly choose a remote CNE to forward the traffic. Thus, we recommend you associate the Direct Connect gateway with all Cloud WAN Regions where the attached segment is present, to ensure optimal routing. The Direct Connect gateway forwards the traffic to the primary transit VIF-1.
• On-premises to AWS traffic flows: Traffic is forwarded symmetrically, following the same path from on-premises to AWS.

Scenario 2: Multiple on-premises locations advertising unique prefixes to the AWS Cloud WAN core network Production segment

In this scenario, we assume multiple on-premises locations are connected to the Cloud WAN core network Production segment, each advertising a unique set of IP prefixes to AWS. Each location is addressed with a distinct set of IPv4 and IPv6 prefixes, so we need a single Direct Connect gateway for all transit VIFs and Cloud WAN core network edges. The Direct Connect gateway determines the correct on-premises location for traffic forwarding from the associated Regions, based on the unique IPv4 and IPv6 prefix advertisements. For simplicity, we do not use a highly resilient architecture for Direct Connect and recommend you review the AWS Direct Connect Resiliency Recommendations.

The architecture is shown in Figure 8. The setup can be extended to more than two globally distributed on-premises locations.

Figure 8: Scenario 2 – Multiple on-premises locations, each advertising unique IPv4 and IPv6 prefixes to the core network Production segment

1. Route advertisements from on-premises to AWS

Figure 9 shows the IPv4 and IPv6 prefixes advertised from the on-premises locations to the Production segment. We configured a Direct Connect attachment to the Direct Connect gateway Production, and associated all CNEs to it. This allows all Cloud WAN CNEs in the core network to learn the BGP routes advertised from on-premises locations directly through the Direct Connect gateway.

Figure 9: Scenario 2 – BGP route advertisements from on-premises to AWS

We use BGP communities on the route advertisements from on-premises to influence the Direct Connect gateway primary and backup paths for each prefix. Routes are propagated from the Direct Connect gateway to AWS Cloud WAN following the same route propagation mechanism we explained in Scenario 1.

2. Route advertisement from AWS to on-premises.

The routes from AWS to on-premises are advertised the same as in the preceding Scenario 1. All on-premises locations learn the AWS Cloud WAN prefixes from the segment that has been associated with the Direct Connect gateway in all the core network Regions.

3. Traffic flows

Figure 10 shows the traffic flows between the VPCs and the specific on-premises locations:

Figure 10: Scenario 2 – Traffic flows between AWS and on-premises

• AWS to on-premises traffic flows: Similar to Scenario 1, Figure 10 illustrates traffic flowing from the VPCs to the CNE in the local region, then to the Direct Connect gateway via the local Direct Connect attachment. The Direct Connect gateway makes the final routing decision, based on the specific advertised routes and BGP communities. In this case, it sends traffic to the correct on-premises location through the primary transit VIFs (VIF-1 or VIF-3, based on the packet’s destination IP address).
• On-premises to AWS traffic flows: Traffic is forwarded symmetrically, following the same paths (VIF-1 or VIF-3 > Direct Connect gateway Production > CNE > VPC).

Scenario 3: Multiple on-premises locations advertising the same prefixes to the AWS Cloud WAN core network Hybrid segment

We recommend advertising specific prefixes from each on-premises location, as shown in the preceding Scenario 2. However, if you need to advertise the same summary prefix from multiple on-premises locations to AWS, you can influence the Cloud WAN routing decision using multiple Direct Connect gateways. This allows you to control which on-premises location the Direct Connect gateways choose as an egress point for the summary prefixes.

In this scenario, we use two Direct Connect gateways to ensure that traffic is routed from AWS to on-premises destinations using the geographically nearest AWS Direct Connect Point of Presence (POP). We attach the two Direct Connect gateways to the Hybrid segment, and associate each with the CNEs that need to use the specific routing behavior. Figure 11 shows the architecture for this scenario.

Figure 11: Scenario 3 – Multiple Direct Connect gateways attached to the Cloud WAN Hybrid segment, each associated with a subset of CNEs

1. Route advertisements from on-premises to AWS

Figure 12 shows route advertisements from on-premises to AWS through the two Direct Connect gateways. Both Direct Connect gateways are learning the same summary prefixes from on-premises, and advertise them to their associated CNEs. Each CNE has the routes learned from the associated Direct Connect gateway, and from the other CNEs in the Core Network with a longer AS_PATH.

Figure 12: Scenario 3 – Route advertisements from on-premises to AWS

2. Route advertisements from AWS to on-premises

A Direct Connect gateway attached to a core network segment advertises to on-premises only the routes its associated CNEs learn from their local attachments in the respective segment. In this scenario:

• Direct Connect gateway 1 advertises to its associated transit VIFs only the IPv4 and IPv6 prefixes from the VPCs attached to the core network Hybrid segment in the ap-southeast-2 and us-west-2 core network Regions.
• Direct Connect gateway 2 advertises to its associated transit VIFs only the IPv4 and IPv6 prefixes from the VPCs attached to the core network Hybrid segment in us-east-1 core network Region.

3. Traffic flows

• AWS to on-premises traffic flows: In this scenario, we associated the CNEs geographically closer to the US West on-premises location with Direct Connect gateway 1. For the CNE closer to the US East on-premises location, we configured Direct Connect gateway 2. This allows us to control which on-premises location is used for the summary prefix. CNEs select the route with the shortest AS_PATH through their directly associated Direct Connect gateway. Thus, the CNEs forward traffic to their associated Direct Connect gateway. The Direct Connect gateway sends traffic to the appropriate transit VIFs based on the configured BGP community tags. When traffic reaches either on-premises location, it can be routed between locations over your on-premises network.
• On-premises to AWS traffic flows: On your on-premises routers you receive the routes from the core network segment that are local to the Regions associated with each Direct Connect gateway. On transit VIFs 1 and 2, you receive the routes from the Hybrid segment that are local to ap-southeast-2 and us-west-2. On transit VIFs 3 and 4 you receive the routes from the Hybrid segment that are local to us-east-1. You can influence the BGP best path selection algorithm on your on-premises routers to achieve symmetrical routing for traffic to AWS on the primary transit VIFs 1 and 3.

Migration best practices

In the second part of this post, we outline migration strategies to minimize downtime when transitioning from architectures using AWS Cloud WAN with AWS Direct Connect via a Transit Gateway. If you want to migrate to AWS Cloud WAN, but your AWS resources (e.g., VPCs) are still attached to Transit Gateways, we recommend following the guidance in the AWS Cloud WAN and AWS Transit Gateway migration and interoperability patterns post. This will help transition your AWS resources from Transit Gateway to AWS Cloud WAN, while maintaining Transit Gateway for hybrid connectivity via Direct Connect. Then, you can proceed with the following steps to migrate Direct Connect from Transit Gateway to AWS Cloud WAN, and decommission Transit Gateways that are no longer needed.

We recommend considering the following migration best practices:

1. Start by creating new Direct Connect gateways to attach to Cloud WAN. Direct Connect gateways are free of charge. Creating new ones allows you to minimize downtime, and simplify migration and rollback procedures.
2. Create new transit VIFs, and associate them with the new Direct Connect gateways. This allows you to control route advertisements on the existing and new paths, without affecting traffic. Verify the new BGP peers are active, but don’t advertise routes from on-premises yet.
3. Attach the new Direct Connect gateways to the respective segments in AWS Cloud WAN, in the respective AWS Regions. During this step, your on-premises routers automatically receive core network routes from the new Direct Connect gateway(s). These new routes have a longer AS_PATH (AS_PATH = Direct Connect gateway ASN, CNE ASN), compared to your existing routes through the old Direct Connect gateway(s) (AS_PATH = Direct Connect gateway ASN). With this new Cloud WAN integration, the Direct Connect gateway maintains all the ASNs in the AS_PATH, instead of overwriting them with its own ASN. No routing path changes should occur in this step, as your on-premises routers prefer the routes received through the old transit VIFs with shorter AS_PATH, instead of the new ones.
4. Verify you are receiving on your on-premises routers the same set of routes on both old and new VIFs. This allows you to verify that your configuration for the new path is correct.
5. Advertise a test route or set of routes from on-premises on the new VIFs, to verify that routes are received on Cloud WAN.
6. Advertise on the new VIFs the same on-premises prefixes as on the old VIFs. Simultaneously, ensure that your on-premises routers prefer routing traffic via the new VIFs. You can achieve this by influencing the BGP best path selection algorithm on your on-premises routers (for example, using Local Preference). This ensures that traffic flows symmetrically between AWS Cloud WAN and your on-premises environments, via the new Direct Connect gateway(s) and transit VIFs. This step leads to routing changes in your environment.

We recommend performing all these steps during a scheduled maintenance window. However, only the last step leads to routing changes in your core network that impact existing traffic flows. To rollback, simply withdraw the routes advertised from on-premises on the new virtual interfaces, and revert the BGP best path selection algorithm to prefer the old VIFs. Cloud WAN will revert to preferring the on-premises routes received from the Transit Gateway(s).

Things to know

• You can associate a Direct Connect gateway only with a single AWS Cloud WAN segment. You can have multiple Direct Connect gateways associated to the same segment. You can also have different Direct Connect gateways associated to multiple Cloud WAN segments. However, you cannot have the same Direct Connect gateway associated to multiple segments.
• When you associate a Direct Connect gateway with an AWS Cloud WAN segment using a Direct Connect attachment, you can choose all or a subset of the Cloud WAN CNEs. We recommend associating the attachment with all CNEs where the respective segment is configured, unless you have bespoke routing requirements, as shown in Scenario 3. If you choose to associate the Direct Connect gateway attachment with only a subset of the CNEs, the Direct Connect gateway receives only the routes from local attachments of the associated CNEs.
• Direct Connect BGP community tags are only relevant to the Direct Connect gateway, and do not influence the Cloud WAN core network routing decisions.
• Cloud WAN route evaluation algorithm is detailed here. Remember that MED is a non-transitive BGP attribute, therefore the Direct Connect gateway resets MED on all routes advertised to Cloud WAN.
• The new Direct Connect attachment on AWS Cloud WAN allows for routes to be propagated dynamically in both directions, without the need of using allowed prefixes list. The use of allowed prefixes list is still required for the Transit Gateway association to a Direct Connect gateway. If you attach a Direct Connect gateway to an isolated segment, VPC routes are not propagated to your on-premises routers, and on-premises routes are not propagated to the core network segment.
• You can attach Direct Connect gateways from multiple accounts to your AWS Cloud WAN core network following the steps described in the documentation.
• The quotas for the number of routes you can advertise from your core network segment to a Direct Connect gateway attachment, and the number of routes you can advertise from your on-premises environments on a Direct Connect virtual interface are detailed in the AWS Cloud WAN documentation and AWS Direct Connect documentation.

Conclusion

In this post, we reviewed how to build hybrid connectivity architectures using AWS Cloud WAN and AWS Direct Connect built-in integration. We shared best practices for designing global hybrid networks on AWS, and enabling seamless connectivity between on-premises environments and the AWS Cloud. This integration provides you with greater flexibility in configuring global hybrid networks, without the need to use intermediary AWS Transit Gateways. It helps you simplify routing management by advertising routes directly between your AWS Cloud WAN segments and on-premises environments. To find out more details about AWS Cloud WAN, check the documentation. If you have questions about this post, start a new thread on AWS re:Post or contact AWS Support.

About the authors