Exploring Architectures with Cisco SD-WAN and AWS Transit Gateway

By Muffadal Quettawala, Partner Solutions Architect at AWS
By Pratik Mankad, Partner Solutions Architect at AWS Marketplace
By Nikolai Pitaev, Technical Marketing Engineer at Cisco Systems

In this post, we will evaluate multiple Cisco SD-WAN (Software Defined Wide Area Network) architectures on Amazon Web Services (AWS), which enable customers to extend the common policy, segmentation, and security of their SD-WAN environments at scale in an operationally efficient manner.

Interconnection of Cisco SD-WAN and AWS Transit Gateway brings the following benefits:

• Automated and easy connectivity provisioning to the most optimal AWS entry point for the customer’s data center, branch, and hub locations.
• Application and data telemetry in and out of AWS for reporting/chargeback.
• Dynamic routing, multiple path selection, and deterministic failover behavior through the use of Cisco’s proprietary Overlay Management Protocol (OMP) and SD-WAN Secure Extensible Network (SEN) policies.
• Regional hubs interconnecting multiple AWS Transit Gateways and AWS regions via SEN.
• Scaling virtual private network (VPN) bandwidth from 1.2Gbps up to 50Gbps using Equal Cost Multi Path (ECMP).

Cisco Systems is an AWS Partner Network (APN) Advanced Technology Partner and a market leader in SD-WAN based on IDC’s 2019 and Gartner reports.

Update from Sept. 15, 2020: The architectures in this post require custom scripting and automation by the customer. While these deployment methodologies are still valid and useful, Cisco vManage now provides native orchestration for these deployments.

To learn more, check out this post: How to Automate and Secure Branch Office Connectivity to AWS with Cisco SD-WAN.

SD-WAN Provides a Secure Cloud Scale Architecture

Cisco SD-WAN provides a secure cloud scale architecture designed to meet the complex needs of modern WANs through three key areas: advanced application optimization, multi-layered security, and cloud integration.

Customers have traditionally been able to extend their branch SD-WAN deployments to the cloud via the Cloud onRamp for IaaS (infrastructure-as-a-service) functionality available via the vManage orchestrator.

This functionality deploys a pair of SD-WAN virtual form-factor routers in a transit SD-WAN virtual private cloud (VPC). The SD-WAN Edge routers (Cisco CSR1000v or vEdge Cloud) terminate the secure SD-WAN tunnel from the branch, and initiate redundant IPSec VPN tunnels that terminate on a virtual private gateway (VGW) in each spoke VPC.

AWS Transit Gateway enables you to easily scale connectivity across thousands of Amazon VPCs, AWS accounts, and on-premises networks to a single gateway. It also gives you a single set of controls that let you connect to a centrally-managed gateway to grow your network easily and quickly.

Figure 1 – AWS Transit Gateway architecture.

Customers can now deploy SD-WAN more natively with AWS networking services, such as AWS Transit Gateway, to combine the benefits of SD-WAN with AWS. To learn more about the benefits of AWS Transit Gateway, see the documentation.

Cisco SD-WAN Branch Connectivity with AWS Transit Gateway

With AWS Transit Gateway, customers no longer need transit VPC architecture where they terminate multiple VPN connections from vEdge cloud virtual routers in SD-WAN VPC to VGW in each spoke VPC.

Instead, spoke VPCs are connected to AWS Transit Gateway using VPC attachments, and customers can:

• Terminate Site-to-Site (S2S) VPN connection from branch to vEdge cloud virtual routers in SD-WAN VPC, and use VPN attachment to connect vEdge cloud virtual routers to AWS Transit Gateway.
• Branch can directly connect to AWS Transit Gateway using VPN attachment (terminate S2S VPN connection from branch directly on AWS Transit Gateway).

In doing so, you can leverage AWS Transit Gateway to connect between SD-WAN branches through the AWS backbone, as well as connect branches with AWS workloads.

Let’s walk through the steps to deploy each topology.

Connecting Branch to AWS Using an SD-WAN VPC as VPN Attachment

Here’s how to terminate S2S VPN connection from branch to vEdge cloud virtual routers in SD-WAN VPC, and use VPN attachment to connect vEdge cloud virtual routers to AWS Transit Gateway.

The example below shows the branch vEdge cloud virtual routers terminating proprietary secure Cisco SD-WAN tunnels from the branch over the internet or via AWS Direct Connect from the data center to a pair of vEdge cloud routers in an SD-WAN VPC on AWS.

The vEdge cloud routers terminate standard IPSec VPN tunnels from SD-WAN routers to AWS Transit Gateway as VPN attachments.

There are two additional application VPCs attached to AWS Transit Gateway. These VPCs will be used to demonstrate connectivity from branch office and/or data center to application on AWS. In this post, we refer to these application VPCs as human resources VPC (HR VPC) and engineering VPC (ENG VPC).

For this example, we want the branch/data center to talk to the HR VPC and ENG VPC workloads, but we don’t want the HR VPC and ENG VPC workloads to talk to each other. This is an example of a segmentation use case.

To achieve this, we have configured three different route tables on the AWS Transit Gateway. Route table1 is associated with the HR VPC; route table2 is associated with the ENG VPC; and route table3 is associated as a VPN attachment with the SD-WAN VPC.

The IP routes in the AWS Transit Gateway route tables associated with the HR VPC and ENG VPC are learned from the SD-WAN VPC using route propagation. The routes in the route table associated with the SD-WAN VPC are propagated statically.

Additionally, a static route is configured on the VPC route table for the HR VPC and ENG VPC with tgw-xxxx as the next hop for non-local routes. Similarly, a static default route is configured on the SD-WAN VPC route table that points to igw-xxxx.

The routes between the SD-WAN VPC and the branch are learned via Cisco’s SD-WAN OMP. By redistributing BGP routes into OMP, the SD-WAN network becomes aware of the AWS infrastructure. Traditional route filtering options like route maps can be used in order to make redistribution more granular.

Let’s examine the pros and cons of this topology:

Pros:

• Sophisticated routing and path selection via secure SD-WAN tunnel from branch/data center to the cloud.
• Telemetry data exchange between the branch and AWS allows the controller to make intelligent decisions for optimized link selection for SD-WAN traffic.

Cons:

• Higher cost, requiring a pair of redundant SD-WAN vEdge routers in each AWS region, and adds licensing and Amazon Elastic Compute Cloud (Amazon EC2) instance consumption cost.
• Complex to manage, as SD-WAN Edge routers must be managed by the end customer.

Figure 2 below shows an architecture diagram of the description above.

Figure 2 – Branch to AWS TGW via SD-WAN VPC as VPN attachment.

Deploying the Reference Architecture

For illustration, only branch-to-AWS connectivity is demonstrated in the example below. The data center-to-AWS via AWS Direct Connect can be configured similarly.

Prerequisites:

• You have an existing Cisco SD-WAN deployment, including SD-WAN controllers (vManage, vSmart and vBond) and SD-WAN Edge routers on-premises.
• You have appropriate licenses for two Cisco SD-WAN Edge virtual routers (vEdge Cloud or CSR1000v).
• You have subscribed to the Cisco SD-WAN vEdge or CSR1000v router through AWS Marketplace.

To get started, follow the steps below in vManage:

• Procure licenses for two Cisco SD-WAN Edge routers using Cisco Smart Licensing. In vManage, you can import the new licenses directly from your Cisco Smart Licensing account.
• Select Configuration > Devices and then Sync Smart Account. Refer to the documentation for more information.
• In vManage under Configuration > Templates, create a basic template for vEdge cloud routers and attach it to appropriate devices. Refer to Cisco SD-WAN documentation for details.
• Make sure you select the Management Interface (VPN512) template.
• Download the cloud-init template for each cloud vEdge from vManage. In Cisco vManage, go to Configuration > Devices, and select an appropriate SD-WAN Edge router and generate cloud-init file. Refer to the Cisco SD-WAN documentation for more details.

Follow these steps to configure the AWS side:

• Encode the downloaded cloud-init for each vEdge cloud router as base64 encoded user data:

base64 -b0 <vedge1-cloud-init.cfg> > vedge1_b64.cfg
base64 -b0 <vedge2-cloud-init.cfg> > vedge2_b64.cfg

• Deploy the vEdge from AWS Marketplace using the vedge.yaml AWS CloudFormation template.
• After ~5 minutes, vManage will show both SD-WAN routers as “reachable” and “active.”
• It’s recommended the software versions for Cisco SD-WAN vManage match the SW versions for Cisco SD-WAN vEdge/CSR1000v. For example, vManage 19.2.0 matches with vEdge 19.2.0 and CSR1000v 16.12.1b.
• Deploy the HR VPC and ENG VPC using the spoke.yaml CloudFormation template. Note the VPC takes 4-5 minutes to complete.
• Deploy the AWS Transit Gateway using the tgw.yaml CloudFormation template. Enter the input parameters for the template. The AWS Transit Gateway template takes 10-12 minutes to complete.

Next, follow these steps in vManage:

• Once SD-WAN Edge virtual routers are deployed, both devices will use basic configuration—provided via the cloud-init file—to successfully join the SD-WAN fabric. vManage will push the basic template, which was defined in the first step.
• In vManage, make sure that under Configuration > Devices the SD-WAN routers have status “In Sync,” the mode is “vManage,” and both are valid.
• In vManage, create/modify templates to add IPSec and BGP connections to AWS Transit Gateway.
• Create a feature template for the service VPN, which connects towards AWS Transit Gateway. Make sure you enable BGP-OMP redistribution.

• Create two IPsec feature templates, as shown below. Use variables for IP addresses for now. During the template push to SD-WAN routers, you will be asked to provide values for all parameters.

• Create a BGP feature template to peer with the AWS Transit Gateway. Note that you need to explicitly call out the required protocols for redistribution; there is no redistribution by default enabled.

• Define two BGP neighbors, as shown below. Use variables for BGP neighbor IP addresses.

• Create a Loopback feature template for management. This is optional, but recommended.
• Use the outputs of the AWS Transit Gateway template to configure the cloud vEdge IPsec tunnels from the SD-WAN VPC. The example below shows two SD-WAN Edge routers selected to use the configuration template.

• You can enter the variable values (Tunnel IP, pre-shared keys, etc.) direct or upload using XLS file.
• For simplicity, the direct input of values for one template is shown below.

• Push the configuration template from vManage to appropriate SD-WAN Edge routers.
• Make sure the template is successfully attached to all relevant devices.

• Now, the tunnels should be configured from both sides. You can verify the state of the VPN tunnel from the AWS console after about 5 minutes.

Follow the steps below in the AWS console to configure the routes:

• Configure the routes in the HR VPC, ENG VPC, and SD-WAN VPC route tables.
• Specify a default route in both the HR VPC and ENG VPC with destination cidr 0.0.0.0/0 and the AWS Transit Gateway Id as the gateway:

aws ec2 create-route –route-table-id <hr_VPC_main_rtb_id> –destination-cidr-block 0.0.0.0/0 –gateway-id <tgw-xxxx>
aws ec2 create-route –route-table-id <eng_VPC_main_rtb_id> –destination-cidr-block 0.0.0.0/0 –gateway-id <tgw-xxxx>

• Specify a default route in the SD-WAN VPC main route table, as follows:

aws ec2 create-route –route-table-id <SDWAN_VPC_main_rtb_id> –destination-cidr-block 0.0.0.0/0 –gateway-id <igw-xxxx>

Connecting Branch to AWS Using an SD-WAN VPC as VPC Attachment

The next architecture we’ll look at is similar to the one detailed above, but with one key difference. The SD-WAN VPC vEdge cloud routers terminate to the AWS Transit Gateway as a VPC attachment, as opposed to VPN attachments.

Note the following configuration modifications when connecting the SD-WAN VPC to AWS Transit Gateway as a VPC attachment:

• In the route table associated with vEdge cloud virtual router’s private subnet that’s used for AWS Transit Gateway, create a default route with 0.0.0.0/0 as the destination and vEdge cloud virtual router’s private subnet elastic network interface (ENI) – eni-xxx as the target.
• In the route table associated with vEdge cloud virtual router’s public subnet, create routes with ENG VPC and HR VPC prefix as the destination and AWS Transit Gateway – tgw-xxx as the target.

Figure 3 – Branch to AWS TGW via SD-WAN VPC as a VPC attachment.

Let’s examine the pros and cons of this topology compared with the topology we examined earlier in the post.

Pros:

• No need to manage IPSec VPN tunnels from SD-WAN VPC to AWS Transit Gateway, thereby reducing management overhead.
• Higher single connection bandwidth. While AWS Transit Gateway can support higher aggregated bandwidth (tested up to 50 Gbps) using ECMP and multiple IPSec VPN tunnels, per tunnel bandwidth is still limited to 1.25 Gbps. If this is a limiting factor, terminating SD-WAN VPC to AWS Transit Gateway as a VPC attachment eliminates this limitation.
• Saves the cost associated with AWS S2S VPN connections.

Cons:

• Loss of dynamic routing support via BGP. Routes in AWS Transit Gateway HR VPC and ENG VPC route tables will need to be statically defined.
• The connection between the SD-WAN VPC and AWS Transit Gateway is unencrypted.
• Failover scenario: If vEdge cloud virtual router is replaced, the route table associated with the private subnet used for AWS Transit Gateway needs to be reprogrammed to used alternate vEdge cloud virtual router’s ENI instead. You can do this manually or automate it, but automation can take minutes which may not be ideal.

Connecting Branch to AWS via Direct AWS S2S VPN Connection

In this deployment model, the branch SD-WAN Edge routers terminate standard IPSEC VPN tunnels from the branch to AWS Transit Gateway as VPN attachments.

There are two additional VPC attachments to the AWS Transit Gateway, represented as HR VPC and ENG VPC.

Similar to the previous architecture, we want the branch/data center be able to reach the HR VPC and ENG VPC workloads, but we don’t want the HR VPC and ENG VPC workloads to talk to each other.

As before, we have configured three route tables on the AWS Transit Gateway. Route table1 is associated with the HR VPC; Route table2 is associated with the ENG VPC; and route table3, in this case, is associated with the branch.

The routes in the route tables associated with HR VPC and ENG VPC are learned from the SD-WAN VPC using route propagation. The routes in the route table associated with the branch are propagated statically.

Additionally, a static route is configured on the VPC route table for the HR VPC and ENG VPC with tgw-xxxx as the next hop for non-local routes. The routes from the AWS Transit Gateway to the branch are learnt via BGP, as well.

Figure 4 below shows an architecture diagram of the description above.

Figure 4 – Branch to AWS TGW without SD-WAN VPC.

Let’s examine the pros and cons of this topology.

Pros:

• Low cost: No need to buy licenses or pay for Amazon EC2 instance consumption cost for SD-WAN routers in the cloud.
• Easy to manage: There’s no need to manage and scale SD-WAN routers in the cloud.
• Native: It leverages AWS Transit Gateway by terminating either a VPN attachment or transit virtual interface (VIF) directly to AWS Transit Gateway.

Cons:

• Limited SD-WAN features: Given the SD-WAN tunnel is not terminated in the cloud, some key features like application optimization and telemetry exchange from the cloud is lost. As a result, routing decisions by the SD-WAN controller are limited to availability of the cloud link, as opposed to quality of the link.
• Large number of IPSec tunnels to manage between branch routers and AWS Transit Gateway.

Service-Insertion with Cisco SD-WAN and AWS Transit Gateway

SD-WAN traffic through the AWS network can be routed via services such as firewalls, IDS/IPS, and load balancers. The architectures described above can be adapted to extend this functionality.

For example, you can provision a service, such as a firewall, in the same or different VPC as the SD-WAN VPC. The route tables associations and propagations on the AWS Transit Gateway can be configured to route all SD-WAN traffic through this firewall service.

A detailed step-by-step architecture for this scenario is out of scope for this post, but you can refer to the following links for more information:

• Service insertion with Cisco SD-WAN
• Service insertion configuration examples
• Service insertion with AWS TGW

Summary

This post describes how to extend connectivity from branch office and/or data center to applications running on AWS using Cisco SD-WAN and AWS Transit Gateway.

Multiple deployment architecture options are explored including:

• Connecting branch to AWS using an SD-WAN VPC as VPN attachment to AWS Transit Gateway.
• Connecting branch to AWS using an SD-WAN VPC as VPC attachment to AWS Transit Gateway.
• Connecting branch to AWS via Direct AWS S2S VPN connection to AWS Transit Gateway.

The pros and cons of each architecture are described so you can chose an architecture that best suits your needs.

.

.

Cisco Systems – AWS Partner Spotlight

Cisco is an APN Advanced Technology Partner. They provides a range of products for transporting data, voice, and video within buildings, across campuses, and around the world.

Contact Cisco | Solution Overview | AWS Marketplace

*Already worked with Cisco? Rate the Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.