Introduction to Traffic Mirroring to GWLB Endpoints as Target

Network architects need the ability to gain insights into real-time traffic between different resources within their VPCs. Since the announcement of VPC Traffic Mirroring in 2019, the VPC feature has provided this by copying network traffic from elastic networking interfaces (ENIs) on customer’s instances as source, and then sending the traffic to a destination target for monitoring and analysis. Customers utilize Traffic Mirroring to monitor traffic from workloads running across several subnets, VPCs, and accounts in their organization. However, as the number of workloads increase over time, customers desire a simpler way to deploy their monitoring appliances. Customers want a centralized model where the monitoring appliances can be deployed in a scalable and highly-available manner, all without having to worry about routing complexity from the source to the destination targets. In addition, customers want a new deployment model that utilizes a Software as a Service (SaaS) consumption from their partners. This would let them eliminate the operational overhead of deploying, managing, and maintaining the monitoring appliances themselves.

To support these requirements, we’re excited to announce Gateway Load Balancer Endpoint (GWLBe) as an additional supported target for VPC Traffic Mirroring. Previously, customers could utilize a Network Load Balancer (NLB) or an ENI as destination targets to forward mirrored traffic for monitoring and analysis. This new ability to deploy traffic monitoring appliances behind Gateway Load Balancers (GWLB) means that customers can easily centralize their traffic monitoring appliances without the complexity of configuring routing between the spoke VPC and services VPC. Mirrored traffic from an ENI can be forwarded to the GWLB-backed monitoring appliances via GWLBe deployed in customers’ subnets containing their workloads. In addition, traffic monitoring appliances can be deployed in an account hosted by an Independent Software Vendor (ISV) partner-offered SaaS solution. This minimizes the operational overhead for the customer.

In this post, you’ll learn how Traffic Mirroring to a GWLBe target works, explore what the deployment models are, and walk through how to deploy them yourself.

Industry Leading Partners

We’re excited to be working with many of our existing Traffic Mirroring partners at the launch of Traffic Mirroring with GWLB Endpoints as a target.  Before diving into the details of how the solution works, we want to highlight the partners that have integrated with this new feature at launch:

• Corelight – blog post >>
• cPacket – blog post >>
• Gigamon – blog post >>
• NETSCOUT – blog post >>

Understanding Traffic Mirroring to GWLBe

In a previous post, Using VPC Traffic Mirroring to monitor and secure your AWS Infrastructure, we detailed two different deployment models for traffic mirroring. In the first model, customers could deploy appliances in the same VPC as the traffic source in what we call a distributed model. Customers utilize this model when they want networking monitoring tools close to the workload, common ownership of monitoring responsibilities, or unplanned traffic inspection. In the second centralized model, customers deploy a single NLB in a centralized VPC. Then, the customer routes traffic from the spoke VPC to the services VPC using VPC peering or AWS Transit Gateway. Customers utilize this model when organizations have multiple application owners or a central security team responsible for monitoring multiple accounts.

Previously with centralized architectures, VPC peering or Transit Gateways were needed to connect the spoke VPC to the services VPC. The introduction of GWLBe as mirrored destination targets means that customers can deploy GWLB and GWLBe to securely send mirrored traffic across VPC boundaries without needing to provide connectivity between the spoke VPC and services VPC. It combines a transparent network gateway and distributes traffic while scaling the appliances on demand. The GWLBe is a VPC endpoint that provides private connectivity between monitoring appliances deployed behind the GWLB and the mirrored traffic source. The use of GWLBe lets you deploy appliances in customer owned VPCs or in partner owned VPCs as an SaaS offering.

Encapsulation

For traffic to reach the appliances behind the GWLB, there are two encapsulations needed. First, any mirrored traffic is encapsulated in a VXLAN header. This VXLAN header consists of:

• VXLAN ID: customers can configure the VNI, or AWS will randomly assign the value if it’s not configured.
• Source IP address: the primary IP address of the source ENI.
• Destination IP address: the IP address of the GWLBe.

In addition, when traffic passes to a GWLBe, the mirrored traffic is encapsulated with a GENEVE header. The combination of both of these encapsulations is how traffic is received by the appliances, as shown in the following figure:

Figure 1: Encapsulations received by the appliances

Deployment models

Customers can deploy GWLBe in two different endpoint types: customer managed or partner managed.

The following diagram shows a deployment of the GWLB for traffic mirroring utilizing customer managed endpoints. The GWLB is deployed in a centralized Service VPC with multiple appliances as targets. The GWLB is set up for each Availability Zone that the customer wants to monitor traffic, and customers can configure their GWLB with cross-zone loadbalancing as an option to protect against single Availability Zone failures. In the spoke VPCs, GWLBe interfaces are deployed. These endpoints are connected to the GWLB to send traffic from the spoke VPC to the Service VPC. Note that if you’re going cross account between Spoke VPCs and Services VPC, then the Availability Zone IDs must be the same, as defined in the documentation. When the customer must mirror the traffic to the appliances, the traffic mirroring session is created, with the GWLBe as the target for the mirroring source. In this scenario, the GWLBe, GWLB, and the appliances are all deployed, managed, and monitored by the customer.

Figure 2: Customer managed deployment

For customers who are looking for a SaaS offering for the monitoring and analytics of mirrored traffic, we recently began supporting managed GWLBe, such as Cloud NGFW. For partners that support managed GWLBe, they can deploy and manage the GWLBe in a customer account. In the following figure, a partner has deployed a GWLB and appliances in a VPC owned by the partner. Through the use of roles, the customer gives the partner permission to deploy a managed GWLBe in a customer VPC. When the customer must mirror the traffic to the appliances, the traffic mirroring session is created, with the managed GWLbe as the target for the mirroring source. In this way, much of the heavy lifiting of maintaining and deploying the appliances for monitoring and analytics is done by the partner, with the customer consuming the service as an SaaS solution.

Figure 3: Partner managed SaaS deployment

Walkthrough

As discussed above, there are two main deployment models when using GWLBe as a target for VPC Traffic Mirroring. We won’t configure the GWLB itself in this post, as detailed instructions are available in the GWLB product documentation and in Introducing AWS Gateway Load Balancer: Supported architecture patterns. Let’s look at the steps required for deploying the endpoints.

Customer-managed GWLB and GWLBe

For this example, we’re using basic Amazon Linux 2 instances as the target appliance to which to send the traffic. This lets us utilize tcpdump to see mirrored traffic coming into each instance behind the GWLB. After we created our GWLB, and as we would with other GWLB architectures, we must configure it as a VPC Endpoint Service. To do this, we navigate to the VPC console, and select Endpoint Services, then Create endpoint service. Next, we select our previously created GWLB from under the Available load balancers section, and fill in some other basic information as shown in the following figure. For the rest of the information, use the default options shown.

Figure 4: Endpoint Service configuration

Once created, we can create a GWLBe to use as the target for Traffic Mirroring in the VPC that we are monitoring. First, we must copy the Endpoint service name from our newly created Endpoint service. Navigate to the newly created Endpoint service, and copy the Service name from the details page.

Figure 5: Endpoint Service name

To create a GWLBe, we do this by navigating to the Endpoints section of the VPC console. Next, we select ‘Other endpoint services’, and enter the service name (from the above referenced Endpoint Service), VPC, and subnet(s) in which we’d like to deploy the endpoint. Enter the VPC Endpoint service name from the previous step, and select Verify service. If the service name is correct, then you’ll see Service name verified message, as show in the following figure. Depending on the configuration of the service, the endpoint may enter a pending-acceptance state (Figure 8). If so, then the service owner must accept the request before the endpoint can be used as a target for Traffic Mirroring.

Figure 6: Configuring the GWLB VPC Endpoint

Now that we have a GWLB endpoint created, we can use it as the target for a Traffic Mirroring session, as we would an ENI or NLB. Refer to the VPC Traffic Mirroring documentation for detailed configuration steps. An example session using the GWLBe as a target is shown in the figure above. We’ll repeat this process for each VPC that we want to mirror.

Figure 7: Traffic Mirroring Session

Now, the monitoring appliances – or in our case an EC2 instance – behind the GWLb start to see mirrored traffic. We’ve captured some of the traffic using tcpdump to demonstrate:

14:45:26.424479 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 202)
    172.31.77.9.60630 > 172.31.75.172.6081: [udp sum ok] Geneve, Flags [none], vni 0x0, options [class Unknown (0x108) type 0x1 len 12 data b269bd63 a894a2fc, class Unknown (0x108) type 0x2 len 12 data 00000000 00000000, class Unknown (0x108) type 0x3 len 8 data 060955c6]
	IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto UDP (17), length 134)
    172.31.57.124.65527 > 172.31.68.15.4789: [no cksum] VXLAN, flags [I] (0x08), vni 5433587
IP (tos 0x0, ttl 255, id 20264, offset 0, flags [DF], proto ICMP (1), length 84)
    172.31.57.124 > 205.251.242.103: ICMP echo request, id 6054, seq 1, length 64
14:45:26.424565 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 202)
    172.31.77.9.60630 > 172.31.75.172.6081: [udp sum ok] Geneve, Flags [none], vni 0x0, options [class Unknown (0x108) type 0x1 len 12 data b269bd63 a894a2fc, class Unknown (0x108) type 0x2 len 12 data 00000000 00000000, class Unknown (0x108) type 0x3 len 8 data 060955c6]
	IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto UDP (17), length 134)
    172.31.57.124.65527 > 172.31.68.15.4789: [no cksum] VXLAN, flags [I] (0x08), vni 5433587
IP (tos 0x0, ttl 245, id 20264, offset 0, flags [DF], proto ICMP (1), length 84)
    205.251.242.103 > 172.31.57.124: ICMP echo reply, id 6054, seq 1, length 64

Multi-account and/or SaaS deployments

For multi-account or SaaS deployments, the GWLB is owned and managed by the account/SaaS Partner and exposed as an Endpoint service. Then, consumers create endpoints in their own VPC following the above steps using the GWLBe Service name that the other account/SaaS Partner provides. This lets us send traffic mirroring data from other AWS accounts that we own (Figure 2), or to GWLB in AWS accounts owned by SaaS partners via the endpoint. Although the basic process is the same, there are a few things to consider when deploying in this model. You can control which service consumers (either that you own, or for a customer) can create a GWLBe to connect to your service. This is done using the Allow principals tab of the service configuration. An example of this configuration is shown in the following figure.

Figure 8: Allowed principals

By using the service name (as we did earlier), we can subscribe to the service from this additional account by creating a GWLBe. Depending on the service configuration, the endpoint may enter a pending-acceptance state, as shown in the following figure. If so, then the service owner must accept the request before the endpoint can be used as a target for Traffic Mirroring.

Figure 9: Endpoint pending acceptance

Additionally, since the same set of appliances can be used to receive packet data from multiple AWS accounts (customer-owned, or in a partner-managed model), we must consider how we’ll differentiate the packet data. Fortunately, GWLB helps with this by providing the GWLBe ID within the GENEVE packet header, using the Type-Length-Value (TLV) triplet for every packet to the appliance. From our example earlier, we can see that the GWLBe ID is vpce-0ec67233938a5fa67. In the sample tcpdump output shown as follows, we can also see the VPC Endpoint ID listed:

14:45:26.424479 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto UDP (17), length 202)
    172.31.77.9.60630 > 172.31.75.172.6081: [udp sum ok] Geneve, Flags [none], vni 0x0, options [class Unknown (0x108) type 0x1 len 12 data 0ec67233 938a5fa67, class Unknown (0x108) type 0x2 len 12 data 00000000 00000000, class Unknown (0x108) type 0x3 len 8 data 060955c6]

Using this information, we can determine which account or customer the packet came from by taking a look at the Endpoint connections tab of the service configuration.

Figure 10: Endpoint service

Conclusion

In this post, we’ve demonstrated that using the GWLBe as a target for Traffic Mirroring lets you monitor traffic from any VPC and any account in a centralized, scalable, and highly-available manner. We demonstrated how industry leading partners are integrating with this feature to provide SaaS solutions to customers, as well as provided a walkthrough of how you can utilize this in your own environments.

To get started, visit the Traffic Mirroring feature within the VPC service console. You can also review the service documentation and frequently asked questions for further information.