Update: Sep 10, 2024 – Corrected a CloudWatch metric name.

Amazon Web Service (AWS) Gateway Load Balancer (GWLB) is a managed AWS service that allows you to insert third-party firewall appliances into the data path. GWLB helps you deploy, scale, and manage third-party appliances, and it acts as a bump-in-the-wire device and passes traffic transparently to its targets. Customers often deploy third-party firewall appliances as GWLB targets for traffic inspection use cases. Refer to the GWLB documentation for more details.

Today, we are launching the new configurable TCP idle timeout feature on GWLB that allows you to configure the GWLB transmission control protocol (TCP) idle timeout from 60 seconds to 6000 seconds.

In this post, we cover the basics and best practices for using this feature, and how to set it up using AWS Management Console and API.

Prerequisites

In the following sections, we assume that you are familiar with fundamental AWS networking services, such as virtual private clouds (VPCs), subnets, and VPC routing tables. We also assume that you are familiar with setting up third-party firewalls as targets of a GWLB. You can find more information about GWLB architectures in the GWLB posts.

What is the GWLB TCP idle timeout?

GWLB creates a flow entry in its flow table when it sees the first packet of a flow. GWLB uses either a 2-tuple, 3-tuple, or a 5-tuple hash to define a flow and routes all packets of a flow to one of its backend targets. GWLB maintains traffic symmetry by coupling each flow entry to one backend target appliance. This traffic symmetry is necessary for a traffic inspection or firewalling use case.

GWLB considers the flows active as long as packets for that flow pass through GWLB. When packets for a flow stop, the flow is considered idle and the idle timer starts. GWLB removes the flow entry from its flow table after the idle time expires.

Why is a configurable TCP timeout needed for GWLB?

Until now, GWLB supported a fixed TCP idle timer of 350 seconds. Packets arriving at GWLB after the idle timeout expiry are considered as a new flow and can be forwarded to a different target.

However, many firewalls and applications, such as legacy databases, have longer default TCP idle timeouts. This mismatch in timeout values of GWLB and the applications can lead to disruption of traffic flow when long-lived idle flows from these applications are forwarded to different targets by GWLB after the idle time.

The GWLB configurable TCP idle timeout feature allows you to align the TCP idle timeout value of firewalls and applications with GWLB, thus ensuring the continuity of traffic flow and reducing traffic disruptions.

Third-party firewalls, such as Palo Alto Networks, Fortinet, Cisco, and Check Point, support TCP idle timeout of more than 350 seconds, and often provide a default TCP idle timeout of 3600 seconds. This mismatch between the TCP idle timeout values between the GWLB and the target appliances can lead to traffic disruptions. We show the sequence of steps leading to traffic disruption for a sample deployment in Figure-1a, Figure-1b, and Figure-1c. In this scenario, we assume that the backend firewall appliances use their default TCP idle timeout of 60 minutes.

Figure-1a shows the flow tables at the GWLB and the backend firewall appliance when the flow is first established.

Figure-1a: GWLB and firewall flow tables for a new flow

[1] Amazon Elastic Compute Cloud (Amazon EC2) instance deployed in VPC-1 starts a TCP flow and sends data to GWLB endpoint in Availability Zone (AZ)-b (GWLB-EP B).

[2] GWLB-EP B sends the traffic to GWLB that is deployed in the firewall VPC.

[3] GWLB doesn’t find this flow in its flow table and creates a new flow entry. The TCP idle timeout is set to 350 seconds. GWLB selects target T2 for this flow.

[4] GWLB encapsulates the original traffic in GENEVE tunnel and sends this data to T2.

[5] T2 creates a flow entry within its table and sets the TCP idle timeout to 3600 seconds.

The GWLB TCP idle timeout kicks in once the data flow stops. Figure-1b shows the GWLB and backend firewall flow tables 350 seconds after the data flow stops.

[6] The GWLB TCP idle timeout expires for Flow-1. At this point, GWLB removes this flow entry from its flow table.

[7] Backend firewall appliance keeps the flow entry in its table. The idle timer is at 3250 seconds.

Figure-1b shows the sequence of events when the source EC2 instance sends more packets of the same flow.

Figure-1b: GWLB and Firewall Flow tables when the traffic flow becomes active again

[8] After remaining idle for more than 350 seconds, the source EC2 instance sends more packets of the same flow to GWLB-EP B.

[9] GWLB-EP B sends the traffic to GWLB.

[10] GWLB doesn’t find this flow in its flow table, GWLB treats the flow as a net-new flow, and it creates a new flow entry (Flow-2). The idle timeout is set to 350 seconds. This time the GWLB hash algorithm selects target T1 as the backend target for the flow.

[11, 12] T1 sees the packets and drops the packets because it is a TCP flow. This is because stateful firewalls need traffic to be symmetrically routed to the same firewall appliance for TCP flows. In this case, T1 didn’t see the TCP handshake for the flow, and thus rejects this flow. This leads to traffic disruptions.

Step [10] assumes GWLB has cross-zone load balancing enabled. However, you can run into the same issue described in steps [11] and [12] even when cross-zone load balancing is disabled and there are multiple firewall appliances deployed in the same AZ.

How can I use the new configurable TCP idle timeout feature?

You can configure TCP idle timeout for the GWLB IP listener. This configuration applies to both IPv4 and IPv6 traffic.

Step-1: Edit the GWLB listener details

On the GWLB details page, select Actions and then select View listener details, as shown in Figure 2.

Figure 2: GWLB listener details

Step-2: Edit the TCP idle timeout value

Under the GWLB attributes, edit TCP idle timeout detail, as shown in Figure 3.

Figure 3: Edit TCP idle timeout

Enter the desired TCP idle timeout and save changes, as shown in Figure 4.

Figure 4: Enter TCP idle timeout value

You can use the newly launched modify-listener-attributes application program interface (API) call to edit the TCP idle timeout value:

aws elbv2 modify-listener-attributes \
--listener-arn arn:aws:elasticloadbalancing:<Region>:<AWS Account ID>:listener/gwy/<GWLB name>/<GWLB ID>/<Listener ID> \
--attributes \
Key=tcp.idle_timeout.seconds,Value=3700 

 You need to provide the TCP idle timeout value as a key value pair while using the modify-listener-attributes API call, as shown in the following image. Refer to the elbv2 CLI documentation for the full list of supported commands.

Step-3: Verify the new configuration

Figure 5: Verify the GWLB IP listener’s new TCP idle timeout

Step-4: Monitor using Amazon CloudWatch metrics

GWLB now supports new Amazon CloudWatch metrics that allow you to monitor the total number of timed out flows. You can use RejectedFlowCount and RejectedFlowCount_TCP metrics to monitor the total rejected flows and rejected TCP flows respectively, as shown in Figure-6.  These metrics increment only when the GWLB rejects flows because of its flow table getting full. We recommend gradually increasing the GWLB TCP idle timeout.

Figure 6: CloudWatch alarm with RejectedFlowCount metric

Recommendations for configuring the GWLB TCP idle timeout value

Setting the GWLB TCP idle timeout to a very high value means the GWLB keeps flow entries active in the flow table for a longer time, and thus the GWLB flow table gets larger. This introduces the risk of exhausting the GWLB flow table size. To alleviate this risk, we recommend monitoring the ActiveFlowCount CloudWatch metric, which indicates the number of flows available in the GWLB flow table.

Recommendation for applications with long-running flows

In this section, we discuss three recommendations for using the GWLB configurable TCP idle timeout feature to avoid traffic disruptions for an inline traffic inspection use case.

Recommendation-1: You can avoid traffic disruptions by configuring the TCP idle timeout on the GWLB to a value just above the firewall appliance’s TCP idle timeout value. We also recommend configuring the firewall appliance to send a TCP RST when its TCP session times out (check with your firewall vendor for support). For long-running TCP flows that stay idle for long durations, this would make sure that the target firewall appliance is the first device in the chain to timeout a flow. The TCP RST from the backend firewall target makes sure that the client’s TCP session is closed gracefully. This configuration allows you to deploy your inspection architecture without making changes to all of your client devices, or making code changes to your applications.

We recommend using this configuration for legacy applications that need long-running flows, and that cannot implement TCP keepalives within the application code, as well as applications that inherit the default settings of the underlying Operating System (OS), such as Red Hat Enterprise Linux. This option is a pragmatic approach if you have a large number of legacy applications, and upgrading all of your applications/clients/servers would need long maintenance windows.

Recommendation-2: Set the client’s/server’s TCP idle timeout value to be lower than the GWLB or the firewall appliance. This would make sure that either the client or the server gracefully refreshes the connection before the flow expires on the GWLB or on the firewall appliance. With this approach, you must update all of your clients/servers. Refer to the post Implementing long-running TCP connections within VPC networking for details. Although this is an elegant solution, it means that you have to update all of your clients/servers/applications.

Recommendation-3: Implement TCP keepalives within your applications. This would make sure that the clients/servers keep their TCP sessions alive by sending TCP keepalives. This option also requires you to make changes to your applications.

Table-1 shows the summary of these recommendations.

Recommendation for short-lived flows

The total number of active flows is one of the dimensions that determines the total cost of using GWLB. For applications where TCP sessions are expected to be short-lived, you can optimize your GWLB usage by reducing the GWLB TCP idle timeout value. For these use cases, we recommend you set the GWLB TCP idle timeout value just higher than the expected session lifetime.

We commonly see this requirement when connected devices such as police cars, ambulances, and combat vehicles stream data into AWS using multiple cellular networks or Mobile Network Operators (MNOs). The connected device’s session flaps as it switches between different MNOs, and the application code creates a new TCP session upon detecting a flap.

Things to know

• Any existing flows of a GWLB are not impacted when the TCP idle timeout value is changed. Traffic for those flows continue without disruption. The new TCP idle timeout only applies to the GWLB new flow entries.
• Any TCP keepalives refresh the GWLB TCP idle timeout value.
• The idle timeout value applies to both IPv4 and IPv6.
• The RejectedFlowCount_TCP metric is available in CloudWatch only when its value is non zero. In other words, you can start using this metric after the GWLB flow table is full and after the GWLB starts rejecting TCP flows.

Conclusion

In this post, we introduced a new feature that allows you to configure the GWLB TCP idle timeout between 60 seconds to 6000 seconds. We explained how to use this feature using the Console and APIs. In addition, we discussed the best practices for using this feature for long-running and short-lived TCP flows, as well as recommendations for how to monitor specific CloudWatch metrics that allow you to set the right value for your deployment.

To learn more about this feature, visit the documentation.

About the authors