Protecting your Amazon EKS web apps with AWS WAF

Analyze the traffic patterns on any public-facing website or web app, and you’ll notice connection requests from all over the world. Apart from the intended traffic, a typical web application responds to requests from bots, health checks, and various attempts to circumvent security and gain unauthorized access.

In addition to impacting your customer’s experience, these requests can also increase your AWS spend. Today, business-critical web apps rely on web application firewalls to block unwanted traffic and protect apps from common vulnerabilities. But many customers struggle with the complexity when it comes to implementing an effective web application firewall. AWS Web Application Firewall (AWS WAF) and AWS Firewall Manager are designed to make it easy for you to protect your web applications and APIs from common web exploits that can disrupt services, increase resource usage, and put data at risk.

This post describes how to use AWS WAF and AWS Firewall Manager to protect web-based workloads that run in an Amazon Elastic Kubernetes Services (Amazon EKS) cluster.

AWS WAF gives you control over the type of traffic that reaches your web applications. It allows you to monitor HTTP(S) requests to web applications and protect them against DDoS attacks, bots, and common attack patterns, such as SQL injection or cross-site scripting.

If you are unfamiliar with web application firewalls (WAF), you’d be pleased to learn that you don’t have to be a regular expressions wrangler or an expert in firewall rules to use AWS WAF. It comes with a set of AWS-managed rules, so you don’t have to write filtering logic to protect against common application vulnerabilities or unwanted traffic.

AWS WAF integrates with Amazon CloudFront, Application Load Balancer (ALB), Amazon API Gateway, and AWS AppSync. If you already use an ALB as an ingress for your Kubernetes-hosted applications, you can add a web application firewall to your apps within a few minutes.

Customers that operate in multiple AWS accounts can use AWS Organizations and AWS Firewall Manager to control AWS WAF rules in multiple accounts from a single place. AWS Firewall Manager monitors for new resources or accounts created to ensure they comply with a mandatory set of security policies. It is a best practice to run EKS clusters in dedicated VPCs, and Firewall Manager can ensure your WAF rules get applied across accounts, wherever your applications run.

Solution

This post demonstrates how to implement a web application firewall using AWS WAF to protect applications running on EKS. We will start by creating an EKS cluster and deploying a sample workload. The sample application that we will use for this walkthrough is a web-based application that we’ll expose using an Application Load Balancer. We’ll then create a Kubernetes ingress and associate an AWS WAF web access control list (web ACL) with an ALB in front of the ingress.

Prerequisites

You will need the following to complete the tutorial:

• AWS CLI version 2
• eksctl
• kubectl
• Helm
• Git

Create an EKS cluster

Let’s start by setting a few environment variables:

WAF_AWS_REGION=us-west-2 #Change this to match your region 
WAF_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
WAF_EKS_CLUSTER_NAME=waf-eks-sample

Create a cluster using eksctl:

Note: This may take approx. 15 mins to create the cluster.

eksctl create cluster \
 --name $WAF_EKS_CLUSTER_NAME \
 --region $WAF_AWS_REGION \
 --managed

Store the cluster’s VPC ID in an environment variable as we will need it for the next step:

WAF_VPC_ID=$(aws eks describe-cluster \
--name $WAF_EKS_CLUSTER_NAME \
--region $WAF_AWS_REGION \
--query 'cluster.resourcesVpcConfig.vpcId' \
--output text)

Install the AWS Load Balancer Controller

The AWS Load Balancer Controller is a Kubernetes controller that runs in your EKS cluster and handles the configuration of the Network Load Balancers and Application Load Balancers on your behalf. It allows you to configure Load Balancers declaratively in the same manner as you handle the configuration of your application.

Install the AWS Load Balancer Controller by running these commands:

## Associate OIDC provider
eksctl utils associate-iam-oidc-provider \
--cluster $WAF_EKS_CLUSTER_NAME \
--region $WAF_AWS_REGION \
--approve

## Download the IAM policy document
curl -o iam-policy.json https://raw.githubusercontent.com/aws-samples/containers-blog-maelstrom/main/eks-waf-blog/iam-policy.json

## Create an IAM policy
WAF_LBC_IAM_POLICY=$(aws iam create-policy \
--policy-name AWSLoadBalancerControllerIAMPolicy-WAFDEMO \
--policy-document file://iam-policy.json)

## Get IAM Policy ARN
WAF_LBC_IAM_POLICY_ARN=$(aws iam list-policies \
--query "Policies[?PolicyName=='AWSLoadBalancerControllerIAMPolicy-WAFDEMO'].Arn" \
--output text)

## Create a service account
eksctl create iamserviceaccount \
--cluster=$WAF_EKS_CLUSTER_NAME \
--region $WAF_AWS_REGION \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--override-existing-serviceaccounts \
--attach-policy-arn=${WAF_LBC_IAM_POLICY_ARN} \
--approve

## Add the helm repo and install the AWS Load Balancer Controller
helm repo add eks https://aws.github.io/eks-charts && helm repo update
helm install aws-load-balancer-controller \
eks/aws-load-balancer-controller \
--namespace kube-system \
--set clusterName=$WAF_EKS_CLUSTER_NAME \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller \
--set vpcId=$WAF_VPC_ID \
--set region=$WAF_AWS_REGION

Verify that the controller is installed.

kubectl get deployment -n kube-system aws-load-balancer-controller

Deploy the sample app

We will use a sample application called Yelb for this demo. It provides an Angular 2-based UI that will represent a real-world application for this post. Here’s a high-level architectural view of Yelb:high-level architectural view of Yelb:

Clone the repository and deploy Yelb in your EKS cluster:

git clone https://github.com/aws/aws-app-mesh-examples.git
cd aws-app-mesh-examples/walkthroughs/eks-getting-started/
kubectl apply -f infrastructure/yelb_initial_deployment.yaml

Check the deployed resources within the yelb namespace:

kubectl get all -n yelb

Note: The Postgres database that Yelb uses is not configured to use a persistent volume.

Expose Yelb using an ingress

Let’s create a Kubernetes ingress to make Yelb available publicly. The AWS Load Balancer Controller will associate the the ingress with an Application Load Balancer.

cat << EOF > yelb-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: yelb.app
  namespace: yelb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb # Updated method to attach ingress class
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: yelb-ui
                port:
                  number: 80
EOF
kubectl apply -f yelb-ingress.yaml  

Test the application by sending a request using curl or by using a web browser to navigate to the URL.

It may take some time for the loadbalancer to become available, use command below to confirm:

kubectl wait -n yelb ingress yelb.app --for=jsonpath='{.status.loadBalancer.ingress}' &&

YELB_URL=$(kubectl get ingress yelb.app -n yelb \
-o jsonpath="{.status.loadBalancer.ingress[].hostname}")

You can obtain the URL using Kubernetes API and also navigate to the site by entering the URL:

echo $YELB_URL

Add a web application firewall to the ingress

Now that our sample application is functional, let’s add a web application firewall to it. The first thing we need to do is create a WAS web ACL. In AWS WAF, a web access control list or a web ACL monitors HTTP(S) requests for one or more AWS resources. These resources can be an Amazon API Gateway, AWS AppSync, Amazon CloudFront, or an Application Load Balancer.

Within an AWS WAF Web ACL, you associate rule groups that define the attack patterns to look for in web requests and the action to take when a request matches the patterns. Rule groups are reusable collections of rules. You can use Managed rule groups offered and maintained by AWS and AWS Marketplace sellers. When you use managed rules, AWS WAF automatically updates your WAF Rules regularly to ensure that your web apps are protected against newer threats. You can also write your own rules and use your own rule groups.

Create an AWS WAF web ACL:

WAF_WACL_ARN=$(aws wafv2 create-web-acl \
--name WAF-FOR-YELB \
--region $WAF_AWS_REGION \
--default-action Allow={} \
--scope REGIONAL \
--visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=YelbWAFAclMetrics \
--description 'WAF Web ACL for Yelb' \
--query 'Summary.ARN' \
--output text)

echo $WAF_WACL_ARN

Store the AWS WAF web ACL’s Id in an environment variable as it is required for updating the AWS WAF web ACL in the upcoming steps:

WAF_WAF_ID=$(aws wafv2 list-web-acls \
  --region $WAF_AWS_REGION \
  --scope REGIONAL \
  --query "WebACLs[?Name=='WAF-for-Yelb'].Id" \
  --output text)

Update the ingress and associate this AWS WAF web ACL with the ALB that the ingress uses:

cat << EOF > yelb-ingress-waf.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: yelb.app
  namespace: yelb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/wafv2-acl-arn: ${WAF_WACL_ARN}
spec:
  ingressClassName: alb 
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: yelb-ui
                port:
                  number: 80
EOF
kubectl apply -f yelb-ingress-waf.yaml 

By adding alb.ingress.kubernetes.io/wafv2-acl-arn annotation to the ingress, AWS WAF is inspecting incoming traffic. However, it’s not blocking any traffic yet. Before we send a request to our sample app using curl, let's wait for the loadbalancer to become ready for traffic

kubectl wait -n yelb ingress yelb.app --for=jsonpath='{.status.loadBalancer.ingress}'

Now lets send traffic to our sample app:

curl $YELB_URL

You should see a response from Yelb’s UI server:

Enable traffic filtering in AWS WAF

We have associated the ALB that our Kubernetes ingress uses with an AWS WAF web ACL Every request that’s handled by our sample application Yelb pods goes through AWS WAF for inspection. The AWS WAF web ACL is currently allowing every request to pass because we haven’t configured any AWS WAF rules. In order to filter out potentially malicious traffic, we have to specify rules. These rules will tell AWS WAF how to inspect web requests and what to do when it finds a request that matches the inspection criteria.

AWS WAF Bot Control is a managed rule group that provides visibility and control over common and pervasive bot traffic to web applications. The Bot Control managed rule group has been tuned to detect various types of bots seen on the web. It can also detect requests that are generated from HTTP libraries, such as libcurl.

Since our sample workload isn’t popular enough to attract malicious traffic, let’s use curl to generate bot-like traffic. Once enabled, we expect users who are accessing our application from a web browser like Firefox or Chrome to be allowed in, whereas traffic generated from curl would be blocked out.

While Bot Control has been optimized to minimize false positives, we recommend that you deploy Bot Control in count mode first and review CloudWatch metrics and AWS WAF logs to ensure that you are not accidentally blocking legitimate traffic. You can use the Labels feature within AWS WAF to customize how Bot Control behaves. Based on labels generated by Bot Control, you can have AWS WAF take an alternative action, such as sending out customized responses back to the client. Customers use custom responses to override the default response, which is 403 (Forbidden), for block actions when they’d like to send a nondefault status, serve a static error page code back to the client, or redirect the client to a different URL by specifying a 3xx redirection status code.

Create a rules file:

cat << EOF > waf-rules.json 
[
    {
      "Name": "AWS-AWSManagedRulesBotControlRuleSet",
      "Priority": 0,
      "Statement": {
        "ManagedRuleGroupStatement": {
          "VendorName": "AWS",
          "Name": "AWSManagedRulesBotControlRuleSet"
        }
      },
      "OverrideAction": {
        "None": {}
      },
      "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "AWS-AWSManagedRulesBotControlRuleSet"
      }
    }
]
EOF

Update the AWS WAF web ACL with the rule:

aws wafv2 update-web-acl \
--name WAF-FOR-YELB \
--scope REGIONAL \
--id $WAF_WAF_ID \
--default-action Allow={} \
--lock-token $(aws wafv2 list-web-acls \
--region $WAF_AWS_REGION \
--scope REGIONAL \
--query "WebACLs[?Name=='WAF-FOR-YELB'].LockToken" \
--output text) \
--visibility-config SampledRequestsEnabled=true,CloudWatchMetricsEnabled=true,MetricName=YelbWAFAclMetrics \
--region $WAF_AWS_REGION \
--rules file://waf-rules.json

Press q to exit the NextLockToken section. After waiting about 10 seconds, test the rule by sending a request:

curl $YELB_URL

As you see, access to the application is no longer accessible via the terminal.

Now lets open the same URL in your browser below and you should see the Yelb UI.

echo http://$YELB_URL

Note that we added AWSManagedRulesBotControlRuleSet rule group to AWS WAF web ACL (see configuration file waf-rules.json). This rule group contains rules to block and manage requests from bots as described in AWS WAF documentation. AWS WAF blocks the requests we send using curl because AWS WAF web ACL rules are configured to inspect and block requests for user agent strings that don’t seem to be from a web browser.

AWS WAF logging and monitoring

Network security teams require AWS WAF logging to meet their compliance and auditing needs. AWS WAF provides near-real-time logs through Amazon Kinesis Data Firehose. AWS WAF logs each request along with information such as timestamp, header details, and the action of the rule that matched. Customers can integrate AWS WAF logs with Security information and event management (SIEM) solutions or other log analysis tools for debugging and forensics. You can enable access logging in AWS WAF, save AWS WAF logs to Amazon S3, and use Amazon Athena to query WAF logs without creating servers. AWS WAF also allows you to redact certain fields during logging, which is helpful if your requests contain sensitive information that should not be logged.

After implementing an AWS WAF, it is critical to regularly review your applications’ traffic to develop a baseline understanding of its traffic patterns. Application and security teams should review AWS WAF metrics and dimensions to ensure that the web ACL rules block requests that can potentially compromise the application’s security and availability.

AWS Shield Advanced and WAF

AWS Shield Advanced subscribers can also engage the AWS Shield response team during an active DDoS attack. The AWS Shield Response team helps you analyze suspicious activity and assists you in mitigating the issue. The mitigation often involves updating or creating AWS WAF rules and AWS WAF web ACLs in your account.

AWS Firewall Manager

AWS Firewall Manager enables customers that operate multiple AWS accounts to centrally manage their web ACL. It simplifies administration and maintenance tasks across multiple accounts and resources for a variety of protections, including AWS WAF, AWS Shield Advanced, Amazon VPC security groups, AWS Network Firewall, and Amazon Route 53 Resolver DNS Firewall.

If you’d like to use AWS Firewall Manager to centralize the control of AWS WAF in multiple AWS accounts, you’d also need:

1. AWS Organizations: Your organization must be using AWS Organizations to manage your accounts, and All Features must be enabled. For more information, see Creating an organization and Enabling all features in your organization.
2. A Firewall Manager administrator account: You must designate one of the AWS accounts in your organization as the Firewall Manager administrator for Firewall Manager. This gives the account permission to deploy security policies across the organization.
3. AWS Config: You must enable AWS Config for all of the accounts in your organization so that Firewall Manager can detect newly created resources. To enable AWS Config for all of the accounts in your organization, use the Enable AWS Config template from the StackSets sample templates.

You can associate Firewall Manager with either a management account or a member account that has appropriate permissions as a delegated administrator. AWS Organizations’ documentation includes more information about using Firewall Manager with AWS Organizations.

Cleanup

Use the following commands to delete resources created during this post:

kubectl delete ingress yelb.app -n yelb
aws wafv2 delete-web-acl --id $WAF_WAF_ID --name  WAF-FOR-YELB --scope REGIONAL \
    --lock-token $(aws wafv2 list-web-acls \
      --region $WAF_AWS_REGION \
      --scope REGIONAL \
      --query "WebACLs[?Name=='WAF-FOR-YELB'].LockToken" \
      --output text) \
    --region $WAF_AWS_REGION
helm delete aws-load-balancer-controller -n kube-system

eksctl delete iamserviceaccount \
--cluster $WAF_EKS_CLUSTER_NAME \
--region $WAF_AWS_REGION \
--name aws-load-balancer-controller

aws iam detach-role-policy \
--policy-arn $WAF_LBC_IAM_POLICY_ARN \
--role-name $(aws iam list-entities-for-policy --policy-arn $WAF_LBC_IAM_POLICY_ARN --query 'PolicyRoles[0].RoleName' --output text)

aws iam delete-policy \
--policy-arn $WAF_LBC_IAM_POLICY_ARN

kubectl patch targetgroupbinding k8s-yelb-yelbui-87f2ba1d97 -n yelb --type='json' -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

kubectl patch svc yelb-ui -n yelb --type='json' -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

kubectl delete ns yelb

eksctl delete cluster --name $WAF_EKS_CLUSTER_NAME --region $WAF_AWS_REGION

Conclusion

This post demonstrates how to protect your web workloads using AWS WAF. Amazon EKS customers benefit from AWS WAF-provided AWS Managed Rules to add a web application firewall to web apps without learning how to write AWS WAF rules. Additionally, AWS WAF Bot Control gives you visibility and control over common and pervasive bot traffic that can consume excess resources, skew metrics, cause downtime, or perform other undesired activities.

We recommend implementing a AWS WAF and testing its effectiveness by conducting penetration tests regularly to identify gaps in your AWS WAF rules. The Guidelines for Implementing AWS WAF whitepaper provides a detailed implementation guide for anyone looking to protect web applications.