How to use AWS WAF Bot Control for Targeted Bots signals and mitigate evasive bots with adaptive user experience

Introduction

The AWS WAF Bot Control rule group includes rules for detecting and managing bot threats. These threats range from easily identified common bots through to coordinated targeted bots that evade detection by operating across multiple hosts. Like any other security controls, Amazon Web Services (AWS) WAF Bot Control for Targeted Bots rules can also introduce ‘false positives’ that inadvertently prevent legitimate users to access the application.

AWS WAF Bot Control for Targeted Bots rules use Block, CAPTCHA, and Challenge as the rule actions for mitigating bot threats based on indicators of bot activity. When it comes to protecting your web applications from bot threats, traditional methods such as blocking or CAPTCHA can often disrupt user experience. By default, Bot activity with a confidence signal of ‘high’ are blocked by AWS WAF, while CAPTCHA is used when there is uncertainty on whether the request is from a bot.

This post is structured into three comprehensive sections.

1. First, we delve into how you can use AWS WAF labels and pass signals to your application.
2. Second, we explore the method of combining these labels to indicate various bot confidence levels.
3. Last, we discuss different mitigation strategies to adapt user experience using the forwarded signals as headers to your application. This way you can balance usability for legitimate users while also blocking evasive bots.

1. Using AWS WAF labels to pass bot activity signals to your application

The sequence diagram in Figure 1 provides an overview of using AWS WAF rules to insert a header into the request to your application when AWS WAF Bot Control for Targeted Bots rules identify bot activity signals. You use this header to drive application logic that adapts the user experience as a way of mitigating the bot threat, rather than using AWS WAF Block or CAPTCHA actions.

Figure 1: Sequence diagram demonstrating AWS WAF inserting a header to the application for bot mitigation

The Challenge action is non-terminating for requests with valid tokens, which means AWS WAF continues evaluating rules in priority order after the AWS WAF Bot Control rule group. The rule group adds labels that indicate bot activity signals that you match against in a subsequent custom rule.

First, we override the action for all the AWS WAF Bot Control for Targeted Bots rule group rules to Challenge instead of Block or CAPTCHA as shown in Figure 2. Figure 3 shows a subset of the labels that may be added to requests depending on match criteria. A full list of labels is available from the AWS WAF Bot Control rule group list documentation. For step-by-step instructions to configure AWS WAF Bot Control rule group instructions, refer to our AWS Security post.

Figure 2: Overriding AWS WAF Bot Control Targeted rule group rules to use the Challenge Action

Figure 3: Some of the labels added to a request after being evaluated by the AWS WAF Bot Control rule group

After configuring the Challenge action, we recommend using application integration SDKs, as this provides insights into the client environment that AWS WAF uses to identify bot activity and also reduce intelligent threat detection costs. You can still use the Count action (for example, if your clients don’t support client-side interactions). However, AWS WAF Bot Control rules are less effective.

The next step is to configure a subsequent rule to match the label. Figure 4 shows a subsequent custom rule that matches on the label awswaf:managed:aws:bot-control:TGT_VolumetricSession with the Count action. Under Custom request add the custom header bot-rule=volumetric, which is forwarded to your application as x-amzn-waf-bot-rule=volumetric.

Figure 4: Custom AWS WAF rule to insert a header when matching an AWS WAF Bot Control label

2. Combining labels to indicate bot confidence levels

The AWS WAF Bot Control rule group for Target Bots includes multiple rules. Creating separate headers for each rule would be overly onerous to manage both in the AWS WAF web ACL, and your application. Instead, you can use three custom rules to insert a single header to indicate a confidence level: low, medium, or high when a request matches one of multiple labels. Define a confidence level within the web ACL so that your application doesn’t need to be changed as new rules become available, or you decide to change the labels being matched at each level.

We have a sample web ACL that implements these custom rules by using an OR rule statement to match on labels with the Count action, and inserting a header x-amzn-waf-bot-confidence={low|medium|high}, as shown in Figure 5.

Figure 5: Custom label matching rules after AWS WAF Bot Control rule group

We combine the AWS WAF Bot Control labels with the following three rules:

1. Low confidence bot signals rule: matches against one of labels that correspond to the rules: TGT_ML_CoordinatedActivityLow (this uses machine learning (ML) to determine low confidence likelihood of distributed, coordinated bot activity), TGT_VolumetricSession (indicating requests exceeding normal thresholds for a single client in the prior five minutes), TGT_TokenReuseIpLow, TGT_TokenReuseCountryLow, and TGT_TokenReuseAsnLow (these rules indicate an AWS WAF token is being used across multiple IP addresses, countries, or Autonomous System Numbers (ASN) in the prior five minutes). This rule inserts the header x-amzn-waf-bot-confidence=low, which is then forwarded to your application.
2. Medium confidence bot signals rule: matches against the medium equivalent labels of the prior coordinated activity and token re-use rules. We also include the labels corresponding to the rules TGT_SignalAutomatedBrowser, TGT_SignalBrowserAutomationExtension, TGT_SignalBrowserInconsistency (rules that identify browser inconsistency indicating bot traffic), and TGT_VolumetricSession (indicating an abnormally high number of requests coming from a single client in the prior five minutes). This rule inserts the header x-amzn-waf-bot-confidence=medium, which is then forwarded to your application.
3. High confidence bot signals rule: matches against the high equivalent labels of the prior coordinated activity and token re-use rules. We also include the rule TGT_VolumetricSessionMaximum (indicating maximum confidence on an abnormally high number of requests coming from a single client in the prior five minutes). This rule inserts the header x-amzn-waf-bot-confidence=high, which is then forwarded to your application.

You must also change the default action for each of the AWS WAF Bot Control for Targeted Bots rules above from CAPTCHA or Block to Challenge as shown previously in Figure 2. The TGT_TokenAbsent rule uses Challenge instead of Count so that all requests need a token. This increases the effectiveness of AWS WAF Bot Control, but it needs careful token management in your clients. Read optimizing web application user experiences with AWS WAF JavaScript integrations for further guidance. As shown in Figure 6, you can leave Block as the default Common Bot rule action with an assumption that no legitimate users falsely appear as self-identified bots. Therefore, there is no impact to the user experience by blocking these requests.

Figure 6: AWS WAF Bot Control Common rule group with default action to Block

3. Mitigation strategies for adapting the user experience using bot activity signals

This section provides different mitigation strategies for using confidence levels to drive adaptive user experiences in your application. The combination of strategies you use, and the confidence level you apply to each strategy (low, medium, high), depends on 1/ your tolerance for trading off the impact of bots, 2/ the user experience impact on your legitimate users, and 3/ the amount of development work you want to undertake to implement these strategies in your application. For example, you may decide to continue blocking high-confidence bot threats with AWS WAF rather than handling mitigations in your application.

Each strategy defined in the following sections uses authentication as a mitigation to bot threats. Authentication increases the cost for bot operators as it necessitates them to manage a set of identities. Another benefit is that you can use further heuristics to determine whether an authenticating user is under the control of a bot. For example, AWS WAF Fraud Control rule groups can help stop bot-driven log in or registration requests before they reach your application. If you are using a third-party identity provider, such as Amazon Cognito, then it may have its own security mechanisms. Cognito’s Threat Protection can adapt authentication based on suspicious behavior.

Strategy 1: Mitigate low confidence bot threats by withholding valuable data for unauthenticated users

Many organizations prefer that new legitimate users accessing the application can engage without friction driven by CAPTCHA, authentication, or other impediments. If AWS WAF identifies low likelihood bot traffic, then one way to trade off this risk is by withholding valuable data for unauthenticated users. For example, if scrapers are gathering valuable pricing data or images, then you adjust your application logic to withhold pricing or use placeholder images. Then, your application drives users toward authenticating to ‘unlock’ the respective data.

Legitimate users landing on your site can engage with minor inconvenience. However, scrapers and other bot threats can’t gather valuable data without logging in. The sequence diagram in Figure 7 shows the user interaction pattern with your application for low likelihood bot threats indicated by the header x-amzn-waf-bot confidence=low. This increases the cost (and thus effectiveness of the mitigation) for bot operators as they now must create identities for your application to avoid detection.

Figure 7: Sequence diagram demonstrating the application withholding valuable data when low likelihood bot activity is detected by AWS WAF

Strategy 2: Mitigate medium confidence bot threats by requiring users to authenticate

AWS WAF Bot Control rules use the CAPTCHA action when they identify certain signals of bot activity. CAPTCHA may be unacceptable for your organization due to its impact on user engagement or requirement to use a mobile application. In this strategy we force the user to authenticate before being able to further engage with your application, which acts as an alternative to CAPTCHA completion. For users that are already authenticated, you can use ‘step up’ authentication to necessitate further identity validation, such as MFA, before permitting them to continue.

The sequence diagram in Figure 8 shows this interaction pattern when your application receives the header x-amzn-waf-bot-confidence=medium. This forces users to create an account with your application, and when it’s authenticated it provides insights into the user’s activity that can be used to further mitigate bots.

Figure 8: Sequence diagram demonstrating the application requiring user authentication when medium likelihood bot activity is detected by AWS WAF

Strategy 3: Mitigate high confidence bot threats by triggering a different business process

Blocking high confidence bot threats is the default strategy for AWS WAF Bot Control rules that identify high likelihood bot activity. This may be unacceptable to your organization when there is no tolerance for missing out on legitimate users using your application. You can instead balance mitigating high confidence bot threats by taking different paths for business processes.

For example, when a user is submitting a form, perhaps checking out inventory, or signing up for a new service, then rather than automatically processing the request, a separate queue is used for manual review of transactions for suspicious characteristics. This increases the cost for your business of the transaction, while legitimate users can still proceed. The sequence diagram in Figure 9 shows a user submitting a form that takes a separate path when the application receives a x-amzn-waf-bot-confidence=high header. A high confidence bot threat should mean the likelihood of inconveniencing your legitimate users is low while still mitigating the threat. You can also combine this with Strategy 1 (withholding valuable data), and Strategy 2 (necessitating authentication) as more mitigations.

Figure 9: Sequence diagram demonstrating the application using a different business process when high likelihood bot activity is detected by AWS WAF

Strategy 4: Mitigate bot-driven users by varying experiences based on user traffic volumes

As further mitigation, you can control the action of individual users in your application based on usage data. For example, a user may only be able to complete so many transactions per day based on the likelihood that they are a bot.

You can use AWS WAF to manage individual user activity in your application for up to a 10 minute window by using a rate-based rule that aggregates based on your application’s session token. This works even if the same session is being re-used across multiple bots. The rate-based rule aggregates on the Authorization header to match a JSON Web Token (JWT). The rule inserts a header x-amzn-waf-rate-exceeded=10 when the number of requests exceeds 10 in a one minute period.

On the application side, you can combine this with the x-amzn-waf-bot-confidence header to adjust the user experience. The sequence diagram in Figure 10 shows an example of requests to a paginated list API that usually returns 20 items instead returning five items for medium confidence bots when also matching the x-amzn-waf-rate-execeded header. This limits the impact of API scraping bots while still engaging legitimate users. For further examples of using rate-based rules, refer to this AWS Security post.

Figure 10: Sequence diagram demonstrating the application returning fewer items when the rate of requests exceeds 10 in a minute and medium likelihood bot activity is detected by AWS WAF

Conclusion

In this post we described four strategies for adapting your application’s user experience based on low, medium, and high bot confidence levels as a way of mitigating bot threats while still making sure that ‘false positive’ legitimate users are able to use your application.

When applying these strategies you need to decide what your tolerance is for false positives to your legitimate users as to whether your application uses low, medium, or high confidence to apply this logic, and the AWS WAF Bot Control rules that apply in each level. These strategies rely on authentication as a mitigation. Therefore, we recommend that you complement this with threat protection capabilities in your identity provider, or AWS WAF Fraud Control rule groups.

To keep updated with AWS WAF, refer to AWS WAF Security Blog and what’s new with AWS Security, Identity, and Compliance. If you have feedback about this post, then submit comments in the Comments section. If you have questions about this post, then start a new thread on AWS WAF re:Post or contact AWS Support.

Author bios