Scaling threat prevention on AWS with Suricata

This post was written by Victor Julien, Kelley Misata, Shakeel Ahmad, and Maritza Mills.

Suricata is a fast, robust, open source network threat detection engine that includes real-time intrusion detection (IDS), an inline intrusion prevention system (IPS), network security monitoring (NSM), and offline packet capture (pcap) processing. Suricata is owned by the community-run, non-profit Open Information Security Foundation (OISF).

Using Suricata on AWS for threat detection and response

Many customers use Suricata on Amazon Web Services (AWS) as an added layer of security and visibility in their environments. A common way of doing this is to deploy Suricata with Amazon Virtual Private Cloud (Amazon VPC) Traffic Mirroring, meaning that a copy of Amazon VPC traffic flows is sent to Suricata for evaluation of potential threats. A popular approach is to install Suricata on an Amazon Elastic Compute Cloud (Amazon EC2) instance and then use that instance as an Amazon VPC traffic mirroring target. When deployed this way, Suricata can be used as an intrusion detection and security monitoring tool inside your AWS environment.

As of November 2020, you can also use Suricata IPS rules as part of the AWS Network Firewall service by importing open source rulesets or authoring your own IPS rules using Suricata rule syntax. AWS Network Firewall lets you define rules to inspect and control traffic to and from your VPC using IP, port, protocol, domain names, and general pattern matches. Network Firewall acts on live traffic, allowing you to proactively stop threats in place and block traffic to or from untrusted addresses.

With AWS Network Firewall, you can leverage Suricata rules by entering them individually via the console or by passing them to the API as a file name. The advantages of leveraging open source rule languages like Suricata is that customers can collaborate with the entire cybersecurity community to prevent and detect potential threats. You can read more on Building an Open Source IDS IPS service for Gateway Load Balancer for in-line inspection and use Amazon EC2 with Suricata on AWS – Quick Start to deploy Suricata on Amazon EC2 for out-of-band inspection with Amazon VPC Traffic Mirroring.

Using Suricata-compatible rules in AWS Network Firewall

Let’s look at how we can utilize Suricata-compatible rules in AWS Network Firewall. For this example, we’ll use Suricata-specific rules from the community, such as Proofpoint’s OPEN ruleset. More details on open source and commercial rules are also available from Proofpoint.

AWS Network Firewall can be set up in various deployment models depending on the requirements. To get more details on the deployment models and how to set up firewalls, you can check out the AWS Network Firewall documentation and the Deployment Model for AWS Network Firewall article. For this example, we’ll assume that you already have AWS Network Firewall and logging correctly set up. You can also follow this sample code to provision AWS Network Firewall.

To use open source rules with AWS Network Firewall, you’ll need to follow these steps:

1. Download/clone rules

For this example, we’ll choose the “User-Agents” category from Proofpoint’s OPEN rules, which can be used to detect suspicious user-agents. Download the rules to your local machine, using wget and the following commands:

wget https://rules.emergingthreats.net/open/suricata-5.0/rules/emerging-user_agents.rules -O emerging-user-agents.rules

Once the rules are downloaded, note the location where the rules are saved, as we will use that in the next steps.

2. Create rule group with Suricata-compatible rules

To create a rule group with above ruleset, navigate to the AWS Console, VPC, Network Firewall rule group and choose Create Network Firewall rule group. Select Stateful rule group as shown in Figure 1.

Figure 1: Create Network Firewall rule group / Stateful rule group.

Once the rule group is selected, define a meaningful Name as well as Capacity and select the Suricata compatible IPS rules option. Once selected, a text input field will be available to input the ruleset. (Refer to the documentation to learn more about rule group capacity.)

Copy and paste the rules from Step 1, which contain the Emerging Threats User-Agents rules, and select Stateful rule group as displayed in Figure 2.

Figure 2: Create Network Firewall rule group / Suricata-compatible IPS rules.

You can also use the following AWS Command Line Interface (AWS CLI) commands to create the above rule group:

aws network-firewall create-rule-group \
	--rule-group-name emerging-user-agents-rules --type STATEFUL \
	--capacity 250 --rules file://emerging-user-agents.rules

Output:

{
    "UpdateToken": "7f31fca3-2e39-4064-8c71-aefedc9d5a82",
    "RuleGroupResponse": {
        "RuleGroupArn": "arn:aws:network-firewall:us-west-2:xxxxxx:stateful-rulegroup/emerging-user-agents-rules",
        "RuleGroupName": "emerging-user-agents-rules",
        "RuleGroupId": "ada8ab57-3851-474f-aee9-8d3dbb2dcd0a",
        "Type": "STATEFUL",
        "Capacity": 250,
        "RuleGroupStatus": "ACTIVE"
    }
}

Note: The HOME_NET rule group variable is used to define a source IP range, which is eligible for processing in a stateful domain list and optionally, Suricata-compatible IPS rule groups. By default, it is set to the Amazon VPC Classless Inter-Domain Routing (CIDR) where firewall endpoints are deployed. With a centralized deployment model, this variable must be expanded to include all CIDR ranges of your VPCs and on-premises networks to make them eligible for processing. See the stateful domain list documentation for more details.

3. Modify firewall policy to add and forward traffic to stateful rule groups

To add the newly created stateful rule from the previous step to the firewall policy, navigate to the AWS Console, VPC, Firewall policies and choose the policy used by your firewall, as shown in Figure 3.

Figure 3: Select the in-use firewall policy.

Now under Stateful rule groups section, select Add rule groups and choose Add stateful rule groups to the firewall policy, as shown in Figure 4.

Figure 4: Add stateful rule groups.

In next step, select the stateful rule group you created in Step 2 and choose Add stateful rule group to update the policy.

Figure 5: Update the policy.

4. Forward traffic

To ensure traffic is forwarded to stateful inspection engine, you also must add a custom-defined stateless rule group, which covers the interesting traffic, or you can set a default action for all stateless traffic to be forwarded to stateful rule groups in the firewall policy. In the example below, we have used the stateless default actions to forward traffic to the stateful rule groups.

Figure 6: AWS Network Firewall Policy with stateful default actions.

You can also complete the preceding steps using the AWS CLI. First, create a policy document JSON file with ARN of stateful rule group created in Step 2:

{
    "StatelessDefaultActions": [
        "aws:forward_to_sfe"
    ],
    "StatelessFragmentDefaultActions": [
        "aws:forward_to_sfe"
    ],
    "StatefulRuleGroupReferences": [
        {
            "ResourceArn": "arn:aws:network-firewall:us-west-2:xxxxxx:stateful-rulegroup/emerging-user-agents-rules"
        }
    ]
}

Next, fetch the latest UpdateToken and update the policy using the created policy document:

UPDATETOKEN=(`aws network-firewall describe-firewall-policy \
	--firewall-policy-name anfw-demo-firewall-policy \
	--output text --query UpdateToken`)

aws network-firewall update-firewall-policy \
	--firewall-policy-name anfw-demo-firewall-policy \
	--firewall-policy file://policy.json --update-token $UPDATETOKEN

Output:

{
    "UpdateToken": "241ecd75-b130-4ef3-9598-b2dbe5c5161b",
    "FirewallPolicyResponse": {
        "FirewallPolicyName": "anfw-demo-firewall-policy",
        "FirewallPolicyArn": "arn:aws:network-firewall:us-west-2:xxxxxx:firewall-policy/anfw-demo-firewall-policy",
        "FirewallPolicyId": "21ac1ba2-87db-4741-a8b7-fed12a2fe9a5",
        "FirewallPolicyStatus": "ACTIVE",
        "Tags": [
            {
                "Key": "Name",
                "Value": "anfw-demo-firewall-policy"
            }
        ]
    }
}

Then, verify the policy:

aws network-firewall describe-firewall-policy \
	--firewall-policy-name anfw-demo-firewall-policy  

Output:

{
    "UpdateToken": "241ecd75-b130-4ef3-9598-b2dbe5c5161b",
    "FirewallPolicyResponse": {
        "FirewallPolicyName": "anfw-demo-firewall-policy",
        "FirewallPolicyArn": "arn:aws:network-firewall:us-west-2:xxxxxx:firewall-policy/anfw-demo-firewall-policy",
        "FirewallPolicyId": "21ac1ba2-87db-4741-a8b7-fed12a2fe9a5",
        "FirewallPolicyStatus": "ACTIVE",
        "Tags": [
            {
                "Key": "Name",
                "Value": "anfw-demo-firewall-policy"
            }
        ]
    },
    "FirewallPolicy": {
        "StatelessDefaultActions": [
            "aws:forward_to_sfe"
        ],
        "StatelessFragmentDefaultActions": [
            "aws:forward_to_sfe"
        ],
        "StatefulRuleGroupReferences": [
            {
                "ResourceArn": "arn:aws:network-firewall:us-west-2:xxxxxx:stateful-rulegroup/emerging-user-agents-rules"
            }
        ]
    }
}

5. Test and monitor

To generate interesting traffic, you must have a compute resource (e.g., EC2 instance), which is protected by AWS Network Firewall. In this article, we’ll use an EC2 instance (mentioned as secure host) to generate a request against signature # 2029569 (also displayed below) from the Proofpoint’s OPEN ruleset imported in Step 3, which detects the suspicious user-agents.

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET USER_AGENTS Observed Suspicious UA (easyhttp client)"; flow:established,to_server; http.user_agent; content:"easyhttp client"; bsize:15; classtype:bad-unknown; sid:2029569; rev:1; metadata:attack_target Client_Endpoint, created_at 2020_03_04, deployment Perimeter, former_category USER_AGENTS, signature_severity Informational, updated_at 2020_03_04;)

To test, use the following command on secure host:

wget -U "easyhttp client" http://www.amazon.com -o /dev/null

This command will generate an HTTP GET request with a user agent as easyhttp client. To check the logs, navigate to AWS Console, CloudWatch, Log groups and select the configured log group for your firewall. You should receive an alert from AWS Network Firewall on the above traffic as displayed in Figure 7.

Figure 7: CloudWatch logs for Alert from AWS Network Firewall.

And that concludes how to import Suricata rule references into AWS Network Firewall.

Considerations when importing rules

Importing rules as shown above is great for quickly remediating gaps in a security posture. Once rules are imported, however, you’ll want to take the time to evaluate the rules for efficiency, performance, and relevancy, and potentially rewrite them based on your needs.

Why open source security matters

As cybersecurity threats continue to become more numerous and sophisticated, collaboration across the security community is imperative for protecting critical systems and ensuring user privacy. The interconnectedness of our shared infrastructures — including internet, healthcare, finance, transportation, and more — means that each of us has a vested interest in making sure our shared systems are secure. Knowledge of threats and how to protect against them is information that should be made accessible to everyone, whether you are protecting your personal computer or working for a large organization that is responsible for protecting millions of users.

Open source software like Suricata makes this security knowledge accessible by providing a neutral platform that brings together contributors from the entire security community, including individual developers, vendors, and public sector contributors. This benefits everyone involved because shared knowledge helps us all stay ahead of new threats and stay prepared to respond to incidents.

In support of this model, AWS has actively contributed back to the Suricata project with code contributions and AWS credits for their testing and integration needs. In one recent contribution, we added support for the GENEVE (Generic Network Virtualization Encapsulation) tunnel protocol to the Suricata engine. This tunneling protocol allows Suricata users with massive/segmented underlying networks to exceed the normal 4094 layer-2 VLAN limit. Within AWS Network Firewall, we use GENEVE to efficiently tunnel and route traffic through a load balancer, called Gateway Load Balancer, to endpoints containing our customer’s firewalls.

How to get involved with Suricata

The Suricata community encourages contributors of all backgrounds and experience levels to participate in the community. They have teamed up with Outreachy to recruit community members from backgrounds typically underrepresented in tech into paid internships to learn about and contribute to the project. If you want to dive deeper and get involved, we encourage you to visit the Suricata project. The community is currently working on the next release, with upcoming additions such as a lightweight rule profiling capability, experimental library support for Rust, and improved protocol support for HTTP/2 and QUIC protocols. These upcoming changes will improve the performance of the Suricata engine and allow users to troubleshoot the performance of their Suricata rules.

Closing thoughts

In this article, we introduced you to Suricata and to the Open Information Security Foundation, which supports it. We also explained the important role open source security projects have in the security community as a whole and walked through an example of how you can import open source rulesets into AWS Network Firewall.

About OISF

The OISF is a 501(c)3 US non-profit organization established in 2009 to provide architecture and infrastructure to open source security software communities and projects like Suricata. OISF serves to protect and maintain the authenticity of open source space while providing a safe and stable home for Suricata. OISF’s team and community members include world-class security and non-profit experts, programmers, and industry leaders dedicated to open source security technologies. OISF funding comes from support from consortium members, training events, and the annual Suricata community conference, SuriCon.