eBPF & Tetragon: Tools for Detecting XZ Utils CVE 2024-3094 Exploit

Tetragon in Action: Join the webinar on April 18th!

See the top security use cases for Tetragon and eBPF. We will be walking through the XZ CVE and mechanisms for using eBPF to detect vulnerable libraries during the OpenSSH startup process.

Get the invite!

Why do big CVEs always hit our feeds around holiday weekends? This past week, the unwelcome arrival of the backdoor in XZ Utils (CVE-2024-3094) created a buzz in the security community. It’s not just the timing that has caused extreme interest in this particular exploit, the complexity and obfuscation of this vulnerability in the XZ Utils package have made it a noteworthy case study on how actors are infiltrating projects and patiently laying down plans that take years to come to fruition. Leveraging eBPF provides security teams a near-zero overhead framework for detecting vulnerable versions of XZ Utils in the kernel, as OpenSSH is launched.

Let’s dissect the technicalities of the exploit throughout the different attack stages. In this blog post using Tetragon, an eBPF-based runtime security product, we provide a working solution to detect the XZ backdoor in OpenSSH. If you are looking for an immediate detection policy to apply, please navigate to the policy example below. 

What is the new XZ Utils CVE? 

Quick FAQ’s: 

1. What is XZ Utils: A collection of open-source software tools for lossless file compression and decompression using the XZ format, offering a high compression ratio and widely used in software distribution or archiving. 
2. What is the exploit for CVE-2024-3094: The exploit utilizes a sophisticated backdoor embedded in the liblzma library, introduced through modified build scripts and obfuscated test files. This backdoor enables attackers to potentially execute arbitrary code or access environments. 
3. What versions of XZ/liblzma are affected: Versions 5.6.0 and 5.6.1 
4. What risks if exploited: Significant risk for remote code execution (RCE) through SSH authentication. This leads to unauthorized access, RCE, data breaches, and compromised system integrity. 
5. How to update or patch: CISA advised downgrading to uncompromised XZ Utils versions (<5.6.0). 
6. How Tetragon detects vulnerable versions: While the OpenSSH process is launching, Tetragon uses eBPF programs in the kernel to trace and detect vulnerable versions of the XZ/liblzma library dependency. 
7. How is the XZ/liblzma library connected to OpenSSH: Notably, some versions of OpenSSH run using liblzma as a dynamically loaded library, and conditions for the attack appear to target environments using OpenSSH and x86-64 Linux systems. 

CVE-2024-3094 exploits a backdoor within the liblzma library, versions 5.6.0 and 5.6.1, part of the XZ Utils software package. On March 29th, Andres Freund (principal software engineer at Microsoft) emailed the oss-security@openwall mailing list about a backdoor identified in XZ Utils, specifically targeting the liblzma library, versions 5.6.0 and 5.61. This vulnerability, designated CVE-2024-3094, has garnered significant attention due to obfuscation techniques and the widespread use of the affected software library in compressing data types across Linux systems. 

Why is the XZ Util exploit significant? 

This is a significant vulnerability for multiple technical and social reasons. Let’s focus on both areas in more detail.

At a technical level the backdoor’s ability to hijack authentication processes in OpenSSH through dependencies marks a significant security concern. If OpenSSH is compromised the attacker has secured an entry point into the target system, this has the potential for unauthorized access or RCE. XZ Utils and the liblzma library are foundational to numerous Linux distributions and applications, in particular OpenSSH to enable remote access to systems. Its widespread use in software distribution systems amplifies the potential attack surface, making it a critical point of security concern.

This mirrors the cause for concern the technical community sounded when issues arose regarding the Apache Log4j vulnerability “Log4Shell”, caused by a supply chain attack when a malicious developer inserted a security flaw that allowed attackers a low threshold entry point for executing code on vulnerable systems.  The industry post-mortem from Log4jshell served as a case study in how to better protect runtime, one key runtime technique that gained visibility is how it would have been detected and blocked in real time inside the kernel using eBPF.

Due to the backdoor’s obfuscated nature and the exploitation of legitimate functionalities like IFUNCs (indirect functions, allowing programs to dynamically choose the best version of a function at startup depending on the system), CVE-2024-3094 poses significant challenges for detection and mitigation. It was only by chance during performance benchmarking that Andres Freund could see the minor impact that the liblzma library was having on logins with SSH taking up more than expected CPU and errors appearing with Valgrind (memory debugging tool). 

Additionally, what’s noteworthy about the creation of this backdoor is that a sophisticated upstream contributor spent years embedding themselves into the XZ project and laying the groundwork for this exploit. Jia Tan, alias of the attacker, socially engineered their way into becoming a maintainer by a (now well-documented) series of steps over two years – involving contributing innocuous patches, gaining maintainer privileges, and planting the seeds of the backdoor. 

They even attempted to commit part of the backdoor to the Linux Kernel! 

How does the exploit work technically? 

Strap in, this is a multi-stage process. This backdoor was introduced through modified source code packages rather than the main git repository, evading standard review processes. The exploit unfolds in several stages:

1. Modified build process: The vulnerability begins at the source, with modified release tarballs of XZ Utils versions 5.6.0 and 5.6.1. Attackers manipulated these release tarballs, incorporating a modified m4/build-to-host.m4 script. This script, when executed during the library’s normal build process, triggers the next phase of the exploit.
   
2. Obfuscated script execution: The altered m4/build-to-host.m4 build script executes another script that is obfuscated within two seemingly innocuous test files:

• tests/files/bad-3-corrupt_lzma2.xz
• tests/files/good-large_compressed.lzma
  
  These files contain encoded data that, when decoded, plays a central role in injecting the malicious backdoor.

3. Backdoor Injection: The obfuscated script evaluates the build environment to ensure it matches the targeted criteria (e.g., x86-64 Linux systems or compiled with gcc). If the criteria are met, it proceeds to alter the liblzma makefile, subtly modifying the compilation process to include malicious code within the liblzma binary itself.

Source: For a breakdown of the corrupted test files and bytecode, Thomas Roccia compiled a helpful cheat sheet on the CVE exploit stages shown above. 

How does Tetragon detect and mitigate?

Tetragon is an open-source, runtime security and enforcement tool that leverages eBPF for in-kernel security visibility. Using eBPF, Tetragon loads programs and tracing policies in the kernel to detect and react to security-significant events. By operating at the kernel level, Tetragon can observe processes as they execute and operate with both kernel and Kubernetes awareness for full context regarding what happens in runtime. 

In the case of CVE-2024-3094, Tetragon has proven a powerful tool for security teams to turn a zero-day into a detected threat quickly. The below tracing policy monitors the OpenSSH process as it launches and loads the XZ/liblzma library. 

Example Tetragon TracingPolicy to apply

The following TracingPolicy to detect the XZ backdoor using Tetragon was contributed by Djalal Harouni, a core contributor to the Tetragon project. 

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "cve-2024-3094-xz-ssh"
  annotations:
    url: "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-3094"
    description: "Detects if OpenSSH is using vulnerable XZ libraries"
    author: "Tetragon.io Team"
spec:
  kprobes:
  - call: "security_mmap_file"
    syscall: false
    return: true
    # message: "OpenSSH daemon using vulnerable XZ libraries CVE-2024-3094"
    # tags: [ "cve", "cve.2024.3094" ]
    args:
    - index: 0
      type: "file"
    - index: 1
      type: "uint32"
    - index: 2
      type: "nop"
    returnArg:
      index: 0
      type: "int"
    returnArgAction: "Post"
    selectors:
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/sbin/sshd"
      matchArgs:
      - index: 0
        operator: "Postfix"
        values:
        - "liblzma.so.5.6.0"
        - "liblzma.so.5.6.1"
      matchActions:
        - action: Post
          rateLimit: "1m"

Next apply the policy to the target namespace. 

kubectl apply -n your-namespace-name -f cve-2024-3094-xz-ssh.yaml

What does this TracingPolicy do? 

This policy monitors the startup process and generates an event that alerts teams if the vulnerable version is being run with OpenSSH. The out of the box configuration for this policy is to generate a check when OpenSSH runs and produce a JSON event alerting if the system is vulnerable, with events being easily piped over to Splunk or a preferred SIEM. 

Note, the default policy does not provide prevention to stop OpenSSH from launching. This policy can be modified as needed to include that capability for teams looking to implement enforcement steps. 

When OpenSSH launches, a list of software dependencies is checked for and started. One of these libraries is the XZ/liblzma library. The above TracingPolicy observes events in the memory map associated with OpenSSH, and evaluates whether the XZ programmatic library uses the vulnerable versions of liblzma.so.5.6.0 or liblzma.so.5.6.1. 

Once identified, teams can quickly stop the exploitable OpenSSH process and rollback the XZ/liblzma version to <5.6.0 as per the recommended steps. 

What is the output when detecting CVE 2024-3094 with Tetragon?

{
  "process_kprobe": {
    "process": {
      "exec_id": "cm9yb25hLXR0eTo0Nzk4MTkyNTU1NjI2Nzo1MzI3NQ==",
      "pid": 53275,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "arguments": "-t",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-04T12:51:56.492325870Z",
      "auid": 4294967295,
      "parent_exec_id": "cm9yb25hLXR0eToyMjAwMDAwMDA6MQ==",
      "refcnt": 1,
      "tid": 53275
    },
    "parent": {
      "exec_id": "cm9yb25hLXR0eToyMjAwMDAwMDA6MQ==",
      "pid": 1,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/lib/systemd/systemd",
      "arguments": "splash",
      "flags": "procFS auid rootcwd",
      "start_time": "2024-04-03T23:32:14.786769146Z",
      "auid": 4294967295,
      "parent_exec_id": "cm9yb25hLXR0eToxOjA=",
      "tid": 1
    },
    "function_name": "security_mmap_file",
    "args": [
      {
        "file_arg": {
          "path": "/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.1",
          "permission": "-rw-r--r--"
        }
      },
      {
        "uint_arg": 3
      }
    ],
    "return": {
      "int_arg": 0
    },
    "action": "KPROBE_ACTION_POST",
    "policy_name": "cve-2024-3094-xz-ssh",
    "return_action": "KPROBE_ACTION_POST"
  },
  "node_name": "rorona-tty",
  "time": "2024-04-04T12:51:56.494017459Z"
}
{
  "process_kprobe": {
    "process": {
      "exec_id": "cm9yb25hLXR0eTo0Nzk4MjE4MDk2MTk5Mjo1MzI3Nw==",
      "pid": 53277,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "arguments": "-D",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-04T12:51:56.747731591Z",
      "auid": 4294967295,
      "parent_exec_id": "cm9yb25hLXR0eToyMjAwMDAwMDA6MQ==",
      "refcnt": 1,
      "tid": 53277
    },
    "parent": {
      "exec_id": "cm9yb25hLXR0eToyMjAwMDAwMDA6MQ==",
      "pid": 1,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/lib/systemd/systemd",
      "arguments": "splash",
      "flags": "procFS auid rootcwd",
      "start_time": "2024-04-03T23:32:14.786769146Z",
      "auid": 4294967295,
      "parent_exec_id": "cm9yb25hLXR0eToxOjA=",
      "tid": 1
    },
    "function_name": "security_mmap_file",
    "args": [
      {
        "file_arg": {
          "path": "/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.1",
          "permission": "-rw-r--r--"
        }
      },
      {
        "uint_arg": 1
      }
    ],
    "return": {
      "int_arg": 0
    },
    "action": "KPROBE_ACTION_POST",
    "policy_name": "cve-2024-3094-xz-ssh",
    "return_action": "KPROBE_ACTION_POST"
  },
  "node_name": "rorona-tty",
  "time": "2024-04-04T12:51:56.748522771Z"
}

​This JSON event provides detailed, real-time visibility into system processes and library loading events, showcasing Tetragon’s ability to monitor and detect potentially malicious activity here and in future CVEs. Specifically, it illustrates how Tetragon captures the loading of the vulnerable liblzma.so.5.6.1 library by the OpenSSH daemon (/usr/sbin/sshd), triggered during system startup by systemd. This detection is critical for identifying systems at risk of CVE-2024-3094 exploitation at runtime.

Breaking down key observability pieces in the JSON output:

Process Information: The process field includes details about the SSH daemon, such as its execution ID, process ID, and the command-line arguments, highlighting the exact instance of OpenSSH involved.

Parent Process Context: The parent field shows that the SSH daemon was started by systemd, providing context for how the service was initiated, which is vital for understanding the service’s startup sequence and potential points of vulnerability exploitation throughout the process tree.

Function Call Detection: The function_name security_mmap_file indicates Tetragon detected a memory mapping event, at the point where the vulnerable XZ/liblzma library was mapped into the process’s memory space.

Library Path and Permissions: The file_arg path /usr/lib/x86_64-linux-gnu/liblzma.so.5.6.1 confirms the specific library version loaded, and its permissions, underscoring the direct detection of the CVE-2024-3094 vulnerable library version.

Action Taken: The action and return_action fields indicate the policy’s response to this detection. While the provided policy logs the event (KPROBE_ACTION_POST), it can be adjusted to take preventive actions, such as stopping the process to mitigate the risk immediately.

Conclusion

This exploitation of the XZ/libzlzma project creates a deep cause for concern and demands quick scoping of our environments and proactive detection as seen in the eBPF program. For teams looking to maintain control over their environment after the initial assessment, this policy offers a near-zero overhead implementation to monitor for vulnerable versions. 

The reliance on open-source software cannot be understated in helping organizations achieve success through the delivery of their platforms. However gaining trust in using those components, whether directly implemented or as part of a wider package, must be earned. It certainly will be the case that this kind of sophistication of exploit will be the start, rather than the end of future examples and CVEs of attacks on our software packages. 

With Tetragon providing an easy-to-use framework of TracingPolicies to detect, audit and act upon the use of software packages running on your systems, you have a flexible implementation to be able to react immediately to the latest security vulnerabilities and CVEs. As part of the Tetragon documentation, provided is an existing library of example policies to get you started across the areas of privilege escalation, fileless executions, SSHD and “kubectl exec”s session monitoring or shared library loading. 

For customers looking to take their security platform to the next level, Isovalent Enterprise for Tetragon offers further capabilities such as detecting known CVEs from a database query execution, file integrity monitoring, and advanced network telemetry from parsing network protocols.

Learn more

• Detecting Log4Shell with eBPF
• Lab: Getting started with Tetragon
• Lab: Discovery journey for SecOps
• Deeper Host Network Observability with eBPF

References 

• A timeline of Jia Tan’s infiltrating the XZ project.
• Thomas Roccia’s build-to-host.m4 breakdown.
• FAQ on XZ Utils backdoor by Sam James.
• Bash stage obfuscation explained.
• MITRE CVE Listing.
• XZ bot for reproducing environment.
• Email from Andres Freund to oss-security@openwall.
• Attacker attempted to get backdoor into Fedora 40 & 41.
• Attempt to commit to the Linux kernel.
• Breakdown of the attack and config scripts.
• Timeline and background to the attack.