Tetragon – eBPF-based Security Observability & Runtime Enforcement

News: Tetragon has reached 1.0!

Tetragon 1.0 is here! Read the announcement to learn about the overhead benchmarks, default observability policy library, and tracing process execs!

Read the blog

We are excited to announce the Tetragon open source project. Tetragon is a powerful eBPF-based security observability and runtime enforcement platform that has been part of Isovalent Cilium Enterprise for several years. Today, we are open sourcing major parts as project Tetragon and open it up for collaboration with the entire community.

What is Tetragon?

Tetragon provides eBPF-based transparent security observability combined with real-time runtime enforcement. The deep visibility is achieved without requiring application changes and is provided at low overhead thanks to smart in-kernel filtering and aggregation logic built directly into the eBPF-based kernel-level collector. The embedded runtime enforcement layer is capable of performing access control on the system call and other enforcement levels.

Smart Observability

The foundation of Tetragon is a powerful observability layer that can introspect the entire system ranging from low-level kernel visibility to track file accesses, network activity, or capability changes, all the way up into the application layers covering aspects such as function calls into vulnerable libraries, tracing process execution, or understanding HTTP requests made. To list just a few of the capabilities, Tetragon can provide visibility into all kinds of kernel subsystems to cover namespace escapes, capability and privilege escalations, file system and data access, networking activity of protocols such as HTTP, DNS, TLS, and TCP, as well as the system call layer to audit system call invocation and follow process execution.

• Deep observability: Extensive visibility into all parts of the system and applications ranging from detecting low-level microbursts in TCP connections, providing HTTP visibility for golden signal dashboards, or the ability to detect the use of particular vulnerable shared libraries. The possibilities of eBPF are massive and Tetragon provides an easy to use framework to cover additional visibility use cases.
• Transparent: No application code changes are needed. All observability data is collected transparently from within the kernel. Applications cannot detect when they are being monitored which is ideal for security use cases.
• Low-Overhead: Minimal overhead is imposed on the system. Performing filtering, aggregation, metric accounting, and histogram collection directly in the kernel with eBPF helps to reduce the overhead. Tetragon uses efficient data structures such as per-CPU hash tables, ring buffers, and LRU maps to provide efficient and fast means of data collection and avoids sending vast amounts of low-signal events to the user space agent.

Real-Time Enforcement

Building on the rich observability, Tetragon provides real-time runtime enforcement. Unlike other systems which have a limited set of enforcement points such as only at the system call level, Tetragon is able to enforce security policies across the operating system in a preventive manner instead of reacting to events asynchronously. Tetragon has the ability to specify allow lists for access control at several layers. Security policies can be injected via Kubernetes (CRDs), a JSON API, or systems such as Open Policy Agent (OPA).

• Preventive security: Enforcement of runtime policies happens in-kernel and synchronously (real-time) allowing to prevent attacks instead of reacting to them asynchronously.
• Pluggable policy architecture: Security policies can come from a variety of sources. Either from the user directly, from systems like Open Policy Agent (OPA), or from third party components via the extendable Kubernetes CRD and JSON interfaces.

Why Tetragon?

Observability and runtime enforcement solutions have traditionally been based on one of the following concepts. Each of them have strengths but also come with weaknesses. Tetragon uses eBPF to bring a wide combination of strengths and gets rid of most weaknesses.

These above solutions operate on the application and system call level and describe methods that are primarily used to provide observability. They are sometimes used in combination with asynchronous enforcement where an agent relies on visibility data collected as described and then reacts to it. They do not provide any visibility into what is happening at the kernel level itself.

This second category involves solutions that directly operate at the kernel level and are primarily targeting enforcement use cases. The built-in kernel systems provide excellent enforcement options but are built with a strong focus of access control as use case. The major drawback of built-in kernel solutions is that they are challenging to extend, e.g. the kernel is not aware of what Kubernetes actually is. New kernel subsystems like LSM-eBPF are very promising and powerful but require recent kernels (≥5.7). Kernel modules solve the extendability problem but are often not an option in many environments due to the security risk they create.

Tetragon, with the help of eBPF, combines most of the strengths from existing solutions:

• Efficiency & Transparency: The application visibility and efficiency of app instrumentation but transparently and without requiring changes to application code.
• Synchronous Enforcement: The in-kernel synchronous enforcement capabilities of seccomp, SELinux, and LSM but with an elevation of the policy level from pure access control to the ability to prevent harm to the systems and components instead of exclusively focusing on restricting access to resources and data.
• Deep Extendable Visibility: The deep system observability capabilities and extendability of custom Linux kernel modules but without the security and availability risks.

In addition, Tetragon provides a modern agent that can natively integrate with modern observability and policy standards such as Kubernetes, Prometheus, fluentd, OpenTelemetry, Open Policy Agent, and also traditional SIEM platforms.

App Behavior Inspection with the Tetragon CLI

The Tetragon CLI, codenamed “amazing-cli” at Isovalent, will likely be your first entry point into observability with Tetragon. It’s the type of CLI that frequently gives goosebumps as it exposes so much information about complex systems in a simple, easy to understand form.

In the following example, we are showing how Tetragon is used to observe the behavior of a Kubernetes pod that runs curl -L github.com:

1. curl -L github.com is started.
2. We see the DNS resolution for github.com and the opening of a TCP connection to port 80.
3. github.com returns HTTP code 301 within 7ms to redirect traffic to port 443. This is done to enable TLS. curl follows the redirect and performs another DNS resolution.
4. A TCP connection port 443 is opened. The TLS handshake is observed. TLS 1.3 is used and the cipher negotiated is AES-128-GCM-SHA256.
5. Data is exchanged, a total of ~80 bytes on port 80, 218KiB on port 443.
6. The process exits with error code 0.

Combining Network & Runtime Visibility

An exciting aspect of Tetragon’s use of eBPF is that it can combine aspects of visibility that have often be treated separately so far. Here is an example of combined network and runtime visibility to demonstrate the ability to identify what processes are involved in what type of network communication. The following example shows the use of Tetragon to observe a Kubernetes pod that was compromised and subject to a lateral movement attack:

In the above diagram we see a classical lateral movement attack via a reverse shell:

1. A Kubernetes pod “crawler-c57f9778c-wtcbc” is running in the Kubernetes namespace “tenant-jobs”. The pod is run via containerd which is running as a sub process of the PID 1 init process. The binary running inside of the pod is called “crawler” which in turn spawns a node process executing “server.js”.
2. The node app makes egress network connections to “api.twitter.com” as well as a Kubernetes service elasticsearch.
3. 5 Minutes after the pod was started, another sub process invoking netcat (nc) was started. Seeing runtime and network observability combined, it is obvious that this is an ongoing reverse shell attack.
4. The attacker can then be observed to be running curl to reach out to the internal elasticsearch server and then use curl to upload the retrieved data to an S3 bucket.

Monitoring Access to Sensitive Files

Tetragon has the ability to monitor file and data access. This following examples illustrates integration with Splunk to track access to sensitive files while providing context of the access such as Kubernetes metadata, container image, binary, and user information:

The above example lists accesses to /etcd/passwd including the process as well as container image and Kubernetes namespace context. Besides monitoring of /etc/passwd or /etc/shadow, other obvious files to monitor on the system would be the UNIX domain socket of the container runtime or any systemd unit or init files that could alter the bootstrapping of the machine.

With Tetragon’s enforcement capability, accesses to such sensitive files can not only be monitored but also prevented.

Detection of Weak TLS Ciphers & Versions

TLS is a cornerstone of security today but usage of older TLS versions or insecure TLS ciphers can pose a serious security threat. This can range from unintentional wrong usage to intentional TLS downgrade attacks.

The above dashboard shows the distribution of TLS protocol versions and ties it to the Kubernetes pod and namespace context. It can also show the distribution of ciphers and even more importantly the cipher key length.

Runtime Aware Network Policy

You are possibly familiar with Kubernetes NetworkPolicies that define the allowed and denied network communication for Kubernetes workloads. Oversimplified, these policies describe that pod A is allowed to talk to pod B or CIDR 10.0.0.0/8 but pod A is not allowed to talk to pod C or CIDR 20.1.1.1/32.

The granularity of these policies are at the level of a pod. It doesn’t matter whether it is app.js running in the pod creating the request or the attack.py script invoking curl. With Tetragon technology, we will be extending Cilium’s Network Policy capabilities to include runtime context:

kind: CiliumNetworkPolicy
[...]
  endpointSelector:
  - matchLabels:
    - name: Frontend
  egress:
  - toEndpoints:
      - matchLabels:             
        - name: Backend
    fromRuntime:
      - binary: app.py
        privileged: false

The above policy leads to a significantly better least privilege policy. It allows Frontend pods to talk to Backend pods but only if:

• the binary in the source pod is called app.py
• the process in the source pod is running unprivileged

The example shown takes the binary name and privileged execution context into account but this concept can be extended to also restrict based on additional parameters such as ensuring that the process is still namespaced, the UID/GID context, or even taking memory hashes of the executable into account.

Conclusion

We are very excited about Tetragon and bringing it to the open source community. The additional observability and security control will improve the life of security, platform, and application teams and it will lead to more secure infrastructure overall, in particular Kubernetes environments. Even more importantly, we are in particular excited about what is yet to come as collaboration expands to the open source community.

If you are interested to learn more about Tetragon, we’ll be hosting a webinar about Tetragon in the coming weeks. It will feature an introduction, demos, and the opportunity to ask questions. In the meantime, join #tetragon on the Cilium Slack to get started and meet the team.

If you are interested in Isovalent Tetragon Enterprise, feel free to contact us.

Further Reading

• cilium/tetragon on GitHub
• Sign up for the Tetragon Deep Dive
• Learn more about Isovalent Cilium Enterprise
• Detecting and Blocking log4shell with Isovalent Cilium Enterprise
• Detecting a Container Escape with Cilium and eBPF
• Isovalent Cilium Enterprise 1.10: Timescape, Runtime Observability & Enforcement, Hubble RBAC