Tetragon – eBPF-based Security Observability & Runtime Enforcement
News: Tetragon has reached 1.0!
Tetragon 1.0 is here! Read the announcement to learn about the overhead benchmarks, default observability policy library, and tracing process execs!Read the blog
We are excited to announce the Tetragon open source project. Tetragon is a powerful eBPF-based security observability and runtime enforcement platform that has been part of Isovalent Cilium Enterprise for several years. Today, we are open sourcing major parts as project Tetragon and open it up for collaboration with the entire community.
What is Tetragon?
Tetragon provides eBPF-based transparent security observability combined with real-time runtime enforcement. The deep visibility is achieved without requiring application changes and is provided at low overhead thanks to smart in-kernel filtering and aggregation logic built directly into the eBPF-based kernel-level collector. The embedded runtime enforcement layer is capable of performing access control on the system call and other enforcement levels.
The foundation of Tetragon is a powerful observability layer that can introspect the entire system ranging from low-level kernel visibility to track file accesses, network activity, or capability changes, all the way up into the application layers covering aspects such as function calls into vulnerable libraries, tracing process execution, or understanding HTTP requests made. To list just a few of the capabilities, Tetragon can provide visibility into all kinds of kernel subsystems to cover namespace escapes, capability and privilege escalations, file system and data access, networking activity of protocols such as HTTP, DNS, TLS, and TCP, as well as the system call layer to audit system call invocation and follow process execution.
- Deep observability: Extensive visibility into all parts of the system and applications ranging from detecting low-level microbursts in TCP connections, providing HTTP visibility for golden signal dashboards, or the ability to detect the use of particular vulnerable shared libraries. The possibilities of eBPF are massive and Tetragon provides an easy to use framework to cover additional visibility use cases.
- Transparent: No application code changes are needed. All observability data is collected transparently from within the kernel. Applications cannot detect when they are being monitored which is ideal for security use cases.
- Low-Overhead: Minimal overhead is imposed on the system. Performing filtering, aggregation, metric accounting, and histogram collection directly in the kernel with eBPF helps to reduce the overhead. Tetragon uses efficient data structures such as per-CPU hash tables, ring buffers, and LRU maps to provide efficient and fast means of data collection and avoids sending vast amounts of low-signal events to the user space agent.
Building on the rich observability, Tetragon provides real-time runtime enforcement. Unlike other systems which have a limited set of enforcement points such as only at the system call level, Tetragon is able to enforce security policies across the operating system in a preventive manner instead of reacting to events asynchronously. Tetragon has the ability to specify allow lists for access control at several layers. Security policies can be injected via Kubernetes (CRDs), a JSON API, or systems such as Open Policy Agent (OPA).
- Preventive security: Enforcement of runtime policies happens in-kernel and synchronously (real-time) allowing to prevent attacks instead of reacting to them asynchronously.
- Pluggable policy architecture: Security policies can come from a variety of sources. Either from the user directly, from systems like Open Policy Agent (OPA), or from third party components via the extendable Kubernetes CRD and JSON interfaces.
Observability and runtime enforcement solutions have traditionally been based on one of the following concepts. Each of them have strengths but also come with weaknesses. Tetragon uses eBPF to bring a wide combination of strengths and gets rid of most weaknesses.
|App instrumentation uses code dependency to gain visibility into the application.
LD_PRELOAD loads a library without awareness of the app to intercept syscalls.
ptrace(2) is a debugging interface provided by the kernel to trace processes and syscalls.
– Good application visibility
– More transparent than app instrumentation
– Even more transparent
– Not transparent, requires application changes
– Can be bypassed
– Not usable for enforcement
– No visibility into the system
– Can be bypassed (static linking)
– Weaker visibility
– Not ideal for enforcement as application can bypass with static linking
– High overhead
– No synchronous enforcement
– Application can detect it is being monitored
– Visibility limited to syscalls
These above solutions operate on the application and system call level and describe methods that are primarily used to provide observability. They are sometimes used in combination with asynchronous enforcement where an agent relies on visibility data collected as described and then reacts to it. They do not provide any visibility into what is happening at the kernel level itself.
|SELinux / LSM
|seccomp(-bpf) allows filtering of system calls
|SELinux and LSM are built-in kernel security frameworks to enforce access control
|Linux kernel modules can be used to implement custom logic for visibility and enforcement
– Transparent to application
– Transparent to application
– Not vulnerable to TOCTTOU attacks
– Transparent to application
– Extendable without kernel changes
– Enforcement only at syscall level
– No deep argument inspection
– No visibility
– No/limited awareness of containers/Kuberentes
– Not extendable
– Need to understand attack vectors
– Many environments do not allow loading kernel modules
– May compromise the stability of the kernel (a bug in the module may crash the kernel)
– Challenging to upgrade without temporarily turning off security protection
This second category involves solutions that directly operate at the kernel level and are primarily targeting enforcement use cases. The built-in kernel systems provide excellent enforcement options but are built with a strong focus of access control as use case. The major drawback of built-in kernel solutions is that they are challenging to extend, e.g. the kernel is not aware of what Kubernetes actually is. New kernel subsystems like LSM-eBPF are very promising and powerful but require recent kernels (≥5.7). Kernel modules solve the extendability problem but are often not an option in many environments due to the security risk they create.
Tetragon, with the help of eBPF, combines most of the strengths from existing solutions:
- Efficiency & Transparency: The application visibility and efficiency of app instrumentation but transparently and without requiring changes to application code.
- Synchronous Enforcement: The in-kernel synchronous enforcement capabilities of seccomp, SELinux, and LSM but with an elevation of the policy level from pure access control to the ability to prevent harm to the systems and components instead of exclusively focusing on restricting access to resources and data.
- Deep Extendable Visibility: The deep system observability capabilities and extendability of custom Linux kernel modules but without the security and availability risks.
In addition, Tetragon provides a modern agent that can natively integrate with modern observability and policy standards such as Kubernetes, Prometheus, fluentd, OpenTelemetry, Open Policy Agent, and also traditional SIEM platforms.
App Behavior Inspection with the Tetragon CLI
The Tetragon CLI, codenamed “amazing-cli” at Isovalent, will likely be your first entry point into observability with Tetragon. It’s the type of CLI that frequently gives goosebumps as it exposes so much information about complex systems in a simple, easy to understand form.
In the following example, we are showing how Tetragon is used to observe the behavior of a Kubernetes pod that runs
curl -L github.com:
curl -L github.comis started.
- We see the DNS resolution for github.com and the opening of a TCP connection to port 80.
- github.com returns HTTP code 301 within 7ms to redirect traffic to port 443. This is done to enable TLS. curl follows the redirect and performs another DNS resolution.
- A TCP connection port 443 is opened. The TLS handshake is observed. TLS 1.3 is used and the cipher negotiated is AES-128-GCM-SHA256.
- Data is exchanged, a total of ~80 bytes on port 80, 218KiB on port 443.
- The process exits with error code 0.
Combining Network & Runtime Visibility
An exciting aspect of Tetragon’s use of eBPF is that it can combine aspects of visibility that have often be treated separately so far. Here is an example of combined network and runtime visibility to demonstrate the ability to identify what processes are involved in what type of network communication. The following example shows the use of Tetragon to observe a Kubernetes pod that was compromised and subject to a lateral movement attack:
In the above diagram we see a classical lateral movement attack via a reverse shell:
- A Kubernetes pod “crawler-c57f9778c-wtcbc” is running in the Kubernetes namespace “tenant-jobs”. The pod is run via containerd which is running as a sub process of the PID 1 init process. The binary running inside of the pod is called “crawler” which in turn spawns a node process executing “server.js”.
- The node app makes egress network connections to “api.twitter.com” as well as a Kubernetes service elasticsearch.
- 5 Minutes after the pod was started, another sub process invoking netcat (
nc) was started. Seeing runtime and network observability combined, it is obvious that this is an ongoing reverse shell attack.
- The attacker can then be observed to be running
curlto reach out to the internal elasticsearch server and then use
curlto upload the retrieved data to an S3 bucket.
Monitoring Access to Sensitive Files
Tetragon has the ability to monitor file and data access. This following examples illustrates integration with Splunk to track access to sensitive files while providing context of the access such as Kubernetes metadata, container image, binary, and user information:
The above example lists accesses to
/etcd/passwd including the process as well as container image and Kubernetes namespace context. Besides monitoring of
/etc/shadow, other obvious files to monitor on the system would be the UNIX domain socket of the container runtime or any systemd unit or init files that could alter the bootstrapping of the machine.
With Tetragon’s enforcement capability, accesses to such sensitive files can not only be monitored but also prevented.
Detection of Weak TLS Ciphers & Versions
TLS is a cornerstone of security today but usage of older TLS versions or insecure TLS ciphers can pose a serious security threat. This can range from unintentional wrong usage to intentional TLS downgrade attacks.
he above dashboard shows the distribution of TLS protocol versions and ties it to the Kubernetes pod and namespace context. It can also show the distribution of ciphers and even more importantly the cipher key length.
Runtime Aware Network Policy
You are possibly familiar with Kubernetes NetworkPolicies that define the allowed and denied network communication for Kubernetes workloads. Oversimplified, these policies describe that pod A is allowed to talk to pod B or CIDR
10.0.0.0/8 but pod A is not allowed to talk to pod C or CIDR
The granularity of these policies are at the level of a pod. It doesn’t matter whether it is
app.js running in the pod creating the request or the
attack.py script invoking
curl. With Tetragon technology, we will be extending Cilium’s Network Policy capabilities to include runtime context:
- name: Frontend
- name: Backend
- binary: app.py
The above policy leads to a significantly better least privilege policy. It allows
Frontend pods to talk to
Backend pods but only if:
- the binary in the source pod is called
- the process in the source pod is running unprivileged
The example shown takes the binary name and privileged execution context into account but this concept can be extended to also restrict based on additional parameters such as ensuring that the process is still namespaced, the UID/GID context, or even taking memory hashes of the executable into account.
We are very excited about Tetragon and bringing it to the open source community. The additional observability and security control will improve the life of security, platform, and application teams and it will lead to more secure infrastructure overall, in particular Kubernetes environments. Even more importantly, we are in particular excited about what is yet to come as collaboration expands to the open source community.
If you are interested to learn more about Tetragon, we’ll be hosting a webinar about Tetragon in the coming weeks. It will feature an introduction, demos, and the opportunity to ask questions. In the meantime, join
#tetragon on the Cilium Slack to get started and meet the team.
If you are interested in Isovalent Tetragon Enterprise, feel free to contact us.
- cilium/tetragon on GitHub
- Sign up for the Tetragon Deep Dive
- Learn more about Isovalent Cilium Enterprise
- Detecting and Blocking log4shell with Isovalent Cilium Enterprise
- Detecting a Container Escape with Cilium and eBPF
- Isovalent Cilium Enterprise 1.10: Timescape, Runtime Observability & Enforcement, Hubble RBAC