Isovalent Enterprise for Tetragon 1.13: Kubernetes Identity Aware Policies, Default Rulesets, HTTP and TLS Visibility, and More!
We are excited to announce the arrival of Isovalent Enterprise for Tetragon 1.13, advancing a host of features simplifying and hardening runtime security.
Since the 1.0 release of Cilium Tetragon, the Cilium Tetragon open source project and Isovalent’s enterprise features have continued to make large strides. KubeCon Europe brought together the cloud native community from all over, sharing use cases, production deployments, and learnings from pushing the limits of eBPF & Tetragon.
Tetragon 1.13 brings exciting new and improving features to dive into with the expansion of default “out-of-the-box” runtime security policies, granular identity awareness for applying policies to specific Kubernetes pods and namespaces, redaction filters for sensitive data, inode-based file integrity monitoring, HTTP & TLS visibility, and more that we dive into below.
Isovalent Enterprise for Tetragon is the hardened, enterprise grade version of the eBPF-based Tetragon runtime security tool.
What is new in Tetragon & Isovalent Enterprise for Tetragon 1.13?
- Kubernetes Identity Aware Policies: improved in-kernel pod label and namespace filtering for granular policy application (more details)
- Tetragon Default Ruleset: promoted to stable feature status, default rules cover six different categories of core security and platform monitoring use cases (more details)
- Sandbox Policies: high level runtime security policies are getting a big leap forward with sandbox policies, flexible and ready-to-apply templates for system call visibility or enforcement (more details)
- Redaction filters: prevent sensitive data from being exfiltrated via Tetragon JSON export (more details)
- File Integrity Monitoring (FIM): FIM has reached stable feature maturity, with some additional improvements worth diving into (more details)
- Privilege Escalation and Kernel CVEs Protection: detect and prevent abuse of unprivileged user namespaces with both known or unknown CVEs (more details)
- HTTP Visibility: parse HTTP traffic, with data about the request, response, latency, and more (more details)
- TLS Visibility: monitor TLS versions, weak TLS ciphers, and malicious SNI names (more details)
- OpenShift Support: improved support for Tetragon on RedHat OpenShift deployments (more details)
- Labels for Prometheus metrics: support for flexible configuration options for Prometheus metrics (more details)
- CRI-O Support: improved support for Tetragon runtime hooks for CRI-O container runtime (more details)
Note on versioning: Wait, 1.13? I thought Cilium Tetragon 1.0 just released in the fall? Correct, the latest open source release of Cilium Tetragon is now at 1.1, after the previous 1.0 release. Isovalent Enterprise for Tetragon is in sync with the OSS version, but tagged as version 1.13.
Cilium Tetragon (OSS) | Tetragon (Isovalent Enterprise) |
1.0 | 1.12 |
1.1 | 1.13 |
Kubernetes identity aware policies
Feature status: Limited | Open Source
Kubernetes container orchestration manages and deploy increasingly granular workloads from clusters, nodes, pods, down to the containerized apps themselves. From a security and platform perspective, the more granular our monitoring tools, the more control we have to observe, detect, and enforce on security-significant events.
Prior to 1.13, Tetragon’s security policies were applied with a clusterwide
scope across Kubernetes workloads. Now, you can tailor policies for discrete subsets of your workloads by applying them to specific pods or namespaces. This improved K8s identity awareness allows stricter enforcement or policy exceptions for specific workloads, specifying policy applications with magnified granularity.
Notably, a positive downstream effect of this improved identity awareness is filtering out unnecessary security events generated by policies on previously out-of-scope workloads. By operating in the kernel, eBPF programs can filter and aggregate events so only security significant events for the policies target workloads are generated to user space. Greatly reducing the volume of events with in-kernel filtering leads to less alert noise and even lower CPU & memory overhead.
In the below example policy, we specify within the tracing policy the exact pod label that will be monitored. In the section matchLabels: app:"nginx"
we have specified that only pods with the "nginx"
label are within scope of this policy that monitors every “sys_pivot_root()”
system call that executes.
The sys_pivot_root()
system call allows an attacker to re-mount the root filesystem to a non-root location, while simultaneously mounting something back on the root. Since this way the attacker can get access to the initial root directory containing trusted or sensitive files or hard links, or mount a malicious directory back to the root destination, it is crucial for security teams to monitor which processes or users have invoked that particular system call.
Similar examples and more, scoped to namespaces or containers, can be explored in the Tetragon docs.
CRI-O runtime support
Feature status: Limited | Open Source
One way that Kubernetes metadata (labels, namespaces, etc) information is available to the Tetragon agent is via the K8s API sever. Relying on the API server, however, can lead to a delay before the container starts and the policy is applied. This might be undesirable, especially for enforcement policies.
OCI (Open Container Initiative) hooks are instrumental for Tetragon to digest Kubernetes-aware metadata, using this information to understand (before the container starts) which security policies are applicable for which running pods and namespaces. In 1.13, the CRI-O container runtime now automatically configures runtime hooks via a Tetragon init container.
Isovalent Enterprise for Cilium: Security Visibility Lab
Simulate the exploitation of a nodejs application, with the attacker spawning a reverse shell inside of a container and moving laterally within the Kubernetes environment.
Start LabTetragon default ruleset
Feature status: Stable | Isovalent Enterprise
Kubernetes comes with limited out-of-the-box security or observability support. The problem of solving security, compliance, or health monitoring burdens are placed on us as platform owners. This often makes it difficult to easily get a strong, reliable baseline in the security posture and health of our K8s runtime. Tetragon’s default ruleset fills this gap, making it easy to deploy and understand your security posture without the heavy lifting.
Tetragon’s default ruleset provides teams with quick, out-of-the-box monitoring for the top core security use cases. These are all built for extensive in-kernel filtering, to only monitor noteworthy events and keep trivially low overhead. Let’s briefly walk through each of the use cases.
- Monitoring Runtime Executions
- Network Observability
- File Integrity Monitoring
- Operating System Integrity
- Container Sandboxing
- Security Sensitive Events
Monitoring Runtime Executions – One of the first things a new user might do is install Tetragon and immediately start seeing the processes executing on their system.
A core functionality of Tetragon, this default ruleset immediately monitors all the executed binaries on a node or specific Kubernetes workload. This sets the foundation for most other security significant events allowing Tetragon to map sockets, files, syscalls, and more back to the process.
Network Observability – Out-of-the-box monitoring for TCP and UDP events, providing the capability to conduct passive network analysis. For example, audit all the listening ports and sockets from each process directly from eBPF events, rather than running an active nmap or other port scan.
From a security perspective, we can quickly answer questions and provide evidence about all sockets opening on a network connection, all processes accepting incoming connections, and all processes listening on a socket.
Building on these default rulesets, Tetragon provides rich network data to monitor the health of your workloads. In the below screenshot that visualizes network health in real time, we see TCP throughput traffic by Bytes and Segments sent and received. See more stunning dashboards and the variety of metadata captured in deeper host network observability with eBPF, with transmit and receive data by Kubernetes workload, node, binary, or Remote DNS name.
File Integrity Monitoring – The below section and recent Tetragon FIM blog expand on the architecture and features of Tetragon’s FIM model in Isovalent Enterprise. Out of the box, this default ruleset provides a baseline for teams to monitor sensitive files (ex: modifications to /etc), auditing for file changes the user, the process, and the parent processes that led to that file activity.
This allows teams to easily build out process ancestry trees and provide attestation for compliance requirements around file monitoring (commonly seen in PCI-DSS, NIST, HIPAA, and more). To learn more about applying Tetragon to compliance use cases, read the joint whitepaper with our friends at ControlPlane, “Mastering Cilium for Kubernetes Compliance”.
Operating System Integrity – Monitor the integrity of the Linux operating system, including the eBPF subsystem. This ruleset will monitor interactions detecting eBPF program loads, eBPF map creations, and kernel module loads.
Container Sandboxing—This default Container Sandbox rule monitors for a set of container events that may indicate a binary attempting to escalate privileges, gain root access, change system users and credentials, or take a security significant action performing lateral movement.
Security Sensitive Events – Monitor specific security events in order to identify typical operations that may affect the security posture of your infrastructure. This includes detecting anonymous binary execution, deleted binary execution, privileged execution and other various privilege-escalation operations (ie setuid
, sudo
, and more).
Correlate process-to-network data, with host-based Kubernetes visibility.
Decode DNS, TLS, HTTP, UDP, TCP, and more while matching to process ancestry information, all with Kubernetes identity-aware metadata (labels, pod names, etc).
Get the BriefSandbox policies
Feature status: Limited | Isovalent Enterprise
Sandbox policies offer simplified and high-level auditing and enforcement of system calls. Instead of writing a net new policy for each set of workloads and accompanying system calls to monitor, these high-level policies use a list of system calls as a denylist
or allowlist
to easily apply.
Essentially, think of this as an improved K8s Identity Aware Seccomp that ingests a list of syscalls as part of a predefined policy.
With sandbox policies, Tetragon acts as this filtering mechanism at the level of system calls, restricting the system calls your workloads can make. Tetragon has distinct advantages over Seccomp being 1) dynamic and 2) allowing for flexible in-kernel selectors. Dynamically, Tetragon does not need to be restarted for the policy to take effect or wait for a pod to reach the ready state., Flexible in-kernel filtering is powerful with selectors based on K8s Identity Aware pod label/namespace filtering, process binaries, hashes, arguments, pids, capabilities, and more metadata.
Within sandbox policies there are three flexible options for the matchaction
, teams can:
Post
: generate an event of typeprocess_sandbox_syscall
to audit system calls by generating events when the syscall is observed,Block
: do not execute the system call; instead, return an error (specifically,-EPERM
) to the application.Signal
: send a signal (specifically,SIGKILL
) to the application to terminate the syscall.
In this snippet of an example sandbox policy, we set up a list of allowed syscalls and prevent all other sensitive syscalls outside this list (operator: "NotIn"
) from executing. As seen in the policy, this is not just to audit syscalls, the actions: - action: Signal
means for any observed syscalls outside this list, the policy will send a signal to terminate the disallowed process.
Redaction filters
Feature status: Stable | Open Source
When tracing security-significant events in the system, there is a chance that JSON exports contain sensitive information (passwords, env variables, etc) thus this data gets exfiltrated into a SIEM system. This stable feature prevents secrets from being exported. These filters are written in JSON and passed to the Tetragon agent via the --redaction-filters
command line flag or the redactionFilters
Helm value.
The above filter redacts all password arguments passed to processes with the arg "--password"
. For example, a process is passing the password "--password=admintest"
the filter will replace the password admintest
with “--password=*****"
. This filter can be stacked with a second filter to capture arguments passed with the “--p”
shorthand, therefore capturing all password arguments.
File Integrity Monitoring (FIM)
Feature status: Stable | Isovalent Enterprise
Tetragon’s file integrity monitoring is a stable, highly scalable solution to detect sensitive file access and monitor system file changes at near zero overhead. See the file monitoring benchmarks, Tetragon uses less than 0.5% overhead to monitor high-volume I/O operations.
Isovalent Enterprise hardens FIM by monitoring kernel inodes
instead of file paths. The inode-based approach provides a stronger, more stable reference point for monitoring file access.
Figure: Inode-based approach for FIM as implemented in Isovalent Enterprise for Tetragon.
To monitor file accesses to a file, regardless of the pathname that is used we need a way to track the underlying file object. One way to do so is by using the inode number. Inodes uniquely identify the underlying file within a single filesystem, see more in the blog linked below.
File Monitoring with eBPF and Tetragon (Part 1)
See the inode architecture and design decisions behind file monitoring and enforcement using eBPF & Tetragon.
Read BlogAnother feature in 1.13 is added support for suffix matches, which greatly expands flexibility for scoping FIM policies. We can now monitor files and directories with (1) prefixes and suffixes and (2) do exact matches.
Diving into this yaml policy, let’s see how each of these four example use cases are implemented in Tetragon 1.13.
- (1) This will match only files like “
/home/…/.ssh/authorized_keys
” - (2) This will match only files like “
/root/*
” - (3) This will match files and directories under “
/etc/
” (previous implementation) - (4) This will match only files which have the exact path =
/etc/passwd
Privilege escalation and kernel CVEs protection
Feature status: Beta | Isovalent Enterprise
User namespaces isolate user security properties such as user IDs and group IDs from the ones in the host. They are extensively used in implementing containers (see for example: Kubernetes User Namespaces).
While they are a useful feature for containerization, they also introduce additional risks because the process that creates a new user namespace will receive a complete set of capabilities and privileges in the new user namespace. This exposes a larger attack surface of the kernel that is often abused by attackers.
The Google kCTF report on Linux kernel exploits shared the figure below, showing that 44% of their exploits required userns
as part of their exploit chain. Detecting the creation of user namespaces by unprivileged users or blocking if the action is not part of expected workload activity is a common best-practice, but still an often exploited attack path.
Tetragon 1.13 monitors processes that gain additional privileges during their execution to detect these privilege escalations. This translates deeper into Tetragon monitoring against specific kernel CVE’s that are known to involve user namespace escalation as part of their attack chain. Isovalent has a growing CVE database with policies that protect against known CVEs covering kernel, container runtime and Linux daemon/services exploits.
Notably, Tetragon also monitors CVEs related to container runtimes (i.e, CVE 2024-21626, the widespread runc vulnerability) or Linux daemons and services (i.e, CVE 2024-3094, the recent XZ Utils backdoor when using OpenSSH – see the policy we used to detect the XZ Utils CVE with eBPF & Tetragon). All this comes together to protect multiple layers of our container and Kubernetes infrastructure, from the kernel level, up to the Linux daemon and workload applications, and into the container runtime component.
The JSON output below highlights the elevation of privileges (process_credentials
), the use of a binary with elevated privileges (binary_properties.privileges_changed
), the capabilities granted to the process (cap
), and the specific command executed (arguments
), collectively indicating a privilege escalation attack.
Breaking down this event, we see a clearer picture of a privilege escalation. The JSON output represents the execution of a sudo passwd
command, which is typically used to change the password of the root user.
Process Credentials: The process_credentials
section reveals that the original UID is 1000, denoting a non-privileged user. However, the effective UID is 0 (root), indicating a significant privilege escalation attempt.
Binary Properties and Privilege Change: The binary_properties.privileges_changed
field specifically mentions PRIVILEGES_RAISED_EXEC_FILE_SETUID
, signifying the execution of a binary with the setuid
bit set to root
. This helps confirm the deliberate escalation of privileges.
Capability Information: The cap
section enumerates the capabilities permitted and effectively used by the process. The effective capabilities include those typically only available to the root user, showing the attempt at privilege escalation of the user.
Process Arguments and Command: The arguments
field shows the command-line arguments passed to the binary (sudo
), indicating an attempt to execute passwd
to change the password of root.
HTTP visibility
Feature status: Beta | Isovalent Enterprise
Understanding the performance of HTTP-based traffic is a key requirement for most platform engineers. It’s one of the many use cases for Cilium and Hubble – the platform provides HTTP-based observability. While useful, it does come with the latency cost of forwarding traffic to a proxy as eBPF was not capable of natively processing HTTP traffic.
With Isovalent Enterprise for Tetragon 1.13, we are leveraging eBPF for an exciting new use case: providing insight into HTTP traffic; eliminating the need for a proxy. eBPF performs L7 parsing of HTTP without any support from a local proxy, providing visibility into the request, response, latency, and even the network socket.
After setting up our KIND environment, we have Tetragon running as a DaemonSet. Let’s generate network events and then dig into a HTTP JSON event. Below, we use the Tetra CLI to compact the observed network events for our tiefighter
pod.
Now, let’s open up the last HTTP event and filter on the http
field to understand what information Tetragon captured.
From the request:
timestamp
: Timestamp of the request.
method
: HTTP method (e.g., POST).
uri
: Requested URI.
version
: HTTP version.
host
: Host of the request.
agent
: User agent making the request.
content_length
: Length of the request content.
From the response:
timestamp
: Timestamp of the response.
version
: HTTP version of the response.
code
: HTTP status code.
reason
: Reason phrase for the status code.
eBPF even captures and outputs latency.
latency
: Duration of the HTTP request-response cycle.
TLS visibility
Feature status: Beta | Isovalent Enterprise
Tetragon captures incredibly valuable information about the cipher used during the TLS connection, the TLS version of the client/server (including the negotiated version), all supported versions by server and client, IP address of the TLS event, source and destination IP address and port, and Server Name Indication (SNI).
Security teams use Tetragon to audit applications for weak TLS versions, ciphers, and key lengths while hardening their workloads and proving compliance attestation.
TLS Visibility Lab
Inspect TLS traffic and identify the version of TLS being used. Then export JSON events to a SIEM with Isovalent Enterprise for Tetragon!
Start LabMaster Cilium and Tetragon for Kubernetes Compliance
This guide is tailored for technical compliance stakeholders, security teams, and platform engineering teams looking for a comprehensive solution to effectively manage Kubernetes environments and implement compliance principles using eBPF and Cilium.
Get the White PaperOpenShift Support
Feature status: Limited | Isovalent Enterprise
Deploy Tetragon on OpenShift! 1.13 brings improved operability for OpenShift deployments with Tetragon via the OLM operator.
Using AWS or Azure? Seamlessly install and validate your Tetragon deployment to start enabling DNS/TLS/HTTP visibility, apply tracing policies, implement file monitoring, see processes, and more using this Tetragon deployment walkthrough on Azure.
Labels for Prometheus metrics
Feature status: Beta | Isovalent Enterprise
Support is now included for flexible configuration options for Prometheus metrics. Previously, some alerting and monitoring models resulted in very high cardinality of produced metrics, in particular in Kubernetes environments with high pod churn. Now, users can exclude or include specific metrics labels to control cardinality.
Isovalent Enterprise for Tetragon: Deeper Host Network Observability with eBPF
Visualize network data with out-of-the-box dashboards, see this blog covering TCP, UDP, DNS events with amazing Grafana dashboards.
Read BlogLearn more
If you’d like to learn more about Isovalent Enterprise for Tetragon 1.13 and related topics, check out the following links:
- Try Tetragon in a hands-on lab, earn a badge and progress down the Security Engineer Track.
- Request a Demo – Schedule a demo session with an Isovalent Solution Architect.
- Read more about the Tetragon 1.0 release – benchmarks, early use cases, and core principles behind the project.
- Learn more about Isovalent & Cilium at our resource library – It lists guides, tutorials, and interactive labs.
- Can I use Tetragon without Cilium? Yes! – eBPF-based enforcement, visibility, and forensics for Kubernetes regardless of your CNI.
Feature status
Here is a brief definition of the feature maturity levels used in this blog post:
- Stable: A feature that is appropriate for production use in a variety of supported configurations due to significant hardening from testing and use.
- Limited: A feature that is appropriate for production use only in specific scenarios and in close consultation with the Isovalent team.
- Beta: A feature that is not appropriate for production use, but where user testing and feedback is requested. Customers should contact Isovalent support before considering Beta features.