Isovalent Enterprise for Tetragon 1.14: Persistent Enforcement, Memory Optimizations, Improved Child Process Visibility, and more!

After a summer mixed with lots of coding contributions, adventures, and even in-person conferences – we are excited to announce the release of Isovalent Enterprise for Tetragon 1.14, delivering a suite of new features that stabilizes and enhances the standard for eBPF-based runtime security and observability. 

This release builds on the strong visibility and enforcement foundation of Tetragon, introducing persistent policy enforcement, improved child process tracking, memory-efficient file integrity monitoring, and more.

Looking ahead to opportunities to connect in-person, Cilium Tetragon is getting lots of buzz at Kubecon NA. We are delighted to see the incredible number of upcoming Tetragon-related talks from Liz, Natalia, Mahé, Anna, John, Kornilios, and many other amazing folks behind the Tetragon project. If you want to learn more about Tetragon before then, feel free to reach out to us directly or over slack.

What is new in Tetragon 1.2 & Isovalent Enterprise for Tetragon 1.14?

• Persistent Enforcement: keep enforcement policies running even when the agent is down (more details)
• Improved Child Process Visibility: follow every child process of a particular binary, the foundation for a variety of core security use cases (more details)
• File Integrity Monitoring (FIM): restrict file access based on filesystem type or binary name, for more efficient memory management (more details)
• Customizable Default Rulesets: 1.13 introduced default rulesets for six different core security categories, 1.14 enables selectively installing these policy groups (more details)
• Map Linux UID to Username: automatically resolves the UID of a process to its corresponding username in process events (more details)
• Systemd Deployment on Kubernetes: install Tetragon on any Kubernetes nodes as a systemd managed service (more details)
• Day 2 Monitoring: high-level Grafana dashboard to monitor the Tetragon agent health and resource (CPU & Memory) consumption (more details)
• Threat Modeling Guide: in-depth threat model for Tetragon security implications, design considerations, and recommended controls (more details)
• Containerd Support for Runtime Hooks: Tetragon runtime hooks are now configured via DaemonSet instead of init container and support NRI (more details)
• OpenShift Support: improved support for testing and CI infrastructure workflows (more details)
• CRI-O Support When Deployed with crun: improved association of pod metadata using cgroup ID’s (more details)

Note on versioning: Isovalent Enterprise for Tetragon is in sync with the OSS version, but tagged on a different step-up number.

Persistent Enforcement

Feature status: Limited | Open Source

Until now, enforcement policies applied by Tetragon were dependent on agent uptime. If the Tetragon agent was restarted or down, the associated eBPF enforcement programs would be removed, resulting in a potential security lapse. 

Now, protection runs 24x7x365. Tetragon allows for persistent enforcement to ensure enforcement policies remain alive, even if the agent becomes unavailable.

When a policy is applied, the eBPF program is loaded directly into the kernel and continues to monitor and enforce rules, such as blocking unauthorized process executions or restricting network access, without the need for the agent to be continuously active (though because the agent is down, you may not generate JSON events if the policy is triggered during this time). 

This persistent behavior significantly minimizes the risk window during agent downtimes and hardens the overall security posture. 

To enable this feature, users simply configure the enforcement policy YAML with the following argument.

tetragon:
  extraArgs:
    "keep-sensors-on-exit": true

Improved Child Process Visibility

Feature status: Stable | Open Source

Monitoring the activity of all child processes spawned by a specific binary is critical for auditing and security purposes. The open source 1.2 release brings a simplified mechanism to track process lineage across complex execution chains using the matchBinaries selector and the followChildren attribute. 

From a security perspective, this feature is particularly useful for monitoring SSH sessions or tracking kubectl exec commands, where a parent process may spawn multiple forks performing various security-significant actions. For example, when an SSH session spawns several processes, Tetragon tracks each branch, allowing you to see exactly what operations were performed during the session.

Previously, users were tasked with using matchPID and followFork to build tracing policies around process ancestries (as seen in the example below). This over relies on knowledge of the parent process and creates downstream challenges for tracing follow-on processes.  

The matchBinaries selector and followChildren attribute address the challenge of needing to know the PID of a process before it launches. Accurately predicting the PID for any process, other than the initial process that launched the container, is difficult since this PID can be any numerical value.

- matchPIDs:
  - operator: "In"
    followForks: true
    values:
    - "<sshd_pid>"

Let’s simplify this process with the matchBinaries selector and the
followChildren attribute (shown below). This is the foundation to build more complex policies. For example, teams can easily build a policy to extract the binaries and monitor all processes after a kubectl exec to audit any privileged pod access. 

- matchBinaries:
  - operator: "In"
    values:
    - "/usr/sbin/sshd"
    followChildren: true

Without visibility into child processes, malicious activities like process injection or lateral movement can go unnoticed. This added context helps security teams in identifying and understanding process behaviors that could indicate security threats or policy violations.

Isovalent Enterprise for Cilium: Security Visibility Lab

Simulate the exploitation of a nodejs application, with the attacker spawning a reverse shell inside of a container and moving laterally within the Kubernetes environment.

Start Lab

File Integrity Monitoring (FIM)

Feature status: Stable | Enterprise

Already a stable feature of the Tetragon arsenal, FIM gets additional tools to further reduce unnecessary memory consumption. 

These updates introduce a more granular approach to file access control by allowing policies to be defined based on filesystem types and specific binaries. 

Memory optimizations with FileSystemType

Let’s dive into how switching to FileSystem Types improves memory consumption, with some benchmarks shared as well. 

File policies keep track of all existing files under a specified file path. This is a powerful mechanism that allows monitoring file access no matter how it happens (via hard or soft links or different mount points) , but as the number of files grows larger, the memory burden will not scale optimally. For example, the /proc/ filesystem may have ~10,000 files under it and policies for monitoring everything under it require metadata for every single file in it. 

Tetragon 1.14, introduces a new pattern type (FileSystemType) for monitoring files based on the filesystem type they reside in. Using this new feature, users can write policies for monitoring files that reside under, for example, the /proc filesystem, with minimal memory overhead.

This means that policies such as:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-monitoring"
spec:
  file:
	file_paths:
	- "/proc/"
	- "/sys"
	monitorHostFiles: true
	podSelector: {}

Can now be replaced with:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-system-monitoring"
spec:
  file:
    file_paths_patterns:
    - type: FileSystemType
      file_system_type: # match any file or directory operation under sysfs or proc file systems
        names: 
        - "sysfs"
        - "proc"

To demonstrate this improvement, we performed a simple benchmark of:

1. Tetragon baseline without a FIM policy,
2. with the  <1.13  type: PathPrefix attribute, 
3. and the 1.14 type: FileSystemType attribute 

Summary of chart:

(1) Tetragon without any tracing policies: 115Mi bytes

(2) Tetragon with the traditional way monitoring /proc and /sys: 590Mi bytes

(3) Tetragon with the new fs type matcher for /proc and /sysfs: 226Mi bytes

The 1.14 policy significantly reduced memory utilization by over 89%. We expect when increasing the number of pods that (2) will also increase, but (3) will remain stable. 

File monitoring with matchBinaries 

When looking at file monitoring based on binary, there are two primary use cases that have emerged. One around restricting access, and one around observing access. 

(Use case 1) Implement a flexible least privileged policy that scopes file access to a Kubernetes workload and only allows specific permissions needed to perform its task, and deny everything else. 

For example, showing the flexibility of a single policy with matchBinaries, we have a binary (/usr/bin/nginx) that under a single policy defines:

1. read /etc/passwd and /etc/shadow
2. read/write /etc/nginx.conf
3. execute /usr/bin/vi 
4. Outside of this defined set of permissions, block all other file operations 

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "nginx-file-accesses"
spec:
  file:
    file_paths_patterns:
    - type: AllFileOps
    selectors:
    # selector (1) that allows /usr/bin/nginx to read /etc/passwd and /etc/shadow
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/nginx"
      matchFilename:
      - operator: "Equals"
        values:
        - "/etc/passwd"
        - "/etc/shadow"
      matchOperations:
      - operator: "In"
        values:
        - "FILE_READ"
      matchActions:
      - action: NoPost # we can also post here if we care about these events
    # selector (2) that allows /usr/bin/nginx to read/write /etc/nginx.conf
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/nginx"
      matchFilename:
      - operator: "Equals"
        values:
        - "/etc/nginx.conf"
      matchOperations:
      - operator: "In"
        values:
        - "FILE_READ"
        - "FILE_WRITE"
      matchActions:
      - action: NoPost # we can also post here if we care about these events
    # selector (3) that allows /usr/bin/nginx to execute /usr/bin/foo
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/nginx"
      matchFilename:
      - operator: "Equals"
        values:
        - "/usr/bin/vi"
      matchOperations:
      - operator: "In"
        values:
        - "FILE_EXEC"
      matchActions:
      - action: NoPost # we can also post here if we care about these events
    # selector (4) that blocks all other file operations from /usr/bin/nginx
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/nginx"
      matchActions:
      - action: Block

(Use case 2) Discover what files or directories specific Kubernetes workloads access. 

The example policy below sets up file access tracing for the Nginx process within a Kubernetes cluster. It captures all file operations (AllFileOps) performed by the Nginx binary (/usr/bin/nginx), providing an event trail of what files or directories this workload interacts with. 

This data can then be used to build default security profiles for commonly used Docker images (redis, nginx, node, postgres, mongo, etc), restricting file access to only necessary paths. 

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "discover-file-accesses-from-nginx"
spec:
  file:
    file_paths_patterns:
    - type: AllFileOps
    selectors:
    - matchBinaries:
      - operator: "In"
        values:
        - "/usr/bin/nginx"

File Monitoring with eBPF and Tetragon (Part 1)

See the inode architecture and design decisions behind file monitoring and enforcement using eBPF & Tetragon.

Read the FIM blog

Customizable Default Rulesets

Feature status: Limited | Enterprise

Default rulesets are an amazing out-of-the-box baseline to get that clear picture of your security baseline, in a matter of minutes teams collect and visualize security events across six core areas: 

• Monitoring Runtime Executions 
• Network Observability
• File Integrity Monitoring
• Operating System Integrity
• Container Sandboxing
• Security Sensitive Events

While all of these categories are interesting, not all are relevant to every environment. 

For that reason, Isovalent Enterprise for Tetragon 1.14 gives customers the flexibility to selectively install and customize default rulesets tailored to specific security requirements, allowing teams to fine-tune their runtime security policies without unnecessary overhead.

By customizing rulesets, you can focus on what’s necessary for your setup, whether that’s minimizing file risk in a production environment or monitoring specific TCP/UDP network events in a development sandbox. 

These rulesets are all installed via Helm, making it as easy as kubectl apply.

Map Linux UID to Username

Feature status: Limited | Enterprise

Tetragon thrives at providing clear context around security-significant events, using Kubernetes and Linux metadata to tell the full story of who (machine or human) is performing what actions. 

One use case that teams have been coming to Isovalent for is to associate which username (root, daemon, sys, etc.) belongs to which Linux UID (specific integer). 

Process-related events are crucial for understanding system activities, but mapping UIDs to usernames has traditionally required additional context retrieval. Tetragon 1.14 simplifies this by automatically resolving the UID of a process to its corresponding username from the user database in /etc/passwd. 

Without immediate user context, security teams spend additional cycles correlating UIDs to usernames to events, delaying incident response. By tying this information together in real-time, Tetragon pushes faster audits and more effective root cause analysis, reducing the time required to respond to potential security incidents.

For example, when auditing process executions; see at a glance which user initiated a potentially risky process, such as an unexpected SSH connection or file read request on a production server. This feature is currently supported on virtual machines, and is enabled by default. 

Systemd Deployment on Kubernetes

Feature status: Limited | Enterprise

As environments grow in complexity, the tools we use to secure them make operations easier, not harder. In some cases it’s essential to have one standard installation approach for the entire fleet. This approach offers a path for unified installation and management experience across environments. 

Isovalent has added systemd managed deployments, allowing the Tetragon 1.14 agent to be deployed uniformly with systemd on Kubernetes nodes. 

This is also useful for teams with restrictions on how they manage distributed agents or images. For example, maybe one team is only approved to run services via systemd versus as a DaemonSet.

The flexibility to manage all agents via systemd has already been seen to streamline the operational process, reduce configuration drift, and facilitate quick troubleshooting across these increasingly hybrid deployment scenarios. 

Day 2 Monitoring

Feature status: Limited | Enterprise

Who watches the watcher? Operational health is critical for maintaining a secure and performant environment, and now the Isovalent platform includes out-of-the-box dashboards to monitor key health metrics. 

Collect and visualize the agent’s health and resource consumption, across memory usage, CPU load, eBPF program execution counts, and more health metrics. 

There are multiple dashboards created to investigate questions such as where is the memory going in the Tetragon agent? Is it some user space cache (ProcessCache, DNSCache, etc.)? Is it eBPF maps?

Or even what causes the CPU spike in the Tetragon agent? If there is a spike, which eBPF sensor is responsible for it? Is it DNS? (it’s always DNS!) I/O file activity?

This first dashboard tracks utilization of the eBPF maps, which act as data stores for Tetragon and facilitates event sharing between eBPF programs and user-space.

High values here can indicate overutilization of eBPF maps and may lead to performance degradation. This graph calculates total utilization across all Tetragon nodes, here showing a stable and low usage rate with no errors.

In this second dashboard, we simplify the current status of our agent health. This gives teams a clear picture of Tetragon agent uptime. In this example, we see nine agents running smoothly, alongside one operator. 

At scale, this gives platform and security teams the high level understanding of health indicators for day two operations. 

In addition to health monitoring, Isovalent also provides out-of-the-box network monitoring dashboards to troubleshoot and resolve network issues around DNS, TCP, UDP, and more. 

Isovalent Enterprise for Tetragon: Deeper Host Network Observability with eBPF

Visualize network data with out-of-the-box dashboards, see this blog covering TCP, UDP, DNS events with amazing Grafana dashboards.

Read Blog

Threat Modeling Guide

Feature status: N/A | Enterprise

Good security looks inward as well. Part of the Isovalent platform includes a detailed threat modeling guide which outlines the security implications, design considerations, and recommended controls for Tetragon. 

A living document that stands outside of a ‘release’, we wanted to highlight this added piece of a holistic approach to security. 

The threat modeling guide covers various risk vectors, including different attacker profiles focused around privilege escalation, data tampering, network attacks, and more providing a comprehensive security assessment framework.

This threat model serves as a reference point for security teams to evaluate secure deployment in their environments, guiding appropriate mitigations and security controls. The approach centers around how different attacker profiles might try to enter and exploit a system, for example here are a sample of five threat actors covered: 

• Kubernetes workload attacker
• Limited privilege host attacker
• Root-equivalent host attacker
• Kubernetes API server attacker
• Network attacker

This gives each team a lens to approach where risks and technical gaps exist in a system, and harden environments depending on what distinct parts of the infrastructure they control and monitor. 

CRI-O Support

Feature status: Limited | Open Source

Tetragon received a bug fix related to missing Kubernetes Pod Information in process events when using CRI-O configured with crun as the low-level runtime. This improves the mechanism for associating Kubernetes metadata for processes. 

Instead of using cgroup names for mapping metadata to processes, the new approach deployed by tetragon now uses cgroup IDs. This provides a more reliable source for metadata association by directly talking to the CRI-O runtime, versus the standard cgroup name of “container” which doesn’t allow for the necessary level of pod identity. 

Ultimately, this bug fix improves Kubernetes pod association in CRI-O, which (like the improvements on containerd below) are the foundation for any Kubernetes runtime security use case.  

OpenShift Support

Feature status: Limited | Enterprise

OpenShift support was announced previously during the 1.13 release! Since then, we have seen OpenShift deployments rapidly growing, and with that comes their own unique requirements and configurations. 

Tetragon 1.14 enhances support for OpenShift with improved testing, CI infrastructure workflows, and a dedicated installation guide. The aim is to help teams get into production faster, and see value in minutes not days. 

This suite of features includes E2E tests, automated CI testing features, and building OLM images via a Github workflow tailored to OpenShift’s model and platform specifications.

Containerd Support

Feature status: Limited | Open Source

Tetragon runtime hooks now support the containerd runtime using a new “tetragon-rthooks” DaemonSet. 

Touched on briefly in the last release, Tetragon used an init container for configuring runtime hooks for CRI-O. This approach, however, was not suitable for containerd which uses the Node Resource Interface (NRI). As a result, it requires a process constantly running to configure hooks and therefore which is why we opted to utilized a new DaemonSet-based approach. The new approach supports both containerd and CRI-O, and has the added benefit that the Tetragon runtime hooks are uninstalled when the tetragon-rthoks DaemonSet is uninstalled. 

Tetragon utilizes runtime hooks to create the eBPF programs and maps implementing enforcement and observability  before the containers start executing, to ensure that the K8s Identity Aware Policies are applied correctly. To do this, Tetragon needs information about whether the policy applies to a certain K8s namespace or pods. 

These runtime hooks are the foundation for identity aware policies to function, and is one of the fundamental pieces giving Tetragon the identity-awareness at container start up to immediately protect specific namespaces or pods from sensitive system calls, specified CVE’s, and a plethora of other runtime use cases. 

Learn More!

If you’d like to learn more about Isovalent Enterprise for Tetragon 1.14 and related topics, check out the following links:

• Try Tetragon in a hands-on lab, earn a badge and progress down the Security Engineer Track. 
• Request a Demo – Schedule a demo session with an Isovalent Solution Architect.
• Learn more about Isovalent & Cilium at our resource library – It lists guides, tutorials, and interactive labs.
• Get the Security & Compliance White Paper – the blueprint for Kubernetes compliance. 
• Extract security data no one else can with the host-based Kubernetes visibility solution brief.

Feature Status

Here is a brief definition of the feature maturity levels used in this blog post:

• Stable: A feature that is appropriate for production use in a variety of supported configurations due to significant hardening from testing and use.
• Limited: A feature that is appropriate for production use only in specific scenarios and in close consultation with the Isovalent team.
• Beta: A feature that is not appropriate for production use, but where user testing and feedback is requested. Customers should contact Isovalent support before considering Beta features.