Isovalent Enterprise for Tetragon 1.15: CPU & Memory Metrics, Audit eBPF Operations, Improved Userspace Filtering!

Join the v1.15 webinar!

Learn technical updates and what's new from Tetragon v1.15 with Natália Réka Ivánkó, Sr. Product Manager Tetragon. See what's exciting for each new feature, practical production deployment scenarios, and answer any open questions!

Register for the v1.15 webinar now

As we step into the new year, the Tetragon team is thrilled to kick off 2025 with several new features. We are very excited for this release of Isovalent Enterprise for Tetragon 1.15, with improvements hardening underlying eBPF operations, pinpointing CPU & Memory usage metrics, simplifying out-of-the-box visibility, and more. 

Tetragon 1.15 brings this improved performance monitoring of individual eBPF programs, enhanced security measures, and advanced filtering options for userspace and kernel events. If you are interested in learning about a specific topic, navigate directly to that feature using the section below. If you are curious about more under the hood details or broader use cases behind a feature, then please feel free to reach out directly. 

What is new in Isovalent Enterprise for Tetragon 1.15?

• Monitoring CPU% of eBPF Programs: display and export CPU usage metrics for individual eBPF programs, improving performance diagnostics (more details)
• Memory Usage Tracking of Programs & Maps: Extends the CLI for detailed analysis of in-kernel memory consumption of individual eBPF maps and programs (more details)
• Audit and Protect BPF Operations: monitor and secure what is interacting with the eBPF subsystem (more details)
• Identify Container init Processes: Adds a new field to identify processes as part of the main container init or not (more details) 
• Improved Userspace Filtering: expanded filtering capabilities using regex and CEL expressions for targeted event management (more details)
• File Monitoring with Distroless Policies: Apply executable restrictions to replicate the behavior of distroless images (more details)
• Digest Support in File Monitoring: Better user experience with SHA256 digest management, simplifying policy enforcement (more details)
• SandboxPolicies Reaches Stable: the high-level Kubernetes identity-aware syscall observability and enforcement policies have reached stable status (more details)
• Aggregated Application JSON Events: Improves memory and cpu overhead by summarizing (vs constant streaming) JSON events for export (more details)
• Improved Default Ruleset Documentation: in-depth and cleaned up ruleset documentation for customers (more details)
• Clean Vulnerability Security Report: comprehensive security assessments from Cure53 with no medium+ CVE’s found (more details)

Monitoring CPU% of eBPF Programs

Understanding resource usage is the key to maintaining performant operations and reducing unnecessary costs. Before 1.15, performance metrics of CPU overhead were captured and stored in the eBPF subsystem, but not being surfaced up through the CLI. Now, use the “tetra” CLI to list eBPF programs along with their CPU usage, allowing the SecOps or Platform team to quickly map out the overhead of individual policies and export metrics. 

Our customer success team is already leveraging this to identify eBPF programs consuming high CPU usage, troubleshoot performance issues linked to specific eBPF policies, and optimize system resources to prevent slowdowns or bottlenecks in prod. 

tetra debug progs

2024-10-31 11:12:45.94715546 +0000 UTC m=+8.038098448

Ovh(%)  Id      Cnt     Time    Name                            Pin
  0.00  22201   0       0       event_execve                    /sys/fs/bpf/tetragon/__base__/event_execve/prog
  0.00  22198   0       0       event_exit_acct_process         /sys/fs/bpf/tetragon/__base__/event_exit/prog
  0.00  22200   0       0       event_wake_up_new_task          /sys/fs/bpf/tetragon/__base__/kprobe_pid_clear/prog
  0.00  22207   0       0       tg_cgroup_rmdir                 /sys/fs/bpf/tetragon/__base__/tg_cgroup_rmdir/prog
  0.00  22206   0       0       tg_kp_bprm_committing_creds     /sys/fs/bpf/tetragon/__base__/tg_kp_bprm_committing_creds/prog
  0.00  22221   0       0       generic_kprobe_event            /sys/fs/bpf/tetragon/syswritefollowfdpsswd/generic_kprobe/__x64_sys_close/prog
  0.00  22225   0       0       generic_kprobe_event            /sys/fs/bpf/tetragon/syswritefollowfdpsswd/generic_kprobe/__x64_sys_write/prog
  0.00  22211   0       0       generic_kprobe_event            /sys/fs/bpf/tetragon/syswritefollowfdpsswd/generic_kprobe/fd_install/prog

In the example above, we see the output from various eBPF programs, each identified by a unique ID, and includes metrics such as execution count, and system-wide overhead percent. This gives us a crucial step in understanding how specific programs interact with the system kernel and can identify bottlenecks or unusual behavior. 

Memory Usage Tracking of Programs & Maps 

Keeping production workloads running smoothly is critical, and memory management is a foundational piece of this overarching puzzle. The release of Tetragon 1.15 upgrades the way you measure and manage memory usage for individual programs. Deeper insight through the tetra CLI allows Platform and SecOps teams to surface up the memory profiles of individual eBPF programs, pinpointing exactly where resources are allocated.

This isn’t just about numbers; it’s about empowering teams to quickly diagnose issues, optimize performance, and ensure systems are running at their best. With a simple command, you unlock a treasure trove of insights into total memory usage across all your eBPF maps. Want to see which maps are used by Tetragon-pinned programs? Need to drill down into specific policies? It’s as simple as using the --path option and exploring the detailed memory map organized within the eBPF filesystem. 

Plus, with a sysdump integration, all this data is neatly stored in JSON format for easy access and analysis in any preferred tooling. It’s not just about managing memory; it’s about making it simple and easy, and with this release, platform owners are more readily equipped to keep systems efficient, responsive, and memory optimized.

In the screenshot above, lines 1-2 of the output display total memory used by (1) all eBPF maps on the system (AllMaps), (2) maps used in programs pinned by Tetragon (PinnedProgsMaps), and (3) all maps pinned by Tetragon (PinnedMaps). Lines 15-25 highlight the top 10 memory-consuming maps, along with their respective percentages of total use.

Audit and Protect BPF Operations 

Sitting in the kernel gives the strong distinct advantages of Tetragon that lead to lower overhead, deeper context, and flexibility to hook into any event on a workload – but with this kernel interaction comes the need for safety mechanisms like the eBPF verifier and, now with 1.15, strong audit trails into who and what is interacting with the eBPF subsystem. 

This introduces a set of TracingPolicies to monitor and secure eBPF subsystem interactions, detailing which processes interact with programs and maps.

Tetragon is already able to determine precisely which processes, containers, and Kubernetes workloads are interacting with the eBPF subsystem. This feature introduces three additional TracingPolicies that: 

1. reports which processes are creating eBPF programs, attaching or detaching them to the agent. 
2. monitors access to eBPF maps. 
3. reports access to the eBPF subsystem and BPFFS file system. 

SecOps teams are implementing these Tracing Policies to log attempts to create or access eBPF programs, triggering alerts for unexpected interactions deviating from legitimate Kubernetes processes. These use cases extend to detect unauthorized modifications to underlying eBPF programs, monitor access to specific sensitive eBPF maps to prevent data leaks, and generally audit BPF interactions to enforce compliance with security policies.

Identify Container init Processes 

Identifying the processes that are part of a container’s main initiation sequence gives clear context for what is part of the initial process tree and what is spawned externally.

In the past, this distinction was not readily apparent, leaving questions around whether processX was part of the main container init process or not. Now, process events have an in_init_tree flag that tracks if processX was part of the main container process. If a process is part of the initial sequence, it is marked as true; otherwise, it is false.

From a security perspective, this enhancement solves multiple use cases that revolve around investigating and hardening runtime after the container is initialized. These process flags allow SecOps teams to detect unauthorized processes injected into containers using commands like kubectl exec or docker exec. Additionally, we see a clearer audit trail for container processes, ensuring that security policies are being consistently enforced across different environments. By peering deeper into init activities, the in_init_tree field gives insight to focus on legitimate processes and identify any deviations that may indicate a security threat after container startup.

It’s as simple as using the process.in_init_tree field within JSON events to filter and analyze container initialization activities. A host process always has an in_init_tree value of false, a process injected into the container via kubectl exec, docker exec, or nsenter also has an in_init_tree value of false, while the container’s entrypoint or any process spawned from it has an in_init_tree value of true.

This allows to build out process ancestry trees highlighting what is not part of the “main container init”. Here we correlate network and runtime activity to understand the story around an attacker compromising a workload in an attempt to exfiltrate sensitive data.

Isovalent Enterprise for Cilium: Security Visibility Lab

Simulate the exploitation of a nodejs application, with the attacker spawning a reverse shell inside of a container and moving laterally within the Kubernetes environment.

Start Lab

Improved Userspace Filtering

When you’re managing security events, the last thing you need is to get lost in a flood of irrelevant data. That’s where Tetragon’s advanced userspace filtering comes in, offering you the ability to focus on critical events with laser-like precision. This improved filtering is designed to help you filter out the noise and zero in on what truly matters for your security operations.

This feature is particularly beneficial for implementing granular security policies. By utilizing new filters such as parent_argument_regex, destination_namespace_regex, in_init_tree, container_id, and cel_expressions, SecOps teams are now able to craft more flexible policies. For example, the parent_argument_regex can be used to filter events only when a specific script is called by a parent binary. Similarly, the destination_namespace_regex helps in excluding internal traffic within clusters, keeping the focus on external threats.

Let’s look at a practical application, consider using CEL expressions to filter process connection events within a specific IP range, allowing for exceptions where needed. This not only reduces the noise, but also aligns monitoring practices with the precise requirements of a security team.

Taking this example further, if your focus is on monitoring processConnect events within a specific CIDR range like 10.0.0.0/16, but you need to exclude certain IPs such as 10.0.20.137 and 10.0.31.128, CEL expressions can handle this nuanced requirement. 

{
  "cel_expressions": [
    "cidr('10.0.0.0/16').containsIP(process_connect.source_ip) && !(process_connect.source_ip in ['10.0.20.137', '10.0.31.128'])"
  ]
}

In this example, the CEL expression checks if the source IP of a processConnect event falls within the 10.0.0.0/16 range while excluding the specified exceptions. This significantly reduces noise and aligns monitoring with more granular security requirements, equipping SecOps teams with the tools needed to effectively observe and control their environments.

File Monitoring with Distroless Policies

In Kubernetes environments, keeping your containers secure often means minimizing what’s running to only the essentials. Tetragon’s 1.15 File Integrity Monitoring (FIM) improvement helps achieve this by using distroless policies to ensure that only specific, approved binaries can execute. This approach effectively mimics distroless images, where the container image is deliberately slimmed to enhance security.

With 1.15, create visibility and enforcement policies that define exactly which binaries are allowed to run. This means SecOps and Platform teams can closely control the execution environment, preventing unauthorized or unexpected binaries from operating.

What’s more, you can disable these policies on the fly to troubleshoot issues without needing ephemeral containers, providing both security and flexibility. By allowing policies to be disabled at runtime without bringing workloads offline, admins quickly address issues while maintaining a stable and secure production environment, minimizing disruption or complexity involved in troubleshooting.

From a security standpoint, these distroless policies offer clear benefits. They let you enforce a lean execution environment, blocking any binaries that don’t align with your defined application baseline. This is especially useful for managing common Docker images, distroless policies are relevant here as they allow you to enforce a similar minimalistic environment across your Kubernetes workloads, which naturally extends into becoming a more performant container. 

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "file-monitoring-demo-enforce-exec"
spec:
  podSelector: # all pods that match those labels will be monitored
    matchLabels:
      app: "nginx"
  file:
    file_paths_patterns:
    - type: AllFileOps # we care about all file operations (then we will filter out non-exec events)
    selectors:
    # selector 1: allowlist exec operation for php-fpm and nginx binaries
    - matchOperations:
      - operator: "In"
        values:
        - "FILE_EXEC"
      matchFilename:
      - operator: "InPattern"
        values: 
        - "/opt/bitnami/php/sbin/php-fpm"
        - "/usr/sbin/nginx"
      matchActions:
      - action: NoPost # this is a normal exec operation so no need to report anything
    # selector 2: block all other exec operations
    - matchOperations:
      - operator: "In"
        values:
        - "FILE_EXEC"
      matchActions:
      - action: Block # block (and report) all other exec operations

This policy combines the three elements needed to create a ‘distroless’ policy. A simplified breakdown follows: 1) – type: AllFileOps is used to monitor all file operations, 2) matchOperations: is used to capture all FILE_EXEC events and by extension binaries, and 3) matchFilename: – operator: “InPattern” to select specific binaries that are allowed to execute. Together, this mimics the intent of a distroless image, locking down the execution of allowed or disallowed binaries.

File Monitoring with eBPF and Tetragon (Part 1)

See the inode architecture and design decisions behind file monitoring and enforcement using eBPF & Tetragon.

Read the FIM blog

Digest Support in File Monitoring

Managing file integrity through digest verification is essential for host-system security, but manually handling these digests is impractical in large-scale environments. Tetragon 1.15 introduces SHA256 digest support to streamline this process. Previously, users needed to manually input digests in their TracingPolicies, which worked with a smaller number of target files, but became cumbersome for large-scale deployments.

The new InFileWithDigest operator in 1.15 automates collecting file digests, allowing you to define policies at a higher level with the target executables. 

Using TracingPolicies, SecOps teams can create high-level user policies that remove the need for users to manually define them. Tetragon automatically collects digests for all target host files and existing pods, and, importantly, checks new pods as they are created for relevant digests, automating the scaling out of any deployed policy. 

matchFilename:
- operator: "InFileWithDigest"
  values:
  - "/bin/sleep"
  - "/bin/bash"

This example policy uses the InFileWithDigest operator to verify that only the expected versions of the /bin/sleep and /bin/bash executables, confirmed by their SHA256 digests, are allowed to run, preventing unauthorized file modifications.

SandboxPolicies Reaches Stable

Exciting news! As part of 1.15, SandboxPolicies (introduced in Isovalent Enterprise for Tetragon 1.13) have reached stable maturity. The primary vision of SandBox Policies was to provide system call observability and enforcement via a higher-level custom resource definition (CRD).

These operate similar to a ‘Kubernetes Identity Aware Seccomp’, which takes in a list of syscalls and monitors them to perform an action ranging from alerting to blocking. Notably over seccomp, Tetragon does not require downtime when updating a policy and has K8s identity awareness to understand filtering on namespaces, pod labels, and other K8s metadata.

With SandboxPolicies, SecOps teams benefit from a simplified and efficient approach to enforce these identity aware security syscall policies across their infrastructure. These are extremely flexible, allowing you to dynamically update target syscalls or matchActions without having to restart the workload.

To learn more about building a Sandbox Policy, the initial announcement shares more through an example SandBox Policy. 

Aggregated Application JSON Events

Managing vast amounts of data can be overwhelming. Teams often lack the capacity to store terabytes of data generated by exporting every single individual process execution, network connection, and event. The primary goal here is to move away from the “firehose of events” model and improve towards this “aggregated/summary event” model.

For instance, if a curl command is executed 5,000 times in a Kubernetes workload to a site like google.com, SecOps teams prefer to have an aggregated summary rather than a flood of the same event.

This aggregation feature marks a pivotal shift from the traditional constant stream of events to a more efficient aggregated/summary model. By exporting aggregated JSON events, teams see a comprehensive view of activities within a Kubernetes workload without the overhead of excessive data storage or a flood of events. We found for a standard set of default policies, the aggregated application event model reduced data volume significantly, whereby instead of ~1TB / node / day we observed ~20MB / node / day in volume using the aggregated model.

Tetragon uses the eBPF maps to store runtime execution details (such as binary, parent binary, and arguments) and connection information. Tetragon aggregates runtime executions if the parent binary, binary, arguments, and Kubernetes workload match. Similarly, it aggregates network connections based on shared sources and destinations.

Isovalent Enterprise for Tetragon: Deeper Host Network Observability with eBPF

Visualize network data with out-of-the-box dashboards, see this blog covering TCP, UDP, DNS events with amazing Grafana dashboards.

Read Blog

Improved Default Ruleset Documentation

An exciting improvement from our customer success team and already being used by customers, the Default Ruleset (introduced in Isovalent Enterprise for 1.13 and improved on in Isovalent Enterprise for 1.14) has seen improvements in documentation for easier usability and faster rollouts. This provides more detailed dedicated pages for each customizable ruleset, including File Integrity Monitoring (FIM), Operating System Integrity (OSI), and Privileges Monitoring, offering clearer insight into functions and applications of these policies.

Each of the six rulesets now have easy-to-follow guidance with: 

• A detailed explanation of what the policy monitors, including specific kernel functions and system calls.
• Examples illustrating how the policy can be triggered.
• Clear instructions on interpreting events, highlighting relevant fields and their meanings.

Correlate process-to-network data, with host-based Kubernetes visibility.

Decode DNS, TLS, HTTP, UDP, TCP, and more while matching to process ancestry information, all with Kubernetes identity-aware metadata (labels, pod names, etc).

Get the Brief

Clean Vulnerability Security Report

Who watches the watcher? As we’ve talked about before (with the threat modeling guide), good security should look inwards to evaluate and close known gaps.

We are excited to share that after extensive penetration testing, reaching back to before the 1.0 release of Cilium Tetragon, there are no open Medium, High, or Critical severity vulnerabilities identified in Isovalent Enterprise for Tetragon. This is a huge milestone for the team, closing out the final four Medium severity tickets that had been remaining. 

Importantly as well, this report verifies the security status of the runtime agent for any organization that requires a report of no Critical, High, or Medium vulnerabilities for deployment in production. If you are interested in learning more about Isovalent Runtime

Security for deployment, please reach out directly and we would be happy to walk through specific use cases or deployment scenarios. 

Learn More!

If you’d like to learn more about Isovalent Enterprise for Tetragon 1.15 and related topics, check out the following links:

• Try Tetragon in a hands-on lab, earn a badge and progress down the Security Engineer Track. 
• Request a Demo – Schedule a demo session with an Isovalent Solution Architect.
• Learn more about Isovalent & Cilium at our resource library – It lists guides, tutorials, and interactive labs.
• Get the Security & Compliance White Paper – the 25+ page blueprint for Kubernetes compliance. 
• Extract security data no one else can with the host-based Kubernetes visibility solution brief.