Can I Use Tetragon without Cilium? Yes!
eBPF-based enforcement, visibility, and forensics for Kubernetes regardless of your CNI
One of the common questions I see across social media when users learn of the capabilities of Tetragon is “Can I Use Tetragon without Cilium?”
The answer is Yes! You can use Tetragon regardless of the Container Network Interface (CNI) implemented. Tetragon provides eBPF-based enforcement, visibility, and forensics for Kubernetes regardless of your CNI.
In this blog post, I’m going to walk you through an example using a cluster with Calico Open-Source configured, and Tetragon installed to dive into process events.
First, for those of you who haven’t dived into the world of eBPF powered observability and security, let’s explain Tetragon.
What is Tetragon?
Tetragon is an Open-Source eBPF based Security Observability and Runtime enforcement platform, which was donated by Isovalent to the CNCF in 2022. Isovalent also provides an extended enterprise version of Tetragon.
Tetragon provides deep visibility without requiring you to change your applications and workloads, due to its smart in-kernel filtering and aggregation logic, built directly into the eBPF-based Kernel-Level collector.
Tetragon detects and is able to react to security-significant events, such as;
- Process execution events
- System call activity
- I/O activity including network & file access
When installed into a Kubernetes environment, Tetragon is Kubernetes-aware. This means that Tetragon can match the Kubernetes metadata, such as namespaces, pods, labels and beyond, to the security and process event information collected against workloads running in the cluster.
You can read a deep dive about Tetragon in our earlier blog post, or dive into the official Tetragon docs.
Getting Started and Installing Tetragon
For this walkthrough, we are going to follow the Tetragon Kubernetes getting-started guide, and then look at some more advanced use-cases. I have also recorded the below video which you can follow along with, as well as going through this blog post.
My environment is a Kind cluster setup with the below configuration, and Calico Open-Source installed without any modifications. Which means you will be able to follow along with this tutorial from your own local machine.
We can validate the configuration below:
To install Tetragon, we will do this using the available Helm Chart, the below commands configure the helm chart repository and install Tetragon, and wait for it to become ready:
This will produce an output like the following example:
To generate some security observability events in our environment, I am going to deploy the following Cilium Demo application, if you have used any of the Isovalent Hands-on-Labs to explore Cilium and Hubble in the past, you may be familiar with this demo application:
We can validate that all components of the demo application are configured, and pods are running with the following command:
Now, to install the tetra CLI, head over to the official documentation, as it lists the installation steps for each platform.
Below are the steps for installation to Linux amd64 architecture.
Viewing process events from our workloads
Once we have the CLI installed, let’s generate some events in the xwing pod, using the below command we exec into the xwing pod to run some commands:
We can view the process events by running the following command, I recommend doing this in a separate terminal window, so that you can leave this command running to capture the output:
This command gets the events which are recorded as logs in the “export-stdout” container that runs as part of the Tetragon services in your cluster. There are other methods to export and access this data collected by Tetragon, either by connecting to the Tetragon containers where the tetra CLI is available, or configuring logs to be forwarded to a SIEM solution.
By piping this output to the tetra cli, we can then use this tool to filter the logs by the Kubernetes metadata, in this example, namespace and pod.
For the full details, we can parse the JSON log file using the jq command, as per the below command example which looks for the pod “xwing”. You can find more information about process exec events in the official documentation.
A Short Introduction to Tracing Policies
Tetragon implements a user-configurable resource called a Tracing Policy. This resource allows users to trace events in the kernel and optionally define actions to take on matches. Policies are made up of two components: hook point and a selector.
- A hook point, is the location Tetragon will trace the event from within the Kernal. Tetragon supports kprobes, tracepoints and uprobes currently. A hook point has an argument (specifies the arguments of the kernel function being traced and to collect in the trace output) and return value (type of the return argument to be captured and returned in the trace output).
- Selectors allow in-kernel BPF filtering and the ability to take actions on matching events. Actions can range from logging the event, overriding a return value of a system call, to more complex examples, such as socket tracking.
In this tutorial, the example Tracing Policies use kprobes as the hook point. Kprobes allow Tetragon to break into the kernel routine and collect debugging and performance information non-disruptively, these are tied to the version of Kernel in-use. For more information regarding the use of kprobes, please visit the official documentation.
Network Observability without Cilium
One of the areas you would expect to need Cilium to be present in your Kubernetes Cluster, is when it comes to network observability using Tetragon. However again, Cilium is not needed for this use case either.
First let’s apply a tracing policy that captures TCP events from our workloads.
Below is the YAML file, which is using Kprobes to monitor for “tcp_” based calls:
Now let’s generate some traffic by connecting to the Isovalent website from the xwing pod.
Going back to our second terminal window, let’s look at the filtered tetragon logs for the xwing pod:
In Tetragon Enterprise Edition, one feature is the ability to provide an FQDN for those returned IP addresses via DNS lookup, if you want to dive into the enterprise features further, provided at the end of this walkthrough are links to our free Hands-on-labs which take you into more advanced use-cases.
Check process capabilities and kernel namespaces access
Tetragon provides the ability to check the process capabilities and kernel namespace access. This information is useful for security teams to determine breaches whereby a process or Kubernetes pod has gained access to privileges or host namespaces which it should not have.
To demo this capability easily, you can follow these steps, however, we also have a dedicated, free hands-on lab that covers this subject in further detail, which I’ve linked at the end of this section.
First, we need to enable visibility into the capability and namespace changes by updating the configmap for Tetragon. This can be easily achieved using the helm upgrade command argument.
To capture the additional data we are now collecting, and show the impact of containers with different privileges, let’s deploy a new pod into our demo-app namespace, which has its security context set to privileged.
Below is the manifest that we will apply to our environment, the command to create this pod will follow:
First let’s change the tetra command we are using to focus on the new pod name, then in a separate terminal window, apply the manifest to the cluster to create the pod.
To create the pod:
Now let’s observe the process events captured for this pod, which we can see has the “CAP_SYS_ADMIN” capability. You can read more details about Linux capabilities here.
This was obviously a very quick overview of this capability, head over to the “Getting Started with Tetragon” lab, which takes you through an end-to-end container escape attack and how to detect this using Tetragon OSS.
Getting Started with Tetragon
Perform a real-world Kubernetes container escape and detect it with Tetragon!
Start Lab
Kubernetes namespace and pod label filtering
The examples so far have used tracingpolicies which are applied at a cluster level. However, in Kubernetes, you may have requirements to apply these policies to a smaller scope. Tetragon now supports filtering based on Kubernetes namespaces and pod labels, currently this feature is in beta.
First to enable this beta feature, we need to make a change to our Tetragon configuration using Helm.
We can confirm the changes we’ve made with the helm upgrade commands in this tutorial with the following command to view the config map tetragon-config.
Applying TracingPolicies to a Kubernetes Namespace
For this example, we are going to use one of the publicly available examples, which implements file level monitoring as a template to create a new policy for tracking activity (reads/writes) within our container. I cannot think of a better example than monitoring the /etc/passwd and /etc/shadow file.
Below is my TracingPolicyNamespaced, which modifies the above linked example to change:
KindtoTracingPolicyNamespaced- Add
/etc/shadowto the list of files to be monitored
I have saved this file locally as file-monitoring-etc-files.yaml
Now we will apply the following YAML file to our cluster, which is scoped against our demo-app namespace.
Now, using the kubectl exec command to execute against the “xwing” and “test-pod” (the privileged pod we deployed earlier) container and run a few commands to view and alter the files using different methods, we see that each method is recorded.
Now when we check our events, I am using the tetra command to capture all events in the namespace demo-app and have removed the --pods argument used in the earlier examples.
Applying TracingPolicies to specific pods based on label selectors
The last example is applying a TracingPolicy based on Kubernetes labels. For this, we will create a policy that stops writes to files inside of the /tmp/ folder for any pod which has the label app=test.
First let’s create the file deny-write-tmp.yaml with the following content:
When applying this to the ‘demo-app’ namespace, take note that the ‘Kind’ field in the above YAML file is set to ‘TracingPolicyNamespaced’ because the changes are specific to that Kubernetes namespace.
Now, let’s apply the chosen label to the privileged pod I deployed earlier.
And to test the policy, we should expect that if I try to write any changes to a file located in the /tmp/ directory, the process is exited.
It’s a little hard to capture this in written form with the outputs expected, so instead I’ve captured it in a video below, at the end showing the expected output from the logs as well.
Below are the Tetra events.
Where can I learn more?
You might have noticed that there wasn’t any mention of the underlying CNI throughout this tutorial, and that’s the reason why I created this blog post! Tetragon doesn’t rely on the underlying CNI at all to bring to your platform these powerful eBPF based observability and enforcement features.
Earlier in the blog post, I gave you the link to our free hands-on lab which takes you through the Open-Source Tetragon offering in further detail. So why not jump into the following labs also getting hands-on with the extended enterprise version of Tetragon with “Isovalent Enterprise for Cilium: Security Visibility“, discover how to gain insights into TLS with “Isovalent Enterprise for Cilium: TLS Visibility” and you can follow up for a live demo from our team here.
Isovalent Enterprise for Cilium: Security Visibility
Identify Late Process Execution, trace lateral movement and data exfiltration with Tetragon Enterprise and Hubble Enterprise.
Start Lab
Isovalent Enterprise for Cilium: TLS Visibility
Mechanisms like TLS ensure that data is encrypted in transit but verifying that a TLS configuration is secure is a challenge. In this lab use Tetragon Enterprise to gain visibility into TLS.
Start Lab
And for those of you who prefer video content to watch along to, I have you covered there as well. A quick start to Tetragon from our security product manager, Natália Ivánkó, and how to use Tetragon to discover container escapes.
Dean Lewis is a Senior Technical Marketing Engineer at Isovalent – the company behind the open-source cloud native solution Cilium.
Dean had a varied background working in the technology fields, from support to operations to architectural design and delivery at IT Solutions Providers based in the UK, before moving to VMware and focusing on cloud management and cloud native, which remains as his primary focus. You can find Dean in the past and present speaking at various Technology User Groups and Industry Conferences, as well as his personal blog.
