Tutorial: Setting Up a Cybersecurity Honeypot with Tetragon to Trigger Canary Tokens

When writing up my previous blog post, “Can I use Tetragon without Cilium?“, I also covered a number of examples of events and triggers that can be implemented using Tetragon, such as file access, process event recording and TCP socket tracing.

As I learned more about how customers are using Tetragon, I came across the Canary Token use-case, which makes perfect sense! Tetragon, using eBPF to interrogate the kernel level events, provides you a number of options to trigger alerts.

In this tutorial blog post, we will learn how to create a cyber security honeypot using Tetragon to trigger canary tokens.

What is a Canary Token?

Canaries in the workplace are synonymous with use in coal mines, where the workers would take a canary into the mines with them, to help detect increasing levels of toxic gases.

Canary in cybersecurity has a different meaning – it refers to a virtual or physical device that can emulate another, developed by the cybersecurity company Thinkst. A canary token (also known as a honey token) is related to a honeypot that can be embedded into your system depending on the function you want it to take. When an attacker is working their way through the network and engages with this snare, an event or alert can be triggered without the attacker knowing they’ve hit a trip wire within your platform.

In this tutorial, we’ll cover just how easy it is to generate a canary token, and how to configure it as part of a Tetragon Tracing Policy. You can also follow along with the below recording.

Create a canary token

Creating a canary token can be done very easily:

Head over to https://canarytokens.org, a site is hosted by Thinkst (the creators of canary tokens), where we will generate a new Web URL token. This service is free of charge.
Click the “Select your token” button and select “Web bug/URL Token” from the drop-down menu.
Fill out your email address and a reminder note of what this token is used for.
Click “Create my CanaryToken” button.

Another option, rather than an email, is a call to a web hook, which could be part of another alerting system, such as Prometheus Alert Manager‘s Webhook Receiver or Slack.

You will now be provided a URL to use as part of a Tetragon Tracing Policy. Copy the URL down.

Create a Tetragon Tracing Policy that triggers the Canary Web URL

First, in our Kubernetes cluster, we need to install Tetragon. I recommend you read the official documentation installion guide for this, however the below commands will also get you started quickly.

For the provided commands, I have tested and run these from a Linux operating system. The same commands should work across platforms, but your terminal may interpret the lines differently when using copy and paste.

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
kubectl rollout status -n kube-system ds/tetragon -w

Then let’s deploy a demo application to use to test the Tetragon Policy.

kubectl create namespace demo-app
kubectl create --namespace demo-app -f https://raw.githubusercontent.com/cilium/cilium/1.14.1/examples/minikube/http-sw-app.yaml

For this part of the tutorial, I am going to use a tracing policy that is based on the File Name Access policy example in the Tetragon documentation repository. This policy will monitor the point when the files are mapped into the application’s virtual memory using the security_file_permission hook. If this matches our specified file, Tetragon will then call the provided URL, in this case, our canary token, which when accessed will alert by triggering an email.

Below is the example policy, which is watching the /etc/passwd file. Change the last line with the Canary URL generated earlier, and save this to your machine as monitor-etc-passwd-canary.yaml. This particular action is one of the only actions in Tetragon which is performed by the Tetragon Agent in the userspace, other actions, such as sigKill, are peformed from the kernel itself.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "monitor-etc-passwd-canary"
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file" # (struct file *) used for getting the path
    - index: 1
      type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE
    selectors:
    - matchArgs:      
      - index: 0
        operator: "Equal"
        values:
        - "/etc/passwd" # filter by filename (/etc/passwd)
      - index: 1
        operator: "Equal"
        values:
        - "2" # filter by type of access (MAY_WRITE)
      matchActions:
      - action: GetUrl
        argUrl: <<< Canary Token URL >>>

Let’s apply the tracing policy.

kubectl apply -f monitor-etc-passwd-canary.yaml

To trigger the canary, we will connect to our demo app xwing pod and edit the /etc/passwd file. This is achieved by using the echo command to append the text “something.strange” to the /etc/passwd file within the container.

We can also run a second command to confirm our public IP address, as this will also be captured in the Alert Email.

kubectl exec -n demo-app xwing it -- /bin/echo "something.strange" >> /etc/passwd

# To get your public IP address use one of the below commands
curl ifconfig.me
curl ipinfo.io/ip
curl ident.me

To confirm that the canary was triggered, below is a screenshot of the email generated by the Canary Token access, which provides some additional information of the trigger, such as the source.

In this example, I have performed these steps all from a single machine, but in a production environment, this information can be helpful to locate the bad actor, and track their movements around your system.

Tetragon Example = Canary Token Trigger Email

Evolving this example into a honeypot

This tutorial example focuses on monitoring the widely known /etc/passwd file, which could in a production environment cause false positive alerts, for example, the linux utility useradd would modify this file.

The aim of a honeypot is to be a decoy to a bad actor that attempts to infiltrate a system, collecting information for the cyber security team.

If we were to think about modifying this tutorial further to fit this description, one potential implementation could be to create a fake SSH private key file, with some randomised text within your container’s file system. As you know no system or application developer would be using or accessing this file, any attempt to read or write to it should be treated as suspicious, and trigger an alert.

Where can I learn more?

To learn more about Canary Tokens, and the different implementations, I recommend taking a look at Thinkst’s Blog, which has a number of in-depth tutorials and coverage of different token configurations.

If this is your first time reading about Tetragon, I recommend heading over to our Tetragon overview blog post, and also check out some of the other implementation examples covered in the “Can I Use Tetragon without Cilium?” blog post.

Tetragon isn’t just for Kubernetes environments either! It can be installed on standalone Linux systems, whether bare metal, virtual machines or as part of a cloud provider machine.

Finally, head over to our free hands-on-labs, and find out not only about Tetragon but the full power of Isovalent Enterprise for Cilium, and the security solutions we provide. As our number of labs keeps growing, we’ve also created a number of curated Learning Paths aligning labs to the different professional roles.

Explore the SecOps Engineer Learning Path

Covering topics such as securing the cluster (using Network Policies), ensuring confidentiality (encryption), integrity (mutual authentication) and connecting with existing firewalls (Egress Gateway).

Learning Path: SecOps Engineer