When the Cilium Service Mesh was still in beta, we ran a survey to assess which features users expected from a Service Mesh and I found it eye-opening that observability and traffic encryption were the most sought-after features.
With many regulatory frameworks – such as PCI or HIPAA – requiring encryption of data in transit, it really shouldn’t have been a surprise to me; especially as Kubernetes does not natively provide this capability.
Encrypting data in transit is something Cilium has actually been able to do natively, without a Service Mesh, since Cilium 1.4 and its support for IPsec. It was followed by the introduction of WireGuard support in 1.10.
What’s particularly notable with this feature is its transparency. As you will see in the tutorial and video below, the management overhead of encrypting traffic with Cilium is surprisingly low.
In this tutorial, we will walk through installing, configuring and managing Cilium Transparent Encryption with IPsec and WireGuard. If alternatively, you prefer learning by doing rather than reading, you can do the Isovalent lab instead by following this link.
If you’d rather watch, then this 10-minute video is for you.
Step 1: Check Environment
For this tutorial, we will be using Kind for our Kubernetes environment. We’ll be using a very simple configuration. Note that disableDefaultCNI is set to true as we will installing Cilium on it, with the Transparent Encryption features on (save this configuration as cluster.yaml if you want to follow along).
After deploying the cluster with kind create cluster --config cluster.yaml, the nodes are in NotReady state, until we deploy Cilium.
In Steps 2 and 3, we will be deploying and managing Cilium with IPsec. If you’re just interested in using WireGuard, you can skip straight to Step 4.
Step 2: Install Cilium with IPsec
One of the common challenges with cryptography is the management of keys. Users have to take into consideration aspects such as generation, rotation and distribution of keys.
We’ll look at all these aspects in this tutorial and see the differences between using IPsec and WireGuard as they both have pros and cons.
The way it is addressed broadly in Cilium is elegant – the IPsec configuration and associated key are stored as a Kubernetes secret. All secrets are automatically shared across all nodes and therefore all endpoints are aware of the keys.
Firstly, we’re going to be creating a Kubernetes secret for the IPsec configuration to be stored.
The format for such IPsec Configuration and key is the following: key-id encryption-algorithms PSK-in-hex-format key-size.
First, let’s generate a random pre-shared key (PSK) with the following command:
Let’s create a Kubernetes secret called cilium-ipsec-keys, and use this newly created PSK:
This command might look confusing at first but essentially we are creating a generic Kubernetes secret from a key-value pair. In our case, the key is the name of the file to be mounted as a volume in the cilium-agent Pods while the value is the IPsec configuration in the format described earlier.
To decode the secret created earlier, you would have to run the following command:
This maps to the following Cilium IPsec configuration :
key-id(an identifier of the key): arbitrarily set to 3encryption-algorithms: AES-GCM GCMPSK: eb8d63e016b73d7b386ac1b63b05fe50c643a041key-size: 128
Now that the IPSec configuration has been generated, let’s install Cilium and IPsec, with the Cilium CLI command cilium install --encryption ipsec:
Notice in the installation logs that the Cilium installer found the encryption keys created earlier.
Let’s verify quickly that Cilium is healthy and that the IPsec has been enabled, as we expect:
Step 3: Manage IPsec on Cilium
So far, it’s been very easy: we created a Kubernetes secret representing our Cilium IPsec configuration and it was automatically distributed across all nodes.
We need to verify that traffic has been encrypted. We will be using the packet capture tool tcpdump for this purpose.
First, let’s run a shell in one of the Cilium agents
Let’s then install the packet analyzer tcpdump to inspect some of the traffic (you may not want to run it like this in production environments 😅).
Let’s now run tcpdump. We are filtering based on traffic on the cilium_vxlan interface.
When using Kind, Cilium is deployed by default in vxlan tunnel mode – meaning we set VXLAN tunnels between our nodes.
In Cilium’s IPsec implementation, we use Encapsulating Security Payload (ESP) as the protocol to provide confidentiality and integrity.
Let’s now run tcpdump and filter based on this protocol to show IPsec traffic:
In my example above, I have four IPs (10.244.0.37, 10.244.1.72, 10.244.2.83 and 10.244.3.193, – yours are likely to be different) that are the IP addresses of Cilium agents and what we are seeing in the logs are heartbeats between the Cilium agents, being transmitted over the IPsec tunnels.
Cilium’s IPsec implementation leverages a framework called xfrm, a very efficient kernel implementation of the IPsec protocol.
The following command shows the various security policies. In very simple terms, a security policy is a set of rules that determine which type of IP traffic needs to be secured using IPsec and how to secure that traffic.
The output above indicates that traffic from 10.244.1.0/24 (PodCIDR on the node where the agent is running from) to 10.244.0.0/24 (PodCIDR for a different node) will be sent and encapsulated down a tunnel from 10.244.1.72 to 10.244.0.37.
Now that we know traffic is being sent through the encrypted tunnels, we need to think about Day 2 Operations. There will come a point where users will want to rotate keys. Periodically and automatically rotating keys is a recommended security practice. Some industry standards, such as Payment Card Industry Data Security Standard (PCI DSS), require the regular rotation of keys.
To rotate the key with Cilium, we will therefore need to patch the previously created cilium-ipsec-keys Kubernetes secret, with kubectl patch secret (remember that the Cilium IPsec configuration and associated key are stored as a Kubernetes secret). During the transition, the new and old keys will be used.
Let’s try this now. First, let’s extract and print some of the variables from our existing secret.
When you run echo $KEYID, it should return 3 but we could have guessed this – we used 3 as the key ID when we initially generated the Kubernetes secret.
Note the value of the existing PSK before we rotate the key.
We’ll increment the Key ID by 1 and generate a new PSK. We’ll use the same key size and encryption algorithm.
Let’s check the IPsec configuration again:
You can see that the Key ID has incremented from 3 to 4 and that the PSK has changed. This example illustrates simple key management with IPsec with Cilium but production use would probably be more sophisticated.
Step 4: Install Cilium with WireGuard
As we saw in the previous sections, IPsec encryption provided a great method to achieve confidentiality and integrity. Even though it was transparent, there was some effort involved though: we needed to determine the cipher and key length used for our IPsec tunnels, we had to create the key and finally we had to manage and frequently rotate the keys.
In Cilium 1.10, we introduced an alternative technology to provide pod-to-pod encryption: WireGuard.
WireGuard, as described on its official website, is “an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography”. Compared to IPsec, “it aims to be faster, simpler, leaner, and more useful, while avoiding the massive headache.”
From a user perspective, we will see that the experience is very similar to the IPsec deployment, albeit operationally even simpler: one of the appeals of WireGuard is that it is very opinionated (it leverages very robust cryptography and does not let the user choose ciphers and protocols, like we did for IPsec) and its simplicity.
Indeed, WireGuard on Cilium is operationally very simple: the encryption key pair for each node is automatically generated by Cilium and key rotation is performed transparently by the WireGuard kernel module.
But before we get started with the installation, we need to check the Kernel version: WireGuard was integrated into the Linux kernel from 5.6.
Our kernel is recent enough to support it. Note that WireGuard was backported to some older Kernels like the currently 5.4-based Ubuntu 20.04 LTS.
We can go ahead with installing Cilium with WireGuard:
Notice this ℹ️ L7 proxy disabled due to Wireguard encryption log – the Cilium installer noticed that WireGuard was enabled and disabled L7 proxy.
WireGuard provides some excellent benefits but users need to be aware of some limitations with other Cilium features, such as incompatibility with L7 policy enforcement and visibility. This means you cannot leverage Hubble for HTTP requests visibility or apply Layer 7 policy rules.
Step 5: Manage WireGuard on Cilium
You might have noticed that, unlike with IPsec, we didn’t have to create a key.
One advantage of WireGuard over IPsec is the fact that each node automatically creates its own encryption key-pair and distributes its public key via the io.cilium.network.wg-pub-key annotation in the Kubernetes CiliumNode custom resource object.
Each node’s public key is then used by other nodes to decrypt and encrypt traffic from and to Cilium-managed endpoints running on that node.
You can verify this by checking the annotation on the Cilium node kind-worker2:
Let’s now run a shell in one of the Cilium agents on the kind-worker2 node.
First, let’s get the name of the Cilium agent and run a shell in it:
Let’s verify that WireGuard was installed:
Let’s explain this briefly:
- We have 3 peers: the agent running on each cluster node has established a secure WireGuard tunnel between itself and all other known nodes in the cluster. The WireGuard tunnel interface is named
cilium_wg0. - The WireGuard tunnel endpoints are exposed on UDP port 51871.
- The public key’s value is the same one we saw before in the annotation.
Let’s now install the packet analyzer tcpdump to inspect some of the traffic. As in Step 3, we are installing and running tcpdump to visualize the traffic.
Instead of capturing traffic on the VXLAN tunnel interface like we did earlier, we are going to capture traffic on the WireGuard tunnel interface itself with tcpdump -n -i cilium_wg0. Let’s generate some traffic now.
We first deploy Pods in two different nodes (by manually pinning them).
Let’s run a ping from one Pod to the other.
If we look at the tcpdump results, we can see the ICMP packets going via the WireGuard interface – transparent encryption has been successfully enabled.
Conclusion
Both Transparent Encryption implementations are both very easy to enable, as demonstrated above. WireGuard provided the added benefits of the automatic key management and rotation. In benchmark testing, WireGuard provided a significantly higher throughput although IPsec performed better on the latency and CPU consumption – ultimately, the choice is yours.
If you want to experience for yourself how the transparent encryption works, you can run our interactive lab:
Thanks for reading.
Learn More
Isovalent Resources:
- Learn more about Isovalent Cilium Enterprise
- Next-Generation Mutual Authentication with Cilium Service Mesh
Cilium and eBPF Resources:
Prior to joining Isovalent, Nico worked in many different roles—operations and support, design and architecture, and technical pre-sales—at companies such as HashiCorp, VMware, and Cisco.
In his current role, Nico focuses primarily on creating content to make networking a more approachable field and regularly speaks at events like KubeCon, VMworld, and Cisco Live.
