Tutorial: Transparent Encryption with IPsec and WireGuard

Nov 22, 2022Cilium

When the Cilium Service Mesh was still in beta, we ran a survey to assess which features users expected from a Service Mesh and I found it eye-opening that observability and traffic encryption were the most sought-after features.

With many regulatory frameworks – such as PCI or HIPAA – requiring encryption of data in transit, it really shouldn’t have been a surprise to me; especially as Kubernetes does not natively provide this capability.

Encrypting data in transit is something Cilium has actually been able to do natively, without a Service Mesh, since Cilium 1.4 and its support for IPsec. It was followed by the introduction of WireGuard support in 1.10.

What’s particularly notable with this feature is its transparency. As you will see in the tutorial and video below, the management overhead of encrypting traffic with Cilium is surprisingly low.

In this tutorial, we will walk through installing, configuring and managing Cilium Transparent Encryption with IPsec and WireGuard. If alternatively, you prefer learning by doing rather than reading, you can do the Isovalent lab instead by following this link.

If you’d rather watch, then this 10-minute video is for you.

Step 1: Check Environment

For this tutorial, we will be using Kind for our Kubernetes environment. We’ll be using a very simple configuration. Note that disableDefaultCNI is set to true as we will installing Cilium on it, with the Transparent Encryption features on (save this configuration as cluster.yaml if you want to follow along).

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
  disableDefaultCNI: true

After deploying the cluster with kind create cluster --config cluster.yaml, the nodes are in NotReady state, until we deploy Cilium.

# kubectl get nodes
NAME                 STATUS     ROLES           AGE   VERSION
kind-control-plane   NotReady   control-plane   21m   v1.24.0
kind-worker          NotReady   <none>          20m   v1.24.0
kind-worker2         NotReady   <none>          20m   v1.24.0
kind-worker3         NotReady   <none>          20m   v1.24.0

In Steps 2 and 3, we will be deploying and managing Cilium with IPsec. If you’re just interested in using WireGuard, you can skip straight to Step 4.

Step 2: Install Cilium with IPsec

One of the common challenges with cryptography is the management of keys. Users have to take into consideration aspects such as generation, rotation and distribution of keys.

We’ll look at all these aspects in this tutorial and see the differences between using IPsec and WireGuard as they both have pros and cons.

The way it is addressed broadly in Cilium is elegant – the IPsec configuration and associated key are stored as a Kubernetes secret. All secrets are automatically shared across all nodes and therefore all endpoints are aware of the keys.

Firstly, we’re going to be creating a Kubernetes secret for the IPsec configuration to be stored.

The format for such IPsec Configuration and key is the following: key-id encryption-algorithms PSK-in-hex-format key-size.

First, let’s generate a random pre-shared key (PSK) with the following command:

root@server:~# PSK=($(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64))
echo $PSK
5c7dc606a6e901eb92accc3063c5259ddca19049

Let’s create a Kubernetes secret called cilium-ipsec-keys, and use this newly created PSK:

root@server:~# kubectl create -n kube-system secret generic cilium-ipsec-keys \
    --from-literal=keys="3 rfc4106(gcm(aes)) $PSK 128"
secret/cilium-ipsec-keys created

This command might look confusing at first but essentially we are creating a generic Kubernetes secret from a key-value pair. In our case, the key is the name of the file to be mounted as a volume in the cilium-agent Pods while the value is the IPsec configuration in the format described earlier.

To decode the secret created earlier, you would have to run the following command:

root@server:~# SECRET="$(kubectl get secrets cilium-ipsec-keys -o jsonpath='{.data}' -n kube-system | jq -r ".keys")"
echo $SECRET | base64 --decode
3 rfc4106(gcm(aes)) 5c7dc606a6e901eb92accc3063c5259ddca19049 128

This maps to the following Cilium IPsec configuration :

  • key-id (an identifier of the key): arbitrarily set to 3
  • encryption-algorithms: AES-GCM GCM
  • PSK: eb8d63e016b73d7b386ac1b63b05fe50c643a041
  • key-size: 128

Now that the IPSec configuration has been generated, let’s install Cilium and IPsec, with the Cilium CLI command cilium install --encryption ipsec:

root@server:~# cilium install --encryption ipsec
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.14.0"
ℹ️  Using Cilium version 1.12.0
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected datapath mode: tunnel
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.12.0 --set cluster.id=0,cluster.name=kind-kind,encryption.enabled=true,encryption.nodeEncryption=false,encryption.type=ipsec,ipam.mode=kubernetes,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️  Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🔑 Found existing encryption secret cilium-ipsec-keys
🚀 Creating ConfigMap for Cilium version 1.12.0...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health

Notice in the installation logs that the Cilium installer found the encryption keys created earlier.

Let’s verify quickly that Cilium is healthy and that the IPsec has been enabled, as we expect:

root@server:~# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:       cilium             Running: 4
                  cilium-operator    Running: 1
Cluster Pods:     3/3 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.0@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade: 4
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.0@sha256:bb2a42eda766e5d4a87ee8a5433f089db81b72dd04acf6b59fcbb445a95f9410: 1


root@server:~# cilium config view | grep enable-ipsec
enable-ipsec                               true

Step 3: Manage IPsec on Cilium

So far, it’s been very easy: we created a Kubernetes secret representing our Cilium IPsec configuration and it was automatically distributed across all nodes.

We need to verify that traffic has been encrypted. We will be using the packet capture tool tcpdump for this purpose.

First, let’s run a shell in one of the Cilium agents

root@server:~# kubectl -n kube-system exec -ti ds/cilium -- bash
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)

Let’s then install the packet analyzer tcpdump to inspect some of the traffic (you may not want to run it like this in production environments 😅).

root@kind-worker2:/home/cilium# apt-get update
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
[...]
Reading package lists... Done

root@kind-worker2:/home/cilium# apt-get -y install tcpdump
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libpcap0.8
[....]
Unpacking tcpdump (4.9.3-4ubuntu0.1) ...
Setting up libpcap0.8:amd64 (1.9.1-3) ...
Setting up tcpdump (4.9.3-4ubuntu0.1) ...
root@kind-worker2:/home/cilium# 

Let’s now run tcpdump. We are filtering based on traffic on the cilium_vxlan interface.

When using Kind, Cilium is deployed by default in vxlan tunnel mode – meaning we set VXLAN tunnels between our nodes.

In Cilium’s IPsec implementation, we use Encapsulating Security Payload (ESP) as the protocol to provide confidentiality and integrity.

Let’s now run tcpdump and filter based on this protocol to show IPsec traffic:

root@kind-worker2:/home/cilium# tcpdump -n -i cilium_vxlan esp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cilium_vxlan, link-type EN10MB (Ethernet), capture size 262144 bytes
11:53:56.608920 IP 10.244.0.37 > 10.244.1.72: ESP(spi=0x00000003,seq=0x40), length 192
11:53:56.609275 IP 10.244.1.72 > 10.244.0.37: ESP(spi=0x00000003,seq=0x42), length 164
11:53:56.609482 IP 10.244.0.37 > 10.244.1.72: ESP(spi=0x00000003,seq=0x41), length 88
11:53:56.611890 IP 10.244.0.37 > 10.244.1.72: ESP(spi=0x00000003,seq=0x42), length 80
11:53:56.612030 IP 10.244.1.72 > 10.244.0.37: ESP(spi=0x00000003,seq=0x43), length 80
11:53:57.543682 IP 10.244.3.193 > 10.244.1.72: ESP(spi=0x00000003,seq=0x40), length 192
11:53:57.543718 IP 10.244.2.83 > 10.244.1.72: ESP(spi=0x00000003,seq=0x40), length 192
11:53:57.544105 IP 10.244.1.72 > 10.244.3.193: ESP(spi=0x00000003,seq=0x40), length 164

In my example above, I have four IPs (10.244.0.37, 10.244.1.7210.244.2.83 and 10.244.3.193,  – yours are likely to be different) that are the IP addresses of Cilium agents and what we are seeing in the logs are heartbeats between the Cilium agents, being transmitted over the IPsec tunnels.

Cilium’s IPsec implementation leverages a framework called xfrm, a very efficient kernel implementation of the IPsec protocol.

The following command shows the various security policies. In very simple terms, a security policy is a set of rules that determine which type of IP traffic needs to be secured using IPsec and how to secure that traffic.

root@kind-worker2:/home/cilium# ip -s xfrm p | grep -A14 "^src 10.244."
src 10.244.1.0/24 dst 10.244.0.0/24 uid 0
        dir out action allow index 209 priority 0 share any flag  (0x00000000)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2022-10-04 11:59:29 use 2022-10-04 12:00:57
        mark 0x3e00/0xff00 
        tmpl src 10.244.1.72 dst 10.244.0.37
                proto esp spi 0x00000003(3) reqid 1(0x00000001) mode tunnel
                level required share any 
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
--

The output above indicates that traffic from 10.244.1.0/24 (PodCIDR on the node where the agent is running from) to 10.244.0.0/24 (PodCIDR for a different node) will be sent and encapsulated down a tunnel from 10.244.1.72 to 10.244.0.37.

Now that we know traffic is being sent through the encrypted tunnels, we need to think about Day 2 Operations. There will come a point where users will want to rotate keys. Periodically and automatically rotating keys is a recommended security practice. Some industry standards, such as Payment Card Industry Data Security Standard (PCI DSS), require the regular rotation of keys.

To rotate the key with Cilium, we will therefore need to patch the previously created cilium-ipsec-keys Kubernetes secret, with kubectl patch secret (remember that the Cilium IPsec configuration and associated key are stored as a Kubernetes secret). During the transition, the new and old keys will be used.

Let’s try this now. First, let’s extract and print some of the variables from our existing secret.

root@server:~# read KEYID ALGO PSK KEYSIZE < <(kubectl get secret -n kube-system cilium-ipsec-keys -o go-template='{{.data.keys | base64decode}}')
echo $KEYID
echo $PSK
3
5c7dc606a6e901eb92accc3063c5259ddca19049

When you run echo $KEYID, it should return 3 but we could have guessed this – we used 3 as the key ID when we initially generated the Kubernetes secret.

Note the value of the existing PSK before we rotate the key.

We’ll increment the Key ID by 1 and generate a new PSK. We’ll use the same key size and encryption algorithm.

root@server:~# NEW_PSK=($(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64))
data=$(echo "{\"stringData\":{\"keys\":\"$((($KEYID+1))) "rfc4106\(gcm\(aes\)\)" $NEW_PSK 128\"}}")
kubectl patch secret -n kube-system cilium-ipsec-keys -p="${data}" -v=1
secret/cilium-ipsec-keys patched

Let’s check the IPsec configuration again:

root@server:~# read NEWKEYID ALGO NEWPSK KEYSIZE < <(kubectl get secret -n kube-system cilium-ipsec-keys -o go-template='{{.data.keys | base64decode}}')
echo $NEWKEYID
echo $NEWPSK
4
7ad83588d62c6f9f7711ed9c3a8d4e1fdd4d598b

You can see that the Key ID has incremented from 3 to 4 and that the PSK has changed. This example illustrates simple key management with IPsec with Cilium but production use would probably be more sophisticated.

Step 4: Install Cilium with WireGuard

As we saw in the previous sections, IPsec encryption provided a great method to achieve confidentiality and integrity. Even though it was transparent, there was some effort involved though: we needed to determine the cipher and key length used for our IPsec tunnels, we had to create the key and finally we had to manage and frequently rotate the keys.

In Cilium 1.10, we introduced an alternative technology to provide pod-to-pod encryption: WireGuard.

WireGuard, as described on its official website, is “an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography”. Compared to IPsec, “it aims to be faster, simpler, leaner, and more useful, while avoiding the massive headache.”

From a user perspective, we will see that the experience is very similar to the IPsec deployment, albeit operationally even simpler: one of the appeals of WireGuard is that it is very opinionated (it leverages very robust cryptography and does not let the user choose ciphers and protocols, like we did for IPsec) and its simplicity.

Indeed, WireGuard on Cilium is operationally very simple: the encryption key pair for each node is automatically generated by Cilium and key rotation is performed transparently by the WireGuard kernel module.

But before we get started with the installation, we need to check the Kernel version: WireGuard was integrated into the Linux kernel from 5.6.

root@server:~# uname -ar
Linux server 5.15.0-1013-gcp #18-Ubuntu SMP Sun Jul 3 04:59:25 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Our kernel is recent enough to support it. Note that WireGuard was backported to some older Kernels like the currently 5.4-based Ubuntu 20.04 LTS.

We can go ahead with installing Cilium with WireGuard:

root@server:~# cilium install --encryption wireguard
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.14.0"
ℹ️  Using Cilium version 1.12.0
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected datapath mode: tunnel
ℹ️  L7 proxy disabled due to Wireguard encryption
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.12.0 --set cluster.id=0,cluster.name=kind-kind,encryption.enabled=true,encryption.nodeEncryption=false,encryption.type=wireguard,ipam.mode=kubernetes,kubeProxyReplacement=disabled,l7Proxy=false,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️  Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.12.0...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health

Notice this ℹ️ L7 proxy disabled due to Wireguard encryption log – the Cilium installer noticed that WireGuard was enabled and disabled L7 proxy.

WireGuard provides some excellent benefits but users need to be aware of some limitations with other Cilium features, such as incompatibility with L7 policy enforcement and visibility. This means you cannot leverage Hubble for HTTP requests visibility or apply Layer 7 policy rules.

Step 5: Manage WireGuard on Cilium

You might have noticed that, unlike with IPsec, we didn’t have to create a key.

One advantage of WireGuard over IPsec is the fact that each node automatically creates its own encryption key-pair and distributes its public key via the io.cilium.network.wg-pub-key annotation in the Kubernetes CiliumNode custom resource object.

Each node’s public key is then used by other nodes to decrypt and encrypt traffic from and to Cilium-managed endpoints running on that node.

You can verify this by checking the annotation on the Cilium node kind-worker2:

root@server:~# kubectl get CiliumNode kind-worker2 -o jsonpath='{.metadata.annotations.io\.cilium\.network\.wg-pub-key}'
F9arJHdZk3NnDF5JTOgl40K4waRYTWwJLeTN7LdIYB8=

Let’s now run a shell in one of the Cilium agents on the kind-worker2 node.

First, let’s get the name of the Cilium agent and run a shell in it:

root@server:~# AGENT=$(kubectl get pods -A -l k8s-app=cilium -o wide | grep "kind-worker2" | awk '{ print($2); }')
echo $AGENT
cilium-8xf2d
root@server:~# 
root@server:~# kubectl -n kube-system exec -ti $AGENT -- bash
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init)

Let’s verify that WireGuard was installed:

root@kind-worker2:/home/cilium# cilium status | grep Encryption
Encryption:              Wireguard       [cilium_wg0 (Pubkey: F9arJHdZk3NnDF5JTOgl40K4waRYTWwJLeTN7LdIYB8=, Port: 51871, Peers: 3)]

Let’s explain this briefly:

  • We have 3 peers: the agent running on each cluster node has established a secure WireGuard tunnel between itself and all other known nodes in the cluster. The WireGuard tunnel interface is named cilium_wg0.
  • The WireGuard tunnel endpoints are exposed on UDP port 51871.
  • The public key’s value is the same one we saw before in the annotation.

Let’s now install the packet analyzer tcpdump to inspect some of the traffic. As in Step 3, we are installing and running tcpdump to visualize the traffic.

Instead of capturing traffic on the VXLAN tunnel interface like we did earlier, we are going to capture traffic on the WireGuard tunnel interface itself with tcpdump -n -i cilium_wg0. Let’s generate some traffic now.

We first deploy Pods in two different nodes (by manually pinning them).

root@server:~# cat pod1.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-worker
spec:
  nodeName: kind-worker
  containers:
  - name: netshoot
    image: nicolaka/netshoot:latest
    command: ["sleep", "infinite"]
root@server:~# cat pod2.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: pod-worker2
spec:
  nodeName: kind-worker2
  containers:
  - name: netshoot
    image: nicolaka/netshoot:latest
    command: ["sleep", "infinite"]

root@server:~# kubectl apply -f pod1.yaml -f pod2.yaml 
pod/pod-worker created
pod/pod-worker2 created

Let’s run a ping from one Pod to the other.

root@server:~# POD2=$(kubectl get pod pod-worker2 --template '{{.status.podIP}}')
echo $POD2
10.244.2.58
root@server:~# kubectl exec -ti pod-worker -- ping $POD2
PING 10.244.2.58 (10.244.2.58) 56(84) bytes of data.
64 bytes from 10.244.2.58: icmp_seq=1 ttl=60 time=1.83 ms
64 bytes from 10.244.2.58: icmp_seq=2 ttl=60 time=0.601 ms
64 bytes from 10.244.2.58: icmp_seq=3 ttl=60 time=0.505 ms
64 bytes from 10.244.2.58: icmp_seq=4 ttl=60 time=0.368 ms

If we look at the tcpdump results, we can see the ICMP packets going via the WireGuard interface – transparent encryption has been successfully enabled.

root@kind-worker2:/home/cilium# tcpdump -n -i cilium_wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cilium_wg0, link-type RAW (Raw IP), capture size 262144 bytes
13:19:18.169232 IP 10.244.3.104 > 10.244.2.58: ICMP echo request, id 161, seq 1, length 64
13:19:18.169481 IP 10.244.2.58 > 10.244.3.104: ICMP echo reply, id 161, seq 1, length 64
13:19:19.170199 IP 10.244.3.104 > 10.244.2.58: ICMP echo request, id 161, seq 2, length 64
13:19:19.170298 IP 10.244.2.58 > 10.244.3.104: ICMP echo reply, id 161, seq 2, length 64
13:19:20.192954 IP 10.244.3.104 > 10.244.2.58: ICMP echo request, id 161, seq 3, length 64
13:19:20.193040 IP 10.244.2.58 > 10.244.3.104: ICMP echo reply, id 161, seq 3, length 64
13:19:21.216807 IP 10.244.3.104 > 10.244.2.58: ICMP echo request, id 161, seq 4, length 64
13:19:21.216891 IP 10.244.2.58 > 10.244.3.104: ICMP echo reply, id 161, seq 4, length 64

Conclusion

Both Transparent Encryption implementations are both very easy to enable, as demonstrated above. WireGuard provided the added benefits of the automatic key management and rotation. In benchmark testing, WireGuard provided a significantly higher throughput although IPsec performed better on the latency and CPU consumption – ultimately, the choice is yours.

If you have any other governance and security requirements (such as the export of logs to a SIEM), you might want to consider Isovalent Cilium Enterprise. Click on the button below to schedule a demo.

Thanks for reading.

Learn More

Isovalent Resources:

Cilium and eBPF Resources:

Nico Vibert
AuthorNico VibertSenior Technical Marketing Engineer