Learn how use the Cilium Agent CLI for advanced Kubernetes network troubleshooting - in this video we cover IPSec encryption. Introduction So we're going to take a look at transparent encryption status and rotating keys using the Cilium agent CLI, cilium-dbg. This lives inside the Cilium agent itself, and the Cilium agent runs on every node, so typically we use this CLI command for more advanced troubleshooting use cases. In my environment we're just going to check that I've got it up and running. I'm using the latest open source version, 1.15.4, and we're actually running this inside the Isovalent lab. You can take this lab yourself at Isovalent.com/labs and look for the one based on transparent encryption with IPsec and WireGuard. Verify IPSec configuration First and foremost, we want to validate that IPsec is indeed configured. We can see that it has been set as enabled, that we've got the key watcher set up, the location for the key files, and how long the keys rotate for. Let's also double-check that it is indeed working as expected. We're going to connect to my Cilium agent pod running on my node. This isn't best practice, but we're just going to use it today to prove that encryption is working because it's transparent inside the cluster. I'm going to install tcpdump so that I can capture some of those encrypted packets. Packet inspection There we go, we can see those encrypted packets. Our lab takes you through this in a little bit more detail as well. Now I want to get the key number that's used, because that's going to be quite important in a moment. We can see that we've got key number five and the ID associated with that. Now I'm going to connect back into one of my Cilium agents, and we're going to pretend that we've had a node that has been compromised for some reason. So we're unsure about the state of the IPsec key inside the environment and we need to act on this now. If I check the status of encryption, we can see that IPsec is working, it's got a single key in use, and there are no errors at the moment. However, I need a way to handle this node being compromised. So we can use the Cilium agent CLI encrypt command and actually flush the current IPsec state. We can force that and skip confirmation, specify the output, target a particular SPI, and also remove stale states and policies from the identity map. I'm just going to flush everything from this node and say yes. If we check the status on this node now, we're going to see that there are no keys in use, which means we've interrupted traffic to this node because the IPsec tunnel is not up and working. Troubleshoot IPSec tunnel state So let's fix that. I'm going to exit that node and generate a whole new key for everything inside my environment. Now we can see that we've generated a new key and patched the IPsec keys at platform level inside the secret. If I read that key, we can also see that we've got a new key ID, so we know it's different from the previous one. Then I go back into my Cilium agent pod, clear some of the output, and check the status again. Now we can see that keys in use have gone back up to one. There are a number of errors generated, but that's because I forced the flushing of those states and policies from my node as part of that compromise resolution. To learn more, download our Cilium cheat sheet from Isovalent.com. It covers both the CLI and the Cilium agent CLI, and hopefully makes life a lot easier when you're using the platform.