Cilium 1.16 introduces a number of updates to improve the observability of the Egress Gateway Traffic Path. In this recording we step you through each of these features. Introduction The Egress Gateway routing feature as part of Cilium allows you to define an exit route for traffic from your containers that is known to the rest of the network. This is particularly useful when you need to know the IP address traffic is coming from, for things like web proxies or more traditional firewalls in your environment. In Cilium 1.16 there have been enhancements so that you can get extra observability for that egress traffic path itself. Let's dive into a few of those features. First I'm going to check which egress address has been used inside my environment already today. This is already an existing feature. I'm checking by looking at the Cilium agent running on one of the nodes that has been selected for egress traffic. We can see the node IP and the egress IP that we're using for traffic going through the environment. We can also see a couple of source IPs already in use. Those are two pods running in my environment and they are trying to reach this external destination server as defined by my egress policy. Policy configuration The first enhancement is the introduction of NAT statistics inside our state database. We can now use Cilium DBG. The command has been updated to cilium-dbg, just to differentiate it from the command you can run locally on your machine, although inside the agent you can still use the cilium alias. Now we can see a number of additional statistics recorded. We have the egress IP address, the remote address the traffic is going to, the port numbers being used, and how many flows have been counted in that database. Essentially we're using network address translation that uses a port to map the connections through that single IP address to the remote destination, and then translate the return traffic back to the backend pods. The reason we now have these statistics is so that you can look at the exhaustion of a particular node as well. Observability These statistics are also exported as a metric that can be scraped by Prometheus or other observability tools. That metric is the NAT endpoint connection metric, and we have one for both IPv4 and IPv6. The metric is exported as a percentage. If that value hits one, it means 100 percent and the available port range has been exhausted on that node. At that point you might start to see assignment problems and traffic problems through that node. The way to alleviate that would be to have multiple egress nodes in the environment with different policies in place for different exit points. If you're looking at Isovalent Enterprise for Cilium, there is an egress HA feature as well which can help with that. Moving on, let's look at the enhancements that have also been made in Hubble so you can search through that data as well. First I'm going to look at traffic going through an egress node and output it to JSON, then use jq to format it because some additional fields have been added to the Hubble output. The first thing you'll notice is that when the traffic goes through the egress node and has been translated, we now have the source translated field. We can see it has been translated to my egress IP. If I scroll down, we've also got the node labels feature, so I can now filter based on the node label for egress or from particular availability zones inside my environment. A little bit further down, for traffic that traverses the egress node we also get the interface details. All of my nodes today use eth0, so we can filter on the node label, the interface, or the SNAT IP argument. Okay, so let's send some more traffic through my environment. I'm going to send more traffic using those two pods in my environment to the external IP address 172.18.0.7. This time I'm using a slightly different filter. I'm still filtering on the node label for this particular egress gateway, but I'm looking for all traffic with a translated source IP ending in 42. Now we can actually see those two pods, the Y-Wing and the X-Wing, and we can see them all going to that port. Conclusion From there I can keep filtering and dive into that more, or I can look at it using the new Hubble common expression language filters as well. That just wraps up some of the egress traffic path observability enhancements that we have in Cilium 1.16.