Cilium 1.16 has landed! The theme of the release is “Faster, Stronger, Smarter” - faster for the blazing performances you will get with netkit, the new virtual network device, stronger for all the security and operational improvements, such as Network Policies Port Range support and smarter for all the new traffic engineering features such as Kubernetes Service Traffic Distribution, Local Redirect Policy and a 5x reduction in tail latency for DNS policies! Cilium is the third fastest-growing project in the CNCF, a testament to the thriving community of over 750 developers. This release includes contributions from engineers at organizations such as DaoCloud, Datadog, Equinix, Google, Microsoft, Seznam, Spectro Cloud, Starling Bank, and many more. Our thanks to them all. Let's review the major new additions in this release. For a comprehensive look at all the new features, bug fixes and changes, check the Cilium 1.16 release notes. Remember to read the upgrade guide notes before proceeding. Cilium 1.16 – New features at a glance The latest open source release of Cilium includes all these new features and improvements. Networking Cilium netkit: container-network throughput and latency as fast as host-network (more details) Service traffic distribution: Kubernetes 1.30 Service Traffic Distribution is the successor to topology-aware routing (more details) Local Redirect Policy promoted to Stable: redirecting the traffic bound for services to the local backend, such as node-local DNS (more details) BGPv2: Fresh new API for Cilium’s BGP feature (more details) BGP ClusterIP advertisement: BGP advertisements of ExternalIP and Cluster IP Services (more details) Node IPAM Service LB: Ability to assign IP addresses from the nodes themselves to Kubernetes services, providing alternative access to services from outside of the cluster (more details) Multicast datapath: Cilium 1.16 supports a multicast datapath, enabling Kubernetes users to deploy applications and benefit from multicast’s efficiencies (more details) Per-pod fixed MAC address: addressing use-cases such as software that is licenced based on a known MAC address (more details) Service mesh & Ingress/Gateway API Gateway API GAMMA support: East-west traffic management for the cluster via Gateway API (more details) Gateway API 1.1 support: Cilium now supports Gateway API 1.1 (more details) Gateway API support for more protocol options: Cilium Gateway API supports new protocol options such as proxyProtocol, ALPN and appProtocol (more details) Local ExternalTrafficPolicy support for Ingress/Gateway API: External traffic can now be routed to node-local endpoints, preserving the client source IP address (more details) L7 Envoy Proxy as dedicated DaemonSet: With a dedicated DaemonSet, Envoy and Cilium can have a separate life-cycle from each other. Now on by default for new installs (more details) Host Network mode & Envoy listeners of subset of nodes: Cilium Gateway API Gateway/Ingress can now be deployed on the host network and on selected nodes (more details) Security Port Range support in Network Policies: This long-awaited feature has been implemented into Cilium (more details) Network Policy validation status: kubectl describe cnp <name> will be able to tell if the Cilium Network Policy is valid or invalid (more details) Control Cilium Network Policy Default Deny behavior: Policies usually enable default deny for the subject of the policies, but this can now be disabled on a per-policy basis (more details) CIDRGroups support for Egress and Deny rules: Add support for matching CiliumCIDRGroups in Egress and Deny policy rules (more details) Load “default” Network Policies from filesystem: In addition to reading policies from Kubernetes, Cilium can be configured to read policies locally (more details) Select nodes as the target of Cilium Network Policies: With new ToNodes/FromNodes selectors, traffic can be allowed or denied based on the labels of the target Node in the cluster (more details) Day 2 operations and scale Improved DNS-based network policy performance: Reduction of up to 5x reduction in tail latency for DNS-based network policies (more details) New ELF loader logic: With this new loader logic, the median memory usage of Cilium was decreased by 24% (more details) KVStoreMesh default option for ClusterMesh: Introduced in Cilium 1.14, and after a lot of adoption and feedback from the community, KVStoreMesh is now the default way to deploy ClusterMesh (more details) Hubble & observability CEL filters support: Hubble supports Common Expression Language (CEL), which provides more complex conditions that cannot be expressed using the existing flow filters (more details) Filtering Hubble flows by node labels: Filter Hubble flows observed on nodes matching the given label (more details) Improvements for egress traffic path observability: Enhancements to Cilium Metrics and Hubble flow data for traffic that traverses Egress Gateways, allowing better troubleshooting of this popular Cilium feature (more details) K8S event generation on packet drop: Hubble is now able to generate a k8s event for a packet dropped from a pod, and that can be verified with kubectl get events (more details) Networking Cilium netkit Containerization has always come at a performance cost, with the most visible one on networking velocity. A standard container networking architecture would result in a 35% drop in network performance compared to the host. How could we bridge that gap? Over the past 7 years, Cilium developers have added several features aimed at reducing this performance penalty. With Cilium 1.16 and the introduction of Cilium netkit, you can finally achieve performance parity between host and container. Cilium is the first public project providing built-in support for netkit, a Linux network device introduced in the 6.7 kernel release and developed by Isovalent engineers. To learn more, read our deep dive into the journey into high-performance container networking, with netkit the final frontier. Service traffic distribution Cilium 1.16 will support Kubernetes' new “Traffic Distribution for Services” model which aims at providing topology-aware traffic engineering. It can be considered the successor to features such as Topology-Aware Routing and Topology-Aware Hints. Introduced in Kubernetes 1.30, Service Traffic Distribution enables users to express preferences on traffic policy for a Kubernetes Service. Currently, the only supported value is “PreferClose”, indicating a preference for routing traffic to endpoints that are topologically proximate to the client. By keeping the traffic within a local zone, users can optimize performance, cost and reliability. Introduced in Kubernetes 1.30, Service Traffic Distribution can be enabled directly in the Service specification (rather than using annotations as it was the case with Topology-Aware Hints & Routing): In the demo below, we have 6 backend pods across two Availability Zones. By default, traffic to the Service is distributed across all 6 pods, regardless of their location. When enabling Traffic Distribution, you can see that the traffic from the client is only sent to the 3 pods in the same AZ. When no local backend is unavailable, traffic is forwarded to backends outside the local zone. Alongside Service Traffic Distribution support, Cilium is also introducing tooling to monitor traffic inter- and intra-zones. Given that Service Traffic Distribution and its predecessors all derive the zones from labels placed on the nodes, users can now monitor cross-zone traffic using Hubble’s new node labels capabilities described later in this blog post and identify how to reduce cost and latency. To learn more, check out the Cilium documentation. Local Redirect Policy promoted to Stable Service Traffic Distribution is not the only traffic engineering feature we wanted to highlight in this release. Local Redirect Policy was initially introduced in Cilium 1.9 and is commonly adopted by users looking at optimizing networking. Just as Service Traffic Distribution helps keep the traffic within the zone, Local Redirect Policy prevents traffic from even leaving the node by directing the traffic bounds for services to the local backend. It’s been particularly popular for DNS use cases: use it with NodeLocal DNSCache to improve DNS lookup latency in your clusters. NodeLocal DNSCache relies on a DNS caching agent running on each cluster node as a DaemonSet. With Cilium’s eBPF-based Local Redirect Policy, DNS traffic from Pod goes to the DNS cache running on the same node as the Pod, reducing latency amongst other benefits. To learn more, check out the documentation or read how Trendyol, one of the five largest e-commerce platforms in EMEA, is optimizing cluster performance with Local Redirect Policy. BGPv2 Most applications running Kubernetes clusters need to talk to external networks. In self-managed Kubernetes environments, BGP is often used to advertise networks used by Kubernetes Pods and Services, to make applications accessible from traditional workloads. Cilium has natively supported BGP since the 1.10 release. It’s quickly become a popular feature as users appreciate not having to install a separate BGP daemon to connect their clusters to the rest of the network. We’ve seen thousands of users taking our “BGP on Cilium” and “Advanced BGP on Cilium” labs and collecting badges along the way. What we’ve noticed recently is an acceleration of its adoption in scenarios such as: Users looking at accessing KubeVirt-managed Virtual Machines running on Red Hat OpenShift clusters Users looking at connecting their external network fabric (such as Cisco ACI) to Kubernetes clusters Expect some content on both architectures in the coming weeks and months. This fast adoption of BGP is, however, a victim of its own success. The current method to deploy BGP on Cilium is through a single CustomResourceDefinition CiliumBGPPeeringPolicy, which comes with a couple of drawbacks: We need to explicitly enumerate per-neighbor settings, even when they match between multiple peers All Peers currently get the same advertisement (there’s no control such as concepts such as prefix-list) Simply put, while the CRD worked in simple topologies, sophisticated networking topologies require a better set of abstractions, similar to what most of the popular BGP daemons use (e.g., peer templates, route maps, etc.…). To provide users the flexibility they need, Cilium 1.16 is introducing a new API - BGPv2 APIs. Instead of a single CRD, you will be able to use a new set of CRDs to define your complex CiliumBGPClusterConfig: Defines BGP instances and peer configurations applied to multiple nodes. CiliumBGPPeerConfig: A common set of BGP peering settings that can be used across multiple peers. CiliumBGPAdvertisements: Defines prefixes that are injected into the BGP routing table. CiliumBGPNodeConfigOverride: Defines node-specific BGP configuration to provide a finer control. Here is a sample BGP configuration - you can find it in the containerlab examples in this repo. For those already running the existing BGPv1 feature: note that the v1 APIs will still be available in Cilium 1.16 but they will eventually be deprecated. Migration recommendations and tooling to help you move to v2 are on the roadmap. We recommend you start any new BGP deployments with the new v2 APIs. To learn more, visit the docs. BGP ClusterIP advertisement In addition to the new BGP APIs, Cilium 1.16 introduces support for new service advertisements. In prior releases, Cilium BGP could already announce the PodCIDR prefixes (for various IPAM scenarios) and LoadBalancer IP services. In 1.16, ExternalIP and ClusterIP Services can also be advertised. The latter might seem like an anti-pattern: ClusterIP Services are designed for internal access only. But there were 2 reasons why this feature was requested: Many users are migrating from other CNIs to Cilium, and some CNIs already support ClusterIP advertisements. ClusterIP services automatically get a DNS record like svc.namespace.svc.cluster.example. So, by synchronizing your Kubernetes Services upstream, you could access your services via their name from outside the cluster. In the demo below, we start before configuring the BGP session. You can see the deathstar service's IP, label, and the BGP config. Note how we now advertise ClusterIP Services but only those with the empire label. We end by checking that the BGP sessions have come up and that the backbone router can see the prefix. If you’d like to learn more, read the docs. Node IPAM Service LB For users that cannot use BGP or Cilium’s L2 Announcement (a feature particularly appreciated for homelabs and a great replacement for MetalLB), Cilium 1.16 is introducing another alternative to access services from outside the cluster: Node IPAM Service LB. Similar to the ServiceLB feature available for the lightweight distribution K3S, Node IPAM Service LB assigns IP Addresses from the nodes themselves to the Kubernetes Services. Enable the node IPAM feature with the nodeIPAM.enabled=true flag and make sure your Service has the loadBalancerClass set to io.cilium/node: Once creation of the service, it will receive an IP address from the node itself. This option is likely to be useful in small environments. To learn more about this feature, check out the documentation. Multicast datapath Multicast is a delivery mechanism popular in traditional IP networking. It provides a smarter and more economical method to deliver packets to a subset of interested parties. Perfect for publishers/subscribers-based applications, it is common for financial applications. Cilium 1.16 supports a multicast datapath, enabling Kubernetes users to deploy applications and benefit from multicast’s efficiencies. To configure multicast on Cilium’s open source edition, you will need to define the multicast group (represented by IP addresses in the Class D reserved range 224.0.0.0 to 239.255.255.255) and the nodes that might run subscribers Pods. This can be done by manually setting them on the relevant Cilium agents, using the cilium-dbg CLI. To learn more about it, check out the documentation. For a fully-managed CRD-based experience, you can instead use Isovalent Enterprise for Cilium. You can learn more about it in a couple of recent blog posts: Isovalent Enterprise for Cilium 1.15: eBPF-based IP Multicast, BGP support for Egress Gateway, Network Policy Change Tracker, and more! Enabling Multicast securely with IPSec in the cloud native landscape with Cilium – Isovalent Per-pod fixed MAC address Some applications require software licenses to be based on network interface MAC addresses. With Cilium 1.16, you will be able to set a specific MAC address for your pods, which should make licensing and reporting easier. Simply add a specific annotation to your pod with the MAC address value: For more information, check out the docs or watch the very short demo below: Service mesh & Ingress/Gateway API Gateway API GAMMA support The Cilium Service Mesh announcement back in 2021 had wide ramifications. It made our industry entirely reconsider the concept of a service mesh and reflect on the widely-accepted sidecar-based architecture. Why did we need a service mesh in the first place? Was it for traffic observability? To encrypt the traffic within our cluster? Ingress and L7 load-balancing? And do we really need a sidecar proxy in each one of our pods? It turns out that Cilium could already do a lot of these things natively: network policies, encryption, observability, tracing. When Cilium added support for Ingress and Gateway API to manage traffic coming into the cluster (North-South), it further alleviated the need to install and manage additional third-party tools ; simplifying the life of platform operators. One of the remaining areas of improvements for Cilium Service Mesh capabilities was traffic management within the cluster: it was possible through customizing the onboard Envoy proxy but it required advanced knowledge of the proxy. With Cilium 1.16, Cilium Gateway API can now be used for sophisticated East-West traffic management - within the cluster - by leveraging the standard Kubernetes Gateway API GAMMA. GAMMA stands for "Gateway API for Mesh Management and Administration". It provides a consistent model for east-west traffic management for the cluster, such as path-based routing and load-balancing internally within the cluster. Let’s review a GAMMA configuration: You will notice that, instead of attaching a route to a (North/South) Gateway like we’ve done so far when using Gateway API for traffic entering our cluster, we can now attach the route to a parent Service, called echo, using the parentRefs field. Traffic bound to this parent service will be intercepted by Cilium and routed through the per-node Envoy proxy. Note how we will forward traffic to the /v1 path to the echo-v1 service and the same for v2. This is how we can, for example, do a/b or green/blue canary testing for internal apps. To learn more, try the newly updated Advanced Gateway API lab, read to the docs or watch this video: Gateway API enhancements In addition to GAMMA support, Cilium 1.16's Gateway API implementation has been boosted with multiple enhancements: Gateway API 1.1 support: in Gateway API 1.1, several features are graduating to Standard Channel (GA), notably including support for GAMMA (mentioned above) and GRPCRoute (supported in Cilium since Cilium 1.15). Features on the Standard release channel denotes a high level of confidence in the API surface and provides guarantees of backward compatibility. New protocol options support: Cilium 1.16 Gateway API now support new protocol options: proxyProtocol: Some Load Balancing solutions use the HAProxy Proxy Protocol to pass source IP information along. With this new feature, Cilium will be able to pass PROXY protocol and will provide another option to preserve the source IP (another one is highlighted below). ALPN: Application-Layer Protocol Negotiation is a TLS extension required for HTTP/2. As gRPC is built on HTTP/2, when you enable TLS for gRPC you will also need ALPN to negotiate whether both client and server support HTTP/2. appProtocol: Kubernetes 1.20 introduced appProtocol support for Kubernetes Services, enabling users to specify an application protocol for a particular port. Local ExternalTrafficPolicy support for Ingress/Gateway API: When external clients access applications running in your cluster, it's sometimes useful to preserve the original client source IP for various reasons such as observability and security. Kubernetes Services can be configured with the externalTrafficPolicy set to Local to ensure the client source IP is maintained. In Cilium 1.16, the Cilium-managed Ingress/Gateway API LoadBalancer Services' external traffic policies can be configured globally via Helm flags or via dedicated Ingress annotation. Envoy enhancements Every Cilium release brings improvements to its usage of Envoy, the lightweight cloud native proxy. Envoy has been a core component of Cilium’s architecture for years and has always been relied upon to provide Layer 7 functionalities to complement eBPF’s L3/L4 capabilities. Cilium 1.16 introduces some subtle changes to Envoy’s use within Cilium: Envoy as a DaemonSet is now the default option: Introduced in Cilium 1.14, the Envoy DaemonSet deployment option was introduced as an alternative to embedding Envoy within the Cilium agent. This option decouples Envoy from the Cilium agent, providing more opacity between the Cilium and Envoy lifecycles. In Cilium 1.16, Envoy as a DaemonSet is now the default for new installations. To learn more, watch the video above or check out the documentation. Host Network mode & Envoy listeners on subset of nodes Host network mode allows you to expose the Cilium Gateway API Gateway directly on the host network. This is useful in cases where a LoadBalancer Service is unavailable, such as in development environments or environments with cluster-external load balancers. Alongside this feature, you can use a new option to only expose the Gateway/Ingress functionality on a subset of nodes, rather than on all of them. This will deploy the Gateway API Envoy listener only on the Cilium Nodes matching the configured labels. An empty selector selects all nodes and continues to expose the functionality on all Cilium nodes. Note both of these features are also available for the Cilium Ingress Controller. To learn more, consult the Cilium Gateway API docs and Cilium Ingress docs. Security Port Range support in network policies Cilium 1.16 supports a long-awaited feature: support for port ranges in network policies. Before this, network policies would require you to list ports one by one in your network rule, even if they were contiguous. The Port Range feature, announced in Kubernetes 1.21 and promoted to Stable in Kubernetes 1.25, lets you target a range of ports instead of a single port in a Network Policy, using the endPort keyword. In the demo below, we start by verifying we have access from a client across to 3 servers listening on ports 8080, 8081 and 8082. It's successful until we deployed a Cilium Network Policy only allowing the 8080-8081 range, thereafter access to 8082 is blocked. Access is successful again after expanding the range to 8080-8082. This feature is available with Kubernetes Network Policies and the more advanced Cilium Network Policies. Network policy validation status Sometimes, Cilium cannot detect and alert when Network Policies are semantically incorrect until after deployment. The demo below shows that even though our network policy is missing a field, it seems accepted. The only way to find out it was rejected is by checking the verbose agent logs, which is not an ideal user experience. Cilium 1.16 adds information about the network policy validation condition in the operator. This means that, as you can see in the demo, you can easily find the status of the policy - valid or invalid - by checking the object with kubectl describe cnp. Control Cilium network policy default deny behaviour Initially introduced as an Enterprise feature in Isovalent Enterprise for Cilium 1.15, this feature has now been open-sourced. This new feature controls the behavior of default deny for network policies, and allows platform owners to safely roll out new broad-based network policies, either namespace or cluster-wide, without the risk of disrupting existing traffic. The below example policy is created to intercept all DNS traffic for observability purposes. However, the enableDefaultDeny for both ingress and egress traffic is set to false. This ensures no traffic is blocked, even if this policy is the first to apply to a workload. A network or security administrator can apply this policy to the platform, without the risk of denying legitimate traffic. The video below captures this example policy and shows the operational risk associated with applying a broad network policy to existing workloads. It then shows how to avoid that risk by using this new EnableDefaultDeny feature in Cilium Network Policies. CIDRGroups support for Egress and Deny rules Another feature designed to simplify the creation and management of network policy: in Cilium 1.16, you can now use CiliumCIDRGroups in Egress and Ingress/Egress Deny Policy rules. A CiliumCIDRGroup is a list of CIDRs that can be referenced as a single entity in Cilium Network Policies and Cilium Cluster Network Policies. To consume this enhancement, first create a CIDR Group as per the YAML example below: You can then use the meta name from the CiliumCIDRGroup within the policy as below for Egress Policies: Load "default" policies from filesystem Over the past few years, Cilium has become most cloud providers' networking platform of choice. After Google (GKE) and Amazon (EKS-A), it was Microsoft that, in 2022, announced that Cilium would be the preferred CNI for AKS (Azure Kubernetes Services). Microsoft has also contributed several features to Cilium, including this latest enhancement to load network policies from the filesystem. This feature will be particularly popular for cloud providers and operators of multi-tenant platforms. Fetching the YAML manifests from a folder lets you load common network policies (Cilium Network Policies and Clusterwide Cilium Network Policies) on Cilium deployments. Copy policies on the underlying disk, mount the directory onto the Cilium agent, activate the feature, and the policies will be enforced. This enables administrators to configure certain broader policies (designed, for example, to alllow/block traffic to certain internal endpoints) without them appearing as policy resources in Kubernetes. Indeed, even though the policy is enforced (as the cilium-dbg policy get command output below shows), kubectl get cnp does not return any output. To learn more, check the docs. Select nodes as the target of Cilium Network Policies The feature started as a CFP by Ondrej Blazek at Seznam, to provide Cilium Network Policies with the ability to use Node Labels with the policy selector statements. Before the Cilium 1.16 release, users who needed to filter traffic from/to Kubernetes nodes in their cluster using Network Policies would need to use either the “remote-node” entity or a CIDR-based policy. Using either of these methods had its pitfalls, such as remote-node selecting all nodes in a cluster mesh configuration. Before Cilium 1.16, to target nodes in a Cilium Network Policy, you would use a policy as the example below: Now, with this new feature, which allows nodes to be selectable by their labels instead of CIDR and/or remote-node entity, you can configure the helm value nodeSelectorLabels=true and use a policy such as the example below, which allows pods with the label env: prod, to communicate with control-plane nodes: The following extraVar can be configured to select which node labels are used for node identity. Day 2 operations and scale Improved DNS-based network policy performance One of the reasons Cilium Network Policies are so popular with cluster administrators are their ability to filter based on Fully-Qualified Domain Names (FQDN). DNS-aware Cilium Network Policies enable users to permit traffic based on specific FQDNs, for example, by using the toFQDN selector in network policies to only allow traffic to, for example, my-remote-service.com. It’s supported by using a DNS proxy that is responsible for intercepting DNS traffic and recording IP addresses seen in the responses. DNS-Based Network policies are extremely useful when implementing API security; improving this feature's performance ultimately offers a better developer and end-user experience. With this latest release, Cilium 1.16 has significantly improved CPU and memory usage and, even more crucially, up to 5x reduction in tail latency. The implementation of toFQDNs selectors in policies has been overhauled to improve performance when many different IPs are observed for a selector: Instead of creating cidr identities for each allowed IP, IPs observed in DNS lookups are now labelled with the selectors toFQDNs matching them. This reduces tail latency significantly for FQDNs with a highly dynamic set of IPs, such as e.g. content delivery networks and cloud object storage services. As you can see from the below graphs, with these enhancements, there is a 5x improvement in tail latency. Elegantly, upon upgrade or downgrade, Cilium will automatically migrate its internal state for toFQDNs policy entries. For more information, consult the upgrade guide. That’s not the only performance improvement in Cilium 1.16. Let’s dive into another enhancement. New ELF loader logic Theoretically, we must recompile the BPF for an endpoint whenever its configuration changes. Compiling is a fairly expensive process, so a mechanism named “ELF templating / substitution” had been developed to avoid recompilation in the most common cases. This substitution process was, however, sub-optimal. In Cilium 1.16, it has been improved, resulting into noticeable memory gains: KVStoreMesh default option for ClusterMesh Introduced in Cilium 1.14, KVStoreMesh’s concept was inspired by the wider Cilium user community. Trip.com, the leading Chinese travel services provider, had been running Cilium Cluster Mesh at a considerable scale and pushed the limits of its architecture. Trip.com eventually forked Cilium and adapted the model to overcome some of the reliability issues they saw at their hyperscale. The solution was to have local caches (running in key-value stores) storing information about remote meshed clusters. This benefits not only extra large cluster mesh deployments but also smaller ones as it reduces the overall control plane resource utilization. Since its introduction in Cilium 1.14, KVStoreMesh has received strong customer feedback and will become the default option from Cilium 1.16 onwards (note that you can still opt out and select the traditional method if you wish to do so). For more information, check out the documentation. Hubble and observability CEL filters support Common Expression Language (CEL) is an implementation of common semantics for expression evaluation. It is designed to be fast, portable, and safe for execution with performance-critical applications. Today, CEL can be found in cloud native technologies such as the Kubernetes API to declare validation rules, policy rules and other constraints and conditions. The motivation behind bringing CEL to Hubble is to support more complex conditions that cannot currently be expressed using the existing flow filters. Currently, the Hubble observe command from the Hubble CLI is constrained by the limit that it can only use one AND filter against all parameters. CEL also allows you to compare flow fields to others, which is not currently possible with the existing Flow Filters. The Hubble CLI now supports the additional argument –cel-expression filter. Below is an example of how to use this new feature. This filter is designed to show flows where the destination port is either “8080” or “443”, and the source of these flows is not originating from an endpoint with the label “reserved:host”. By filtering out this special identity, we will not be shown any flows of traffic that originate from or are designated to one of the local host IPs. Below is a more complex example: filtering for all flows where the source namespace does not match the destination namespace. This allows us to see all traffic that leaves a namespace. The final component of the filter is removing any flows whereby the source has the identification label of one of the reserved identities, such as world. For more information, check out the pull request. Filtering Hubble flows by node labels Hubble now captures the node labels for flows, allowing you to filter by particular nodes in your cluster. This can be helpful for Kubernetes deployments across availability zones and help to identify cross-availability zone traffic between workloads regardless of their source or destination namespaces, for example. With Hubble providing this level of visibility, it will help platform owners identify misconfigured services that allow cross-AZ traffic and cause an increase in costs from the cloud provider. Typically, most deployments should be set to local preferred traffic, with remote traffic set as a fallback, meaning all traffic should be localised within the same AZ, only traversing the AZ when the service fails. Below is a recording showing how to use these new filters to view cross-availability zone traffic. Improvements for egress traffic path observability The Cilium Egress Gateway feature allows you to select defined exit routes on your network for your containers. This feature is particularly useful when traffic leaving your cluster transits via external traffic management devices. These devices need to understand the specific endpoints from which traffic originates. The Egress Gateway feature works by implementing deterministic source NAT for all traffic that traverses through a node; allocating a predictable IP to traffic coming from a particular Pod or a specific namespace. In Cilium 1.16, several enhancements are implemented to aid better observability for traffic using Egress Gateway nodes. The first is the creation of additional metrics within the Cilium Agent, which tracks the number of allocated ports in each NAT connection tuple: {source_ip, endpoint_ip, endpoint_port}. These additional statistics help monitor the saturation of the endpoint connections based on the allocation of source ports. The Cilium Metric nat_endpoint_max_connection has also been implemented to monitor these statistics in your alerting platform. The Cilium documentation has also been updated to include a troubleshooting section for Egress Gateway explaining these new statistics and metrics in further detail. Hubble flow data has been further updated with Egress Gateway traffic paths in mind. Earlier in this blog post, we’ve already covered the ability to capture and filter traffic flows based on Node Label, so let’s look at two further new filters. In the below Hubble flow output, the pod xwing is contacting an external device on IP address 172.18.0.7; this traffic is subject to address translation by the Egress Gateway. The new fields implemented are: IP.source_xlated node_labels interface Here is a JSON output of a flow recorded by Hubble: These additional fields are further combined with updates to the Hubble CLI, which includes the following new arguments; In the example below, we filter all traffic using the specific node label in our environment to identify egress gateway nodes. The traffic has been translated to the IP address 172.18.0.42. In the recording below, you can see these new features to extend Egress Gateway Observability in action. K8S event generation on packet drop detection Status: Alpha This release extends the visibility of flow information from Hubble to Kubernetes Pod events. Platform consumers need additional access to the platform components to view any data generated by Cilium and Hubble. With this new feature, Cilium inserts packet drop information caused by Cilium Network Policies into the Kubernetes Pod events. Please note this feature is currently Alpha. Below is an example output of the enhanced Kubernetes Pod Event showing the packet drop. Enabling this feature is as simple as the below helm values; Conclusions Since the previous release, many new end users have stepped forward to tell their stories of why and how they're using Cilium in production. These use cases cover multiple industries: software (G Data CyberDefense and WSO2 ), retail (Nemlig.com), news and media (SmartNews), finance (Sicredi, PostFinance, Rabobank) and cloud providers (DigitalOcean). Read below some of the testimonies: To learn more from the developers involved in creating the features covered in this blog post, why don't you join Duffie in the Cilium 1.16 eBPF and Cilium eCHO show: Getting started To get started with Cilium, use one of the resources below: Learn more about Cilium (Introduction, tutorials, AMAs, …) Get involved in the Cilium community Join Cilium Slack to interact with the Cilium community Previous releases Cilium 1.15 – Gateway API 1.0 Support, Cluster Mesh Scale Increase, Security Optimizations and more! Cilium 1.14 – Effortless Mutual Authentication, Service Mesh, Networking Beyond Kubernetes, High-Scale Multi-Cluster, and Much More Cilium 1.13: Gateway API, mTLS datapath, Service Mesh, BIG TCP, SBOM… Cilium 1.12: Ingress, Multi-Cluster, Service Mesh, External Workloads, and much more Cilium 1.11: Service Mesh Beta, Topology Aware Routing, OpenTelemetry, … Cilium 1.10: WireGuard, BGP Support, Egress IP Gateway, New Cilium CLI, XDP Load Balancer, Alibaba Cloud Integration and more Cilium 1.9: Maglev, Deny Policies, VM Support, OpenShift, Hubble mTLS, Bandwidth Manager, eBPF Node-Local Redirect, Datapath Optimizations, and more