Kubernetes changes the way we think about networking. In an ideal Kubernetes world, the network would be flat, and the Pod network would control all routing and security between the applications using Network Policies. In many Enterprise environments, though, the applications hosted on Kubernetes need to communicate with workloads outside the Kubernetes cluster, subject to connectivity constraints and security enforcement. Because of the nature of these networks, traditional firewalling usually relies on static IP addresses (or at least IP ranges). This can make it difficult to integrate a Kubernetes cluster, which has a varying and, at times, dynamic number of nodes, into such a network. Cilium’s Egress Gateway feature changes this by allowing you to specify which nodes should be used by a pod to reach the outside world. This blog post will walk you through deploying Cilium and Egress Gateway in AKS (Azure Kubernetes Service) using BYOCNI as the network plugin. What is an Egress Gateway? The egress gateway feature allows redirecting traffic originating in pods destined to specific CIDRs outside the cluster to be routed through particular nodes. When the egress gateway feature is enabled and egress gateway policies are in place, packets leaving the cluster are masqueraded with selected, predictable IPs associated with the gateway nodes. This feature can be used with legacy firewalls to allow traffic to legacy infrastructure only from specific pods within a given namespace. These pods typically have ever-changing IP addresses. Even if masquerading were to be used to mitigate this, the IP addresses of nodes can also change frequently over time. For example, with the following resource: All pods that match the label=egress-node will be routed through the 192.168.11.4 IP when they reach out to any address in the 192.168.11.0/24 IP range, which is outside the cluster. What is Isovalent Enterprise for Cilium? Isovalent Enterprise for Cilium is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns. Why Isovalent Enterprise for Cilium? While Egress Gateway in Cilium is a great step forward, most enterprise environments should not rely on a single point of failure for network routing. For this reason, Isovalent introduced Egress Gateway High Availability (HA), which supports multiple egress nodes. The multiple egress nodes can be configured using a egressGroups parameter in the IsovalentEgressGatewayPolicy resource specification that we will detail in Scenario 2 in the tutorial below. Whenever one of these two groups doesn’t contain healthy nodes, traffic will only be routed through the other one. If both groups contain healthy nodes, traffic will be randomly distributed between them. Pre-Requisites The following prerequisites need to be taken into account before you proceed with this tutorial: An Azure account with an active subscription- Create an account for free Azure CLI version 2.48.1 or later. Run az --version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI. If using ARM templates or the REST API, the AKS API version must be 2022-09-02-preview or later. The kubectl command line tool is installed on your device. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.26, you can use kubectl version 1.25, 1.26, or 1.27 with it. To install or upgrade kubectl, see Installing or updating kubectl. Install Cilium CLI. Limitations to keep in mind? You must remember certain limitations, which will be added over time. The Egress gateway feature is partially incompatible with L7 policies. Specifically, when an egress gateway policy and an L7 policy both select the same endpoint, traffic from that endpoint does not go through the egress gateway, even if the policy allows it. Egress Gateway is incompatible with Isovalent’s Cluster Mesh feature. Which network plugin can I use for Egress Gateway in AKS? Considering two scenarios, we will create an Azure Kubernetes (AKS) Cluster with Bring Your Own CNI (BYOCNI) as the network plugin for this tutorial. Scenario 1- Egress Gateways in a single Availability Zone. Pre-Requisites: The AKS cluster is created in VNET A, subnet A The Egress Gateway is created in VNET A, subnet B VNET= 192.168.8.0/22 Subnet A= 192.168.10.0/24 Subnet B= 192.168.11.0/24 A test VM is created in VNET A, subnet B Set the subscription Choose the subscription you want to use if you have multiple Azure subscriptions. Replace SubscriptionName with your subscription name. You can also use your subscription ID instead of your subscription name. AKS Cluster creation Create an AKS cluster with the network plugin as BYOCNI. Note- You can also create an AKS cluster with BYOCNI using Terraform. Create an unmanaged AKS nodepool in a different subnet. Create an AKS nodepool in the egressgw-subnet (created in the previous step). Assign a label to the unmanaged nodepool Create a node pool with a label and specify a name for the --name parameters and labels for the --labels Parameter. Labels must be a key/value pair and have a valid syntax. Check the status of the nodes. Note- this doesn’t create a new NIC. It means traffic from the client pod is EGW-redirected to the egress-node: "true" node’s eth0 192.168.11.5, and from there, it’s also automatically NATed to the node's assigned public IP. Install Isovalent Enteprise for Cilium Users can contact their partner Sales/SE representative(s) at sales@isovalent.com to access the requisite documentation and how to install Isovalent Enteprpise for Cilium on an AKS cluster with BYOCNI as the network plugin. Make sure that the following Helm values are taken into account while enabling the Egress Gateway feature: Restart Cilium Operator and Cilium Daemonset Restart the cilium operator and cilium daemonset for egress gateway changes to take effect. Check the status of the pods. Create an Egress Gateway Policy The API provided by Isovalent to drive the Egress Gateway feature is the IsovalentEgressGatewayPolicy resource. The selectors field of an IsovalentEgressGatewayPolicy resource is used to select source pods via a label selector. This can be done using matchLabels: One or more destination CIDRs can be specified with destinationCIDRs: The group of nodes that should act as gateway nodes for a given policy can be configured with the egressGroups field. Nodes are matched based on their labels, with the nodeSelector field: Sample policy as below: Testing Egress Gateway Deploy a client pod and apply the IsovalentEgressGatewayPolicy, and observe that the pod’s connection gets redirected through the Gateway node. The client pod gets deployed to one of the two nodes (managed), and the IEGP (Isovalent Egress Gateway Policy) selects one or both the nodes ( depending on the egress gateway IPs specified) as the Gateway node. Sample client pod yaml: Create the client pod and check that it's up and running and pinned on one of the worker nodes as specified in the yaml file for the client pod. Apply an Egress Gateway Policy Apply an Egress Gateway Policy Label the Egress Gateway Node To let the policy select the node designated as the Egress Gateway, apply the label, egress-node:true to it: Create a test VM in the Egress Gateway subnet. Create a VM in the same subnet as Egress Gateway and run a simple service on port 80 (like NGINX) that will respond to traffic sent from a pod on one of the worker nodes. Test VM IP, in this case, is 192.168.11.4 Traffic Generation (towards the server in Egress GW subnet) Send traffic toward the test VM. Traffic Generation (outside of the cluster towards the Internet) Send traffic to a public service. Note the IP it returns is the egress gateway node’s Public IP. Take a tcpdump from one of the egress gateway nodes. Install tcpdump on the egress gateway node via apt-get install tcpdump As you can see 10.0.0.165 is the client-pod IP that the egress gateway node is receiving packets from and 192.168.11.5 is the egress gateway node’s eth0 IP address. Scenario 2- Egress Gateways in a Multi-Availability Zone environment. Geo redundancy across availability zones is a must, and combined with HA for the Egress GW, it is a solution that enterprises are always willing to consider. Pre-Requisites: The AKS cluster is created in VNET A, subnet A The Egress Gateway is created in VNET A, subnet B VNET= 192.168.8.0/22 Subnet A= 192.168.10.0/24 Subnet B= 192.168.11.0/24 A test VM is created in VNET A, subnet B Set the subscription Choose the subscription you want to use if you have multiple Azure subscriptions. Replace SubscriptionName with your subscription name. You can also use your subscription ID instead of your subscription name. AKS cluster creation with nodepools across AZ’s Create an AKS cluster with the network plugin as BYOCNI and nodepools across different Availability Zones. Create an unmanaged AKS nodepool in a different subnet. Create an AKS nodepool in the egressgw-subnet (created in the previous step). Assign a label to the unmanaged nodepool Create a node pool with a label and specify a name for the --name parameters and labels for the --labels Parameter. Labels must be a key/value pair and have a valid syntax. Check the status of the nodes. Note- this doesn’t create a new NIC. It means traffic from the client pod is EGW-redirected to the egress-node: "true" node’s eth0 192.168.11.5 or 192.168.11.4, and from there, it’s also automatically NATed to the node assigned public IP. Check that all nodes have been created in different Availability Zones. Install Isovalent Enteprise for Cilium Users can contact their partner Sales/SE representative(s) at sales@isovalent.com to access the requisite documentation and how to install Isovalent Enteprpise for Cilium on an AKS cluster with BYOCNI as the network plugin. Make sure that the following Helm values are taken into account while enabling the Egress Gateway feature: Restart Cilium Operator and Cilium Daemonset Restart the cilium operator and cilium daemonset for egress gateway changes to take effect. Check the status of the pods. Create an Egress Gateway Policy The API provided by Isovalent to drive the Egress Gateway feature is the IsovalentEgressGatewayPolicy resource. Testing Egress Gateway Deploy a client pod and apply the IsovalentEgressGatewayPolicy, and observe that the pod’s connection gets redirected through the Gateway node. The client pod gets deployed to one of the two nodes (managed), and the IEGP (Isovalent Egress Gateway Policy) selects one or both the nodes ( depending on the egress gateway IPs specified) as the Gateway node. Sample client pod yaml: Create the client pod and check that it's up and running and pinned on one of the worker nodes as specified in the yaml file for the client pod. Apply an Egress Gateway Policy Apply an Egress Gateway Policy Label the Egress Gateway Node To let the policy select the node designated as the Egress Gateway, apply the label, egress-node:true to it: Create a test VM in the Egress Gateway subnet. Create a VM in the same subnet as Egress Gateway and run a simple service on port 80 (like NGINX) that will respond to traffic sent from a pod on one of the worker nodes. Test VM IP, in this case, is 192.168.11.4 Traffic Generation (towards the server in Egress GW subnet) Send traffic toward the test VM. Traffic Generation (outside of the cluster towards the Internet) Send traffic to a public service. Note the IP it returns is the egress gateway node’s Public IP. Take a tcpdump from one of the egress gateway nodes. Install tcpdump on the egress gateway node via apt-get install tcpdump. As you can see 10.0.0.225 is the client-pod IP that the egress gateway node is receiving packets from and 192.168.11.5 is the egress gateway node’s eth0 IP address. Availability Zone Affinity It is possible to control the AZ affinity of the egress gateway traffic with azAffinity. This feature relies on the well-known, topology.kubernetes.io/zone node label to match or prefer gateway nodes within the same AZ of the source pods ("local" gateways) based on the configured mode of operation. The following modes of operation are available: disabled: This mode uses all the active gateways available, regardless of their AZ. This is the default mode of operation. By taking a tcpdump from both the egress nodes, we can see that the traffic flows across both the egress nodes. sample egress policy for the mode of operation localOnly: This mode selects only local gateways. If no local gateways are available, traffic will not pass through the non-local gateways and will be dropped. By taking a tcpdump from both the egress nodes, we can see that the traffic flows across one of the local gateways. sample egress policy for the mode of operation localOnlyFirst: This mode selects only local gateways as long as at least one is available in a given AZ. When no more local gateways are available, non-local gateways will be selected. By taking a tcpdump from both the egress nodes, we can see that the traffic flows across one of the local gateways. sample egress policy for the mode of operation localPriority: this mode selects all gateways, but local gateways are picked up first. In conjunction with maxGatewayNodes, this can prioritize local gateways over non-local ones, allowing for a graceful fallback to non-local gateways in case the local ones become unavailable. By taking a tcpdump from both the egress nodes, we can see that the traffic flows across one of the local gateways. sample egress policy for the mode of operation How can you scale the Egress Gateway solution? With Isovalent Enterprise for the Cilium 1.16 release, we introduced the support for Egress Gateway IPAM and combined it with BGP to cater to the Egress GW scaling needs. We have also used Azure Route Server to demonstrate how Cilium can use the same Egress GW Node IP’s advertisement for multiple BGP Peers and how application VMs peering with the Azure Route Server from different VNETs can access the services. Pre-Requisites Azure Route Server has a limit of 8 BGP Peers. Application VM running on-premise. The Egress Gateway configuration is already described in this section. Note the node IPs of the Egress Gateways. These will be used to add as peers in Azure Route Server configuration. VNET for peering with the Azure Route Server for propagating routes towards an Application VM. Subnet for the Azure Route Server. This subnet will be in the same VNET where egress gateway nodes are also running. VNET Peering across the two VNETs (Azure Route Server and Application VM residing in an another VNET). Egress Gateway IPAM Egress Gateway IPAM allows users additional control of the IP address distribution to their Egress Gateway nodes. It removes the complexity of manually targeting each Egress Gateway node with a specific configuration for IP address management. The IPAM feature allows you to specify an IP pool in the IsovalentEgressGatewayPolicy from which Cilium leases egress IPs and assigns them to the selected egress interfaces. The IP pool should be specified in the egressCIDRs field of the IsovalentEgressGatewayPolicy and may be composed of one or more CIDRs: In the policy example below, you can see the new addition of the key egressCIDR. Log in to the nodes to observe the secondary IP on the NIC of the Egress Gateway Node(s). 10.100.255.48/32 10.100.255.49/32 Azure Route Server configuration Sign in to the Azure portal. In the search box at the portal’s top, enter route server, and select Route Server from the search results. On the Route Servers page, select + Create. Take note of the Azure Route Server peer IP addresses. 192.168.8.68 192.168.8.69 Azure Route Server peering with Cilium Once the deployment is complete, select Go to resource to go to the myRouteServer. Under Settings, select Peers. Select + Add to add a peer. 192.168.8.36 is the IP address of one of the Egress GW nodes. 192.168.8.37 is the IP address of one of the Egress GW nodes. On the Add Peer page, enter the following information: Name ASN IPv4 Address BGP support for Egress Gateway BGP is one of the most commonly used methods to connect Kubernetes clusters to existing networks and Cilium’s built-in implementation of BGP enables seamless connectivity between Kubernetes services and the existing network. This feature supports BGP in advertising the Egress NAT IPs to external routers. BGP support can be enabled by: Sample BGP peering Policy with Azure Route Server: Check the status of BGP peers across Cilium and Azure Route Server. Check if routes are being exchanged between Cilium and Azure Route Server. Common questions for Egress Gateway? How is the traffic encapsulated from the worker node to the egress node? The traffic is encapsulated from a worker node to an egress node regardless of the cluster’s routing mode, and in this case, the AKS cluster with BYOCNI uses VXLAN as the encapsulation. How can you find the identity of the source endpoint if the traffic is encapsulated? VNI of the VXLAN header equals the Identity of a source endpoint. In this case, the VNI maps to 53596 You can then track the identity using the Cilium CLI, which indicates that it’s the busybox Pod. How can you find the identity of the remote endpoint if the traffic is encapsulated? Traffic is encapsulated over VXLAN from the server to the busy box pod behind one of the worker nodes. In this case, VNI=6 is the identity of the server VM called remote-node by Cilium. You can then track the identity using the Cilium CLI, indicating it’s the remote node. How can you check the learned routes in the Azure Route Server? Using Azure CLI, you can check the routes that the Azure Route server has learned. How can you check the routes advertised by the Azure Route Server? You can use Azure CLI to check the routes that the Azure Route server has advertised (including the application VM in the subnet 192.168.40.0/22). Conclusion Hopefully, this post gave you a good overview of deploying Cilium and Egress Gateway in AKS (Azure Kubernetes Service) using BYOCNI as the network plugin. If you have any feedback on the solution, please share it with us. Talk to us, and let's see how Isovalent can help with your use case. Try it out Start with the Egress Gateway lab and explore Egress Gateway in action. Further Reading To dive deeper into the topic of Egress Gateway, check out these two videos: Cilium Tech talk series, Egress Gateway Egress Gateway HA: concept, requirements, demo