The debate around Ingress or Gateway API is settled. Gateway API is the better of the two when providing a more flexible and extensible API for more advanced and robust traffic management and control in Kubernetes. Some users are either using Ingress as the standard API object with Cilium or rolling out Gateway API with cloud providers like AWS, Azure, etc. Imagine scrolling through multiple documents and links to figure out which annotation would serve the need for your use-case. This tutorial shows how to use cloud-specific annotations for the Load Balancer service for Ingress & Gateway API on an Elastic Kubernetes Service and Azure Kubernetes Service cluster. Pre-requisites The following prerequisites need to be taken into account before you proceed with this tutorial: Access to AWS. Create a new account for free. The Cilium operator requires the following EC2 privileges to perform ENI creation and IP allocation. Ensure you have a cluster IAM role if you’re going to create your cluster with eksctl. To ensure that Cilium works properly in an environment that requires firewall rules to enable connectivity, you must add the respective firewall rules. You should have an Azure Subscription Install Azure CLI Install kubectl Install Helm Install eksctl Install awscli Cilium CLI (optional): Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s). Gateway API related CRD’s should be installed before installing Cilium. Demo environment In this blog, we will be testing the respective annotations for Ingress and Gateway API on Azure Kubernetes Service (AKS) and Elastic Kubernetes Service (EKS) based clusters. AKS Create an AKS cluster with BYOCNI. Ensure that the AKS cluster is created with a Service Principal with the correct roles and permissions. Set the subscription Choose the subscription you want to use if you have multiple Azure subscriptions. Replace SubscriptionName with your subscription name. You can also use your subscription ID instead of your subscription name. AKS Resource Group Creation Create a Resource Group Disable Kube-Proxy on the AKS cluster. Create a file by the name of kube-proxy.json with the following contents. AKS Cluster creation Pass the --network-plugin parameter with the parameter value of none. Also, observe that we will be disabling kube-proxy. Set the Kubernetes Context Log in to the Azure portal, browse Kubernetes Services>, select the respective Kubernetes service created (AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context. [code snippet] Cluster status check Check the status of the nodes and make sure they are in a “Ready” state. Install Isovalent Enterprise for Cilium Users can contact their partner Sales/SE representative(s) at sales@isovalent.com to access the requisite documentation and learn how to install Isovalent Enterprise for Cilium on an AKS cluster with BYOCNI as the network plugin. Validate Cilium version Check the version of cilium with cilium version: Cilium Health Check cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status: EKS Create a ClusterConfig file to create an EKS cluster (e.g., default-ipv4.yaml) in ENI mode. Create the Cluster Check the status of the nodes and make sure they are in a “Ready” state. Patch the AWS daemonset When running in ENI mode, Cilium Enterprise manages ENIs and IP address allocation in place of the AWS VPC CNI plugin. Because of this, and to avoid conflicting behavior, the kube-system/aws-node DaemonSet must be deleted or, at least, prevented from running. To achieve this in a non-destructive way, run the following command: Install Isovalent Enterprise for Cilium Users can contact their partner Sales/SE representative(s) at sales@isovalent.com to access the requisite documentation and learn how to install Isovalent Enterprise for Cilium on an EKS cluster. Validate Cilium version Check the version of cilium with cilium version: Cilium Health Check cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status: What are annotations in Kubernetes? The Service API, part of Kubernetes, is an abstraction to help you expose groups of Pods over a network. Each Service object defines a logical set of endpoints (usually these endpoints are Pods) along with a policy about how to make those pods accessible. If your workload speaks HTTP, you might choose to use an Ingress to control how web traffic reaches that workload. Ingress is not a Service type, but it acts as the entry point for your cluster. An Ingress lets you consolidate your routing rules into a single resource so that you can expose multiple components of your workload, running separately in your cluster, behind a single listener. The Gateway API for Kubernetes provides extra capabilities beyond Ingress and Service. You can add Gateway to your cluster – it is a family of extension APIs, implemented using CustomResourceDefinitions– and then use these to configure access to network services that are running in your cluster. Services can be annotated to automatically select available load balancers and to provide specific availability sets that host them. Elastic Kubernetes Service (EKS) and Azure Kubernetes Service (AKS) have a respective way of dealing with annotations (listed below) that can be used with Ingress/Gateway API and have been validated for Cilium. Note: All these annotations have been extensively tested, and the test results for all cannot be captured, keeping in mind the brevity of the blog. How do you set up Cloud Annotations with Gateway API? Gateway API is a Kubernetes SIG-Network subproject to design a successor for the Ingress object. It is a set of resources that model service networking in Kubernetes and is designed to be role-oriented, portable, expressive, and extensible. Sample helm config file to enable gateway-api in your EKS or AKS cluster. Create a demo app to test the gateway API and name it basic-http.yaml. Apply the manifest for gateway API. Check the status of the service that is created. Check the status of the gateway that is created When the Gateway is functional, you can check the routes to verify if they are configured correctly. Send requests to the Gateway IP (Load Balancer IP) from a test pod. How do you set up Cloud Annotations with Ingress? Ingress makes your network service ( e.g, HTTP or HTTPS) available using a protocol-aware configuration mechanism that understands web concepts like URIs, hostnames, paths, and more. Ingress lets you map traffic to different backends based on rules you define via the Kubernetes API. Cilium uses the standard Kubernetes Ingress resource definition, with an ingressClassName of cilium. This can be used for path-based routing and TLS termination. The ingress controller creates a service of the LoadBalancer type for which annotations can be used in cloud provider environments (AKS, EKS, GKE, etc). Cilium allows you to specify the load balancer mode for the Ingress resource: dedicated: The Ingress controller will create a dedicated load balancer for the Ingress. shared: The Ingress controller will use a shared load balancer for all Ingress resources. For Testing IPv6 Ingress/Gateway API in AKS with Cilium, make sure the service is created as IPv6 Ingress/Gateway API in EKS with ENI mode is a roadmap item. Sample helm config file to enable dedicated ingress in your EKS or AKS cluster. Create a demo app to test the ingress and name it ingress.yaml. Apply the manifest for ingress. Check the status of the service that is created. Check the status of the ingress that is created. Send requests to the Ingress IP (Load Balancer IP) from a test pod. Sample helm config file to enable shared ingress in your EKS or AKS cluster. Applying a similar manifest as above with distinct names, two Ingress services are sharing the same Load Balancer IP. Note- With any annotation being added/changed to a shared ingress, cilium-operator and cilium-agent will have to be restarted. Conclusion Hopefully, this post gave you an idea of how to use annotations in your AKS and EKS environments with Ingress or Gateway API with Cilium. You can also see these in action in our labs: Cilium Gateway API Advanced Gateway API Use Cases References Azure Load Balancer Annotations AWS Load Balancer Annotations