The debate around Ingress or Gateway API is settled. Gateway API is the better of the two when providing a more flexible and extensible API for more advanced and robust traffic management and control in Kubernetes. Some users are either using Ingress as the standard API object with Cilium or rolling out Gateway API with cloud providers like AWS, Azure, etc. Imagine scrolling through multiple documents and links to figure out which annotation would serve the need for your use-case. This tutorial shows how to use cloud-specific annotations for the Load Balancer service for Ingress & Gateway API on an Elastic Kubernetes Service and Azure Kubernetes Service cluster.
Pre-Requisites
The following prerequisites need to be taken into account before you proceed with this tutorial:
The Cilium operator requires the following EC2 privileges to perform ENI creation and IP allocation.
Ensure you have a cluster IAM role if you’re going to create your cluster with eksctl.
To ensure that Cilium works properly in an environment that requires firewall rules to enable connectivity, you must add the respective firewall rules.
Cilium CLI (optional): Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s).
Gateway API related CRD’s should be installed before installing Cilium.
In this blog, we will be testing the respective annotations for Ingress and Gateway API on Azure Kubernetes Service (AKS) and Elastic Kubernetes Service(EKS) based clusters.
Create an AKS cluster with BYOCNI.
Ensure that the AKS cluster is created with a Service Principal with the correct roles and permissions.
Set the subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
Replace SubscriptionName with your subscription name.
You can also use your subscription ID instead of your subscription name.
az account set --subscription SubscriptionName
AKS Resource Group Creation
Create a Resource Group
clusterName="byocni"resourceGroup="byocni"location="eastus"
az group create -l eastus -n byocni
Disable Kube-Proxy on the AKS cluster.
Create a file by the name of kube-proxy.json with the following contents.
Log in to the Azure portal, browse Kubernetes Services>, select the respective Kubernetes service created (AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
az aks get-credentials --name $clusterName --resource-group $resourceGroup
Cluster status check
Check the status of the nodes and make sure they are in a “Ready” state.
kubectl -n kube-system exec ds/cilium -- cilium status
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)KVStore: Ok Disabled
Kubernetes: Ok 1.29(v1.29.14)[linux/amd64]Kubernetes APIs: ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideEnvoyConfig", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumEnvoyConfig", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Secrets", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]KubeProxyReplacement: True [eth0 10.224.0.5 fe80::7e1e:52ff:fe7e:1a2b (Direct Routing)]Host firewall: Disabled
SRv6: Disabled
CNI Chaining: none
CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium: Ok 1.15.15-cee.1 (v1.15.15-cee.1-78ed04be)NodeMonitor: Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 7/254 allocated from 10.0.1.0/24,
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Host Routing: Legacy
Masquerading: IPTables [IPv4: Enabled, IPv6: Disabled]Controller Status: 47/47 healthy
Proxy Status: OK, ip10.0.1.161, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range: min 256, max 65535Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 18.63 Metrics: Disabled
Encryption: Disabled
Cluster health: 3/3 reachable (2025-03-27T12:26:17Z)Modules Health: Stopped(0) Degraded(0) OK(13)
Cilium Health Check
cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status:
kubectl -n kube-system exec ds/cilium -- cilium-health status
Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)Probe time: 2025-03-27T12:27:17Z
Nodes:
aks-nodepool1-23628489-vmss000002 (localhost):
Host connectivity to 10.224.0.5:
ICMP to stack: OK, RTT=362.204µs
HTTP to agent: OK, RTT=171.102µs
Endpoint connectivity to 10.0.1.35:
ICMP to stack: OK, RTT=351.804µs
HTTP to agent: OK, RTT=382.604µs
aks-nodepool1-23628489-vmss000000:
Host connectivity to 10.224.0.6:
ICMP to stack: OK, RTT=1.619216ms
HTTP to agent: OK, RTT=1.955319ms
Endpoint connectivity to 10.0.0.22:
ICMP to stack: OK, RTT=1.695617ms
HTTP to agent: OK, RTT=1.177812ms
aks-nodepool1-23628489-vmss000001:
Host connectivity to 10.224.0.4:
ICMP to stack: OK, RTT=2.414124ms
HTTP to agent: OK, RTT=1.932519ms
Endpoint connectivity to 10.0.2.75:
ICMP to stack: OK, RTT=2.458024ms
HTTP to agent: OK, RTT=2.08272ms
What are Annotations in Kubernetes?
The Service API, part of Kubernetes, is an abstraction to help you expose groups of Pods over a network. Each Service object defines a logical set of endpoints (usually these endpoints are Pods) along with a policy about how to make those pods accessible.
If your workload speaks HTTP, you might choose to use an Ingress to control how web traffic reaches that workload. Ingress is not a Service type, but it acts as the entry point for your cluster. An Ingress lets you consolidate your routing rules into a single resource so that you can expose multiple components of your workload, running separately in your cluster, behind a single listener.
The Gateway API for Kubernetes provides extra capabilities beyond Ingress and Service. You can add Gateway to your cluster – it is a family of extension APIs, implemented using CustomResourceDefinitions– and then use these to configure access to network services that are running in your cluster.
Services can be annotated to automatically select available load balancers and to provide specific availability sets that host them. Elastic Kubernetes Service (EKS) and Azure Kubernetes Service (AKS) have a respective way of dealing with annotations (listed below) that can be used with Ingress/Gateway API and have been validated for Cilium.
Note:
All these annotations have been extensively tested, and the test results for all cannot be captured, keeping in mind the brevity of the blog.
All three annotations need to be used. This should be an existing subnet.
Load Balancer Service sends traffic directly to the Kubernetes pods behind the service, eliminating the need for an extra network hop through the worker nodes in the Kubernetes cluster.
Load Balancer Service sends traffic to the instances, and the kube-proxy on the individual worker nodes forward it to the pods through one or more worker nodes in the Kubernetes cluster.
How do you set up Cloud Annotations with Gateway API?
Gateway API is a Kubernetes SIG-Network subproject to design a successor for the Ingress object. It is a set of resources that model service networking in Kubernetes and is designed to be role-oriented, portable, expressive, and extensible.
Sample helm config file to enable gateway-api in your EKS or AKS cluster.
Send requests to the Gateway IP (Load Balancer IP) from a test pod.
curl --fail -s http://a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com/details/1 -v
* Host a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com:80 was resolved.
* IPv6: (none)* IPv4: 3.39.64.72, 43.201.158.54
* Trying 3.39.64.72:80...
* Connected to a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com (3.39.64.72) port 80> GET /details/1 HTTP/1.1
> Host: a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
>* Request completely sent off
< HTTP/1.1 200 OK
< content-type: application/json
< server: envoy
< date: Tue, 01 Apr 2025 06:13:24 GMT
< content-length: 178< x-envoy-upstream-service-time: 2<* Connection #0 to host a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com left intact{"id":1,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}
How do you set up Cloud Annotations with Ingress?
Ingress makes your network service ( e.g, HTTP or HTTPS) available using a protocol-aware configuration mechanism that understands web concepts like URIs, hostnames, paths, and more.
Ingress lets you map traffic to different backends based on rules you define via the Kubernetes API. Cilium uses the standard Kubernetes Ingress resource definition, with an ingressClassName of cilium.
This can be used for path-based routing and TLS termination.
The ingress controller creates a service of the LoadBalancer type for which annotations can be used in cloud provider environments (AKS, EKS, GKE, etc).
Cilium allows you to specify the load balancer mode for the Ingress resource:
dedicated: The Ingress controller will create a dedicated load balancer for the Ingress.
shared: The Ingress controller will use a shared load balancer for all Ingress resources.
For Testing IPv6 Ingress/Gateway API in AKS with Cilium, make sure the service is created as
Applying a similar manifest as above with distinct names, two Ingress services are sharing the same Load Balancer IP.
Note- With any annotation being added/changed to a shared ingress, cilium-operator and cilium-agent will have to be restarted.
kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default basic-ingress cilium * 10.228.0.4 80 2d11h
default basic-ingress1 cilium * 10.228.0.4 80 4s
Conclusion
Hopefully, this post gave you an idea of how to use annotations in your AKS and EKS environments with Ingress or Gateway API with Cilium. You can also see these in action in our labs:
Amit Gupta is a senior technical marketing engineer at Isovalent, powering eBPF cloud-native networking and security. Amit has 22+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source. He has previously worked with Motorola, Juniper, Avi Networks (acquired by VMware). He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area outside of work. Amit is an avid runner and cyclist and also spends considerable time helping kids in orphanages.
In this short lab, you will learn about Gateway API, a new Kubernetes standard on how to route traffic into a Kubernetes cluster. The Gateway API is the next generation of the Ingress API.
Gateway API addresses some the Ingress limitations by providing an extensible, role-based and generic model to configure advanced L7 traffic routing capabilities into a Kubernetes cluster.
In this lab, you will learn how you can use the Cilium Gateway API functionality to route HTTP and HTTPS traffic into your Kubernetes-hosted application, including load balancing / traffic splitting and TLS passthrough or termination.
Industry insights you won’t delete. Delivered to your inbox weekly.
