Cloud Annotations for Gateway API and Ingress with Cilium

The debate around Ingress or Gateway API is settled. Gateway API is the better of the two when providing a more flexible and extensible API for more advanced and robust traffic management and control in Kubernetes. Some users are either using Ingress as the standard API object with Cilium or rolling out Gateway API with cloud providers like AWS, Azure, etc. Imagine scrolling through multiple documents and links to figure out which annotation would serve the need for your use-case. This tutorial shows how to use cloud-specific annotations for the Load Balancer service for Ingress & Gateway API on an Elastic Kubernetes Service and Azure Kubernetes Service cluster.

Pre-requisites

The following prerequisites need to be taken into account before you proceed with this tutorial:

• Access to AWS. Create a new account for free.
• The Cilium operator requires the following EC2 privileges to perform ENI creation and IP allocation.
• Ensure you have a cluster IAM role if you’re going to create your cluster with eksctl.
• To ensure that Cilium works properly in an environment that requires firewall rules to enable connectivity, you must add the respective firewall rules.
• You should have an Azure Subscription
• Install Azure CLI
• Install kubectl
• Install Helm
• Install eksctl
• Install awscli
• Cilium CLI (optional): Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s).
• Gateway API related CRD’s should be installed before installing Cilium.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_gateways.yaml

Demo environment

In this blog, we will be testing the respective annotations for Ingress and Gateway API on Azure Kubernetes Service (AKS) and Elastic Kubernetes Service (EKS) based clusters.

AKS

Create an AKS cluster with BYOCNI.

Ensure that the AKS cluster is created with a Service Principal with the correct roles and permissions.

Set the subscription

Choose the subscription you want to use if you have multiple Azure subscriptions.

• Replace SubscriptionName with your subscription name.
• You can also use your subscription ID instead of your subscription name.

az account set --subscription SubscriptionName

AKS Resource Group Creation

Create a Resource Group

clusterName="byocni"
resourceGroup="byocni"
location="eastus"

az group create -l eastus -n byocni

Disable Kube-Proxy on the AKS cluster.

Create a file by the name of kube-proxy.json with the following contents.

{
  "enabled": false,
  "mode": "IPVS",
  "ipvsConfig": {
    "scheduler": "LeastConnection",
    "TCPTimeoutSeconds": 900,
    "TCPFINTimeoutSeconds": 120,
    "UDPTimeoutSeconds": 300
  }
}

AKS Cluster creation

Pass the --network-plugin parameter with the parameter value of none. Also, observe that we will be disabling kube-proxy.

az aks create -l eastus -g byocni -n amit--network-plugin none --kubernetes-version 1.29 --ip-families ipv4,ipv6 --node-vm-size standard_b4ms --vm-set-type VirtualMachineScaleSets --node-count 3 --zones 1 2 3 --kube-proxy-config kube-proxy.json

Set the Kubernetes Context

Log in to the Azure portal, browse Kubernetes Services>, select the respective Kubernetes service created (AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.

[code snippet]

az aks get-credentials --name $clusterName --resource-group $resourceGroup

Cluster status check

Check the status of the nodes and make sure they are in a “Ready” state.

kubectl get nodes -o wide

NAME                                STATUS   ROLES    AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-nodepool1-23628489-vmss000000   Ready    <none>   24h   v1.29.14   10.224.0.6    <none>        Ubuntu 22.04.5 LTS   5.15.0-1082-azure   containerd://1.7.26-1
aks-nodepool1-23628489-vmss000001   Ready    <none>   24h   v1.29.14   10.224.0.4    <none>        Ubuntu 22.04.5 LTS   5.15.0-1082-azure   containerd://1.7.26-1
aks-nodepool1-23628489-vmss000002   Ready    <none>   24h   v1.29.14   10.224.0.5    <none>        Ubuntu 22.04.5 LTS   5.15.0-1082-azure   containerd://1.7.26-1

kubectl describe nodes | grep -E 'InternalIP'

  InternalIP:  10.224.0.6
  InternalIP:  fd87:ef94:e938:40b5::6
  InternalIP:  10.224.0.5
  InternalIP:  fd87:ef94:e938:40b5::5
  InternalIP:  10.224.0.4
  InternalIP:  fd87:ef94:e938:40b5::4

Install Isovalent Enterprise for Cilium

• Users can contact their partner Sales/SE representative(s) at sales@isovalent.com to access the requisite documentation and learn how to install Isovalent Enterprise for Cilium on an AKS cluster with BYOCNI as the network plugin.

Validate Cilium version

Check the version of cilium with cilium version:

kubectl -n kube-system exec ds/cilium -- cilium status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.29 (v1.29.14) [linux/amd64]
Kubernetes APIs:         ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideEnvoyConfig", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumEnvoyConfig", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Secrets", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    True   [eth0   10.224.0.5 fe80::7e1e:52ff:fe7e:1a2b (Direct Routing)]
Host firewall:           Disabled
SRv6:                    Disabled
CNI Chaining:            none
CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                  Ok   1.15.15-cee.1 (v1.15.15-cee.1-78ed04be)
NodeMonitor:             Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok
IPAM:                    IPv4: 7/254 allocated from 10.0.1.0/24,
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Host Routing:            Legacy
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       47/47 healthy
Proxy Status:            OK, ip 10.0.1.161, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 256, max 65535
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 18.63   Metrics: Disabled
Encryption:              Disabled
Cluster health:          3/3 reachable   (2025-03-27T12:26:17Z)
Modules Health:          Stopped(0) Degraded(0) OK(13)

Cilium Health Check

cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status:

kubectl -n kube-system exec ds/cilium -- cilium-health status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
Probe time:   2025-03-27T12:27:17Z
Nodes:
  aks-nodepool1-23628489-vmss000002 (localhost):
    Host connectivity to 10.224.0.5:
      ICMP to stack:   OK, RTT=362.204µs
      HTTP to agent:   OK, RTT=171.102µs
    Endpoint connectivity to 10.0.1.35:
      ICMP to stack:   OK, RTT=351.804µs
      HTTP to agent:   OK, RTT=382.604µs
  aks-nodepool1-23628489-vmss000000:
    Host connectivity to 10.224.0.6:
      ICMP to stack:   OK, RTT=1.619216ms
      HTTP to agent:   OK, RTT=1.955319ms
    Endpoint connectivity to 10.0.0.22:
      ICMP to stack:   OK, RTT=1.695617ms
      HTTP to agent:   OK, RTT=1.177812ms
  aks-nodepool1-23628489-vmss000001:
    Host connectivity to 10.224.0.4:
      ICMP to stack:   OK, RTT=2.414124ms
      HTTP to agent:   OK, RTT=1.932519ms
    Endpoint connectivity to 10.0.2.75:
      ICMP to stack:   OK, RTT=2.458024ms
      HTTP to agent:   OK, RTT=2.08272ms

EKS

Create a ClusterConfig file to create an EKS cluster (e.g., default-ipv4.yaml) in ENI mode.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
     name: cluster1
     region: ap-northeast-2
     version: "1.30"

iam:
  withOIDC: true

managedNodeGroups:
   - name: ng-1
      iam:
        attachPolicyARNs:
          - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
          - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
     desiredCapacity: 2
     privateNetworking: true
     # taint nodes so that application pods are
     # not scheduled/executed until Cilium is deployed.
     # Alternatively, see the note below.
     taints:
       - key: "node.cilium.io/agent-not-ready"
         value: "true"
         effect: "NoExecute"

Create the Cluster

eksctl create cluster -f ./default-ipv4.yaml

Check the status of the nodes and make sure they are in a “Ready” state.

kubectl get nodes -o wide

NAME                                                 STATUS   ROLES    AGE    VERSION               INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION                    CONTAINER-RUNTIME
ip-192-168-150-148.ap-northeast-2.compute.internal   Ready    <none>   2d9h   v1.30.9-eks-5d632ec   192.168.150.148   <none>        Amazon Linux 2023.6.20250303   6.1.129-138.220.amzn2023.x86_64   containerd://1.7.25
ip-192-168-177-162.ap-northeast-2.compute.internal   Ready    <none>   2d9h   v1.30.9-eks-5d632ec   192.168.177.162   <none>        Amazon Linux 2023.6.20250303   6.1.129-138.220.amzn2023.x86_64   containerd://1.7.25

Patch the AWS daemonset

When running in ENI mode, Cilium Enterprise manages ENIs and IP address allocation in place of the AWS VPC CNI plugin. Because of this, and to avoid conflicting behavior, the kube-system/aws-node DaemonSet must be deleted or, at least, prevented from running. To achieve this in a non-destructive way, run the following command:

kubectl -n kube-system patch daemonset aws-node --type='strategic' -p='{"spec":{"template":{"spec":{"nodeSelector":{"io.cilium/aws-node-enabled":"true"}}}}}'

Install Isovalent Enterprise for Cilium

• Users can contact their partner Sales/SE representative(s) at sales@isovalent.com to access the requisite documentation and learn how to install Isovalent Enterprise for Cilium on an EKS cluster.

Validate Cilium version

Check the version of cilium with cilium version:

kubectl -n kube-system exec ds/cilium -- cilium status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.30+ (v1.30.10-eks-bc803b4) [linux/amd64]
Kubernetes APIs:         ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideEnvoyConfig", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumEnvoyConfig", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Secrets", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    True   [ens5   192.168.177.162 fe80::8c6:7aff:fe7d:4081 (Direct Routing), ens6   192.168.188.207 fe80::8a3:25ff:fec3:5c53]
Host firewall:           Disabled
SRv6:                    Disabled
CNI Chaining:            none
CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                  Ok   1.15.15-cee.1 (v1.15.15-cee.1-78ed04be)
NodeMonitor:             Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok
IPAM:                    IPv4: 9/17 allocated,
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Host Routing:            Legacy
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       60/60 healthy
Proxy Status:            OK, ip 192.168.180.133, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 256, max 65535
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 1.82   Metrics: Disabled
Encryption:              Disabled
Cluster health:          2/2 reachable   (2025-03-27T14:00:03Z)
Modules Health:          Stopped(0) Degraded(0) OK(13)

Cilium Health Check

cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status:

kubectl -n kube-system exec ds/cilium -- cilium-health status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), clean-cilium-state (init), install-cni-binaries (init)
Probe time:   2025-03-27T14:01:03Z
Nodes:
  ip-192-168-177-162.ap-northeast-2.compute.internal (localhost):
    Host connectivity to 192.168.177.162:
      ICMP to stack:   OK, RTT=232.614µs
      HTTP to agent:   OK, RTT=207.302µs
    Endpoint connectivity to 192.168.166.233:
      ICMP to stack:   OK, RTT=235.086µs
      HTTP to agent:   OK, RTT=482.347µs
  ip-192-168-150-148.ap-northeast-2.compute.internal:
    Host connectivity to 192.168.150.148:
      ICMP to stack:   OK, RTT=787.418µs
      HTTP to agent:   OK, RTT=856.838µs
    Endpoint connectivity to 192.168.141.45:
      ICMP to stack:   OK, RTT=782.037µs
      HTTP to agent:   OK, RTT=706.165µs

What are annotations in Kubernetes?

The Service API, part of Kubernetes, is an abstraction to help you expose groups of Pods over a network. Each Service object defines a logical set of endpoints (usually these endpoints are Pods) along with a policy about how to make those pods accessible.

• If your workload speaks HTTP, you might choose to use an Ingress to control how web traffic reaches that workload. Ingress is not a Service type, but it acts as the entry point for your cluster. An Ingress lets you consolidate your routing rules into a single resource so that you can expose multiple components of your workload, running separately in your cluster, behind a single listener.
• The Gateway API for Kubernetes provides extra capabilities beyond Ingress and Service. You can add Gateway to your cluster – it is a family of extension APIs, implemented using CustomResourceDefinitions– and then use these to configure access to network services that are running in your cluster.

Services can be annotated to automatically select available load balancers and to provide specific availability sets that host them. Elastic Kubernetes Service (EKS) and Azure Kubernetes Service (AKS) have a respective way of dealing with annotations (listed below) that can be used with Ingress/Gateway API and have been validated for Cilium.

Note: All these annotations have been extensively tested, and the test results for all cannot be captured, keeping in mind the brevity of the blog.

How do you set up Cloud Annotations with Gateway API?

Gateway API is a Kubernetes SIG-Network subproject to design a successor for the Ingress object. It is a set of resources that model service networking in Kubernetes and is designed to be role-oriented, portable, expressive, and extensible.

• Sample helm config file to enable gateway-api in your EKS or AKS cluster.

gatewayAPI:
  enabled: true
k8sServiceHost: #################################################################
k8sServicePort: 443
kubeProxyReplacement: true

• Create a demo app to test the gateway API and name it basic-http.yaml.

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  listeners:
  - protocol: HTTP
    port: 80
    name: web-gw
    allowedRoutes:
      namespaces:
        from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-app-1
spec:
  parentRefs:
  - name: my-gateway
    namespace: default
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /details
    backendRefs:
    - name: details
      port: 9080
  - matches:
    - headers:
      - type: Exact
        name: magic
        value: foo
      queryParams:
      - type: Exact
        name: great
        value: example
      path:
        type: PathPrefix
        value: /
      method: GET
    backendRefs:
    - name: productpage
      port: 9080

• Apply the manifest for gateway API.

kubectl apply -f gateway-api.yaml

• Check the status of the service that is created.

kubectl describe svc cilium-gateway-my-gateway

Name:                     cilium-gateway-my-gateway
Namespace:                default
Labels:                   io.cilium.gateway/owning-gateway=my-gateway
Annotations:              service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector:                 <none>
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.125.253
IPs:                      10.100.125.253
LoadBalancer Ingress:     a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com
Port:                     port-80  80/TCP
TargetPort:               80/TCP
NodePort:                 port-80  32203/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:                   <none>

• Check the status of the gateway that is created

kubectl describe gateway my-gateway

Name:         my-gateway
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2025-03-28T10:40:36Z
  Generation:          1
  Resource Version:    895312
  UID:                 4fdb39f6-21fc-4202-88c1-213ca93ae10b
Spec:
  Gateway Class Name:  cilium
  Infrastructure:
    Annotations:
      service.beta.kubernetes.io/aws-load-balancer-type:  nlb
  Listeners:
    Allowed Routes:
      Namespaces:
        From:  Same
    Name:      web-gw
    Port:      80
    Protocol:  HTTP
Status:
  Addresses:
    Type:   Hostname
    Value:  a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com
  Conditions:
    Last Transition Time:  2025-03-28T10:40:36Z
    Message:               Gateway successfully scheduled
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2025-03-28T10:40:38Z
    Message:               Gateway successfully reconciled
    Observed Generation:   1
    Reason:                Programmed
    Status:                True
    Type:                  Programmed
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2025-03-28T10:40:38Z
      Message:               Listener Programmed
      Observed Generation:   1
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
      Last Transition Time:  2025-03-28T10:40:38Z
      Message:               Listener Accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2025-03-28T10:40:38Z
      Message:               Resolved Refs
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Name:                    web-gw
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
Events:       <none>

• When the Gateway is functional, you can check the routes to verify if they are configured correctly.

kubectl get httproute
NAME         HOSTNAMES   AGE
http-app-1               3d19h

kubectl describe httproute http-app-1
Name:         http-app-1
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1
Kind:         HTTPRoute
Metadata:
  Creation Timestamp:  2025-03-28T10:40:36Z
  Generation:          1
  Resource Version:    895303
  UID:                 a3355035-728f-47f0-a228-bb2239b1f3d5
Spec:
  Parent Refs:
    Group:      gateway.networking.k8s.io
    Kind:       Gateway
    Name:       my-gateway
    Namespace:  default
  Rules:
    Backend Refs:
      Group:
      Kind:    Service
      Name:    details
      Port:    9080
      Weight:  1
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /details
    Backend Refs:
      Group:
      Kind:    Service
      Name:    productpage
      Port:    9080
      Weight:  1
    Matches:
      Headers:
        Name:   magic
        Type:   Exact
        Value:  foo
      Method:   GET
      Path:
        Type:   PathPrefix
        Value:  /
      Query Params:
        Name:   great
        Type:   Exact
        Value:  example
Status:
  Parents:
    Conditions:
      Last Transition Time:  2025-03-28T10:40:36Z
      Message:               Accepted HTTPRoute
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2025-03-28T10:40:36Z
      Message:               Service reference is valid
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         io.cilium/gateway-controller
    Parent Ref:
      Group:      gateway.networking.k8s.io
      Kind:       Gateway
      Name:       my-gateway
      Namespace:  default
Events:           <none>

• Send requests to the Gateway IP (Load Balancer IP) from a test pod.

curl --fail -s http://a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com/details/1 -v

* Host a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com:80 was resolved.
* IPv6: (none)
* IPv4: 3.39.64.72, 43.201.158.54
*   Trying 3.39.64.72:80...
* Connected to a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com (3.39.64.72) port 80
> GET /details/1 HTTP/1.1
> Host: a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< content-type: application/json
< server: envoy
< date: Tue, 01 Apr 2025 06:13:24 GMT
< content-length: 178
< x-envoy-upstream-service-time: 2
<
* Connection #0 to host a0075645d674441a2a288d59c2ee3917-03f4d1bbe1eb6c76.elb.ap-northeast-2.amazonaws.com left intact
{"id":1,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}

How do you set up Cloud Annotations with Ingress?

Ingress makes your network service ( e.g, HTTP or HTTPS) available using a protocol-aware configuration mechanism that understands web concepts like URIs, hostnames, paths, and more.

• Ingress lets you map traffic to different backends based on rules you define via the Kubernetes API. Cilium uses the standard Kubernetes Ingress resource definition, with an ingressClassName of cilium.
• This can be used for path-based routing and TLS termination.
• The ingress controller creates a service of the LoadBalancer type for which annotations can be used in cloud provider environments (AKS, EKS, GKE, etc).
• Cilium allows you to specify the load balancer mode for the Ingress resource:
  • dedicated: The Ingress controller will create a dedicated load balancer for the Ingress.
  • shared: The Ingress controller will use a shared load balancer for all Ingress resources.
• For Testing IPv6 Ingress/Gateway API in AKS with Cilium, make sure the service is created as

ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: PreferDualStack

• IPv6 Ingress/Gateway API in EKS with ENI mode is a roadmap item.
• Sample helm config file to enable dedicated ingress in your EKS or AKS cluster.

ingressController:
  enabled: true
  enforceHttps: false
  loadbalancerMode: dedicated
  service:
    annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    type: LoadBalancer
k8sServiceHost: ###################################################################
k8sServicePort: 443
kubeProxyReplacement: true

• Create a demo app to test the ingress and name it ingress.yaml.

# Basic ingress for istio bookinfo demo application, which can be found in below
# https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: basic-ingress
  namespace: default
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    #service.beta.kubernetes.io/azure-dns-label-name: "tmetesting"
    #service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "app-testing"
    #service.beta.kubernetes.io/azure-load-balancer-ipv4: "10.228.0.4"
    #service.beta.kubernetes.io/azure-load-balancer-ipv6: "fd87:ef94:e938:40b4::4"
    #service.beta.kubernetes.io/azure-load-balancer-resource-group: "app-testing"
spec:
  ingressClassName: cilium
  rules:
  - http:
      paths:
      - backend:
          service:
            name: details
            port:
              number: 9080
        path: /details
        pathType: Prefix
      - backend:
          service:
            name: productpage
            port:
              number: 9080
        path: /
        pathType: Prefix

• Apply the manifest for ingress.

kubectl apply -f http-ingress.yaml

• Check the status of the service that is created.

kubectl describe svc cilium-ingress-basic-ingress

Name:                     cilium-ingress-basic-ingress
Namespace:                default
Labels:                   cilium.io/ingress=true
Annotations:              service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector:                 <none>
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.203.145
IPs:                      10.0.203.145
LoadBalancer Ingress:     10.224.0.7
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31037/TCP
Endpoints:
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  31602/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason               Age   From                Message
  ----    ------               ----  ----                -------
  Normal  EnsuredLoadBalancer  59m   service-controller  Ensured load balancer

• Check the status of the ingress that is created.

kubectl get ingress

NAME            CLASS    HOSTS   ADDRESS      PORTS   AGE
basic-ingress   cilium   *       10.224.0.7   80      62m

• Send requests to the Ingress IP (Load Balancer IP) from a test pod.

curl --fail -s http://10.224.0.7/details/1 -v

*   Trying 10.224.0.7:80...
* Connected to 10.224.0.7 (10.224.0.7) port 80 (#0)
> GET /details/1 HTTP/1.1
> Host: 10.224.0.7
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< server: envoy
< date: Wed, 04 Sep 2024 10:58:11 GMT
< content-length: 178
< x-envoy-upstream-service-time: 5
<
* Connection #0 to host 10.224.0.7 left intact
{"id":1,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"

• Sample helm config file to enable shared ingress in your EKS or AKS cluster.

ingressController:
  enabled: true
  enforceHttps: false
  loadbalancerMode: shared
  service:
    annotations:
        service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    type: LoadBalancer
k8sServiceHost: ###################################################################
k8sServicePort: 443
kubeProxyReplacement: true

• Applying a similar manifest as above with distinct names, two Ingress services are sharing the same Load Balancer IP.

Note- With any annotation being added/changed to a shared ingress, cilium-operator and cilium-agent will have to be restarted.

kubectl get ingress -A

NAMESPACE   NAME             CLASS    HOSTS   ADDRESS      PORTS   AGE
default     basic-ingress    cilium   *       10.228.0.4   80      2d11h
default     basic-ingress1   cilium   *       10.228.0.4   80      4s

Conclusion

Hopefully, this post gave you an idea of how to use annotations in your AKS and EKS environments with Ingress or Gateway API with Cilium. You can also see these in action in our labs:

• Cilium Gateway API
• Advanced Gateway API Use Cases

References

• Azure Load Balancer Annotations
• AWS Load Balancer Annotations

/authors/amit-gupta

Amit Gupta, Senior Technical Marketing Engineer