HyperText Transfer Protocol (HTTP) might have been created in 1989, but it has certainly withstood the test of time. It laid the foundation for the World Wide Web, and it remains ever-popular and the protocol of choice for the vast majority of APIs (despite the more recent alternatives like GraphQL and gRPC). For the past few decades, we've used Layer 7-aware load balancers to control, alter, and route HTTP traffic as it entered our network. In this blog post, we are going to explore how you can alter HTTP traffic (redirect, rewrite, and mirror) as it enters Kubernetes clusters with Cilium's Gateway API, focusing on three specific use cases:
- HTTP Redirect – to tell the client that the resource they are trying to access has moved
- HTTP Path Rewrite – to rewrite the URL or entire path used by the client
- HTTP Mirroring – to copy the traffic sent from the client to another backend
In the world of Kubernetes, the Gateway API’s role is to modify and control traffic as it enters our clusters.
In the previous “Getting Started with the Cilium Gateway API tutorial,” we explored how you can use Cilium’s Gateway API support to load-balance HTTP traffic and alter HTTP header requests and responses.
We’ve also recorded several short tutorials on the Gateway API, including the one below on “TLS Passthrough,” where traffic can be encrypted to the server.
2qfDWu-v8RE - "Tutorial: Redirect, Rewrite, and Mirror HTTP with Cilium Gateway API" Video
In this tutorial, we will explore these use cases. If you'd rather watch me present it, check out this episode of the eBPF and Cilium weekly show eCHO:
dfdc-aoMiy8 - "Tutorial: Redirect, Rewrite, and Mirror HTTP with Cilium Gateway API" Video
If you'd rather do it yourself, follow the steps below. Let's start!
Demo environment
Before we start with the first use case—redirecting HTTP traffic—let’s prepare our environment. This tutorial will use an Azure Kubernetes Service (AKS) cluster in BYOCNI mode and an Elastic Kubernetes Service (EKS) cluster in ENI mode. In this mode, a cluster is deployed without a Container Network Interface (CNI) so that we can bring our own Cilium in this instance.
AKS-BYOCNI
Create an AKS cluster with BYOCNI and fetch the credentials once the cluster is up and running.
export NAME="$(whoami)-$RANDOM" export AZURE_RESOURCE_GROUP="${NAME}-group" az group create --name "${AZURE_RESOURCE_GROUP}" -l westus2 az aks create \ --resource-group "${AZURE_RESOURCE_GROUP}" \ --name "${NAME}" \ --network-plugin none az aks get-credentials --resource-group "${AZURE_RESOURCE_GROUP}" --name "${NAME}"Truncated O/P:
==========={ "id": "/subscriptions/subscription-id/resourceGroups/nicovibert-4665-group", "location": "westus2", "name": "nicovibert-4665-group", "properties": { "provisioningState": "Succeeded" }, "type": "Microsoft.Resources/resourceGroups" }Note that your nodes won't show as Ready for now - there's no CNI, so connectivity is limited.
kubectl get nodes NAME STATUS ROLES AGE VERSION aks-nodepool1-35842206-vmss000000 NotReady agent 8m21s v1.26.6 aks-nodepool1-35842206-vmss000001 NotReady agent 8m8s v1.26.6 aks-nodepool1-35842206-vmss000002 NotReady agent 8m18s v1.26.6When using kubectl, you might even see error logs, such as:
Couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the requestThis is just because of the lack of connectivity before installing Ciliium.
EKS-ENI
Create a ClusterConfig file to create an EKS cluster (e.g., default-ipv4.yaml)
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: cluster2 region: ap-northeast-1 version: "1.29" managedNodeGroups: - name: ng-1 desiredCapacity: 2 privateNetworking: true # taint nodes so that application pods are # not scheduled/executed until Cilium is deployed. # Alternatively, see the note below. taints: - key: "node.cilium.io/agent-not-ready" value: "true" effect: "NoExecute"Create the cluster
eksctl create cluster -f ./default-ipv4.yamlCheck the status of the nodes and make sure they are in a “Ready” state.
kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-192-168-132-209.ap-south-1.compute.internal Ready <none> 4d20h v1.29.3-eks-ae9a62a 192.168.132.209 <none> Amazon Linux 2 5.10.219-208.866.amzn2.x86_64 containerd://1.7.11 ip-192-168-185-135.ap-south-1.compute.internal Ready <none> 4d20h v1.29.3-eks-ae9a62a 192.168.185.135 <none> Amazon Linux 2 5.10.219-208.866.amzn2.x86_64 containerd://1.7.11When running in ENI mode, Cilium Enterprise manages ENIs and IP address allocation in place of the AWS VPC CNI plugin. Because of this, and to avoid conflicting behavior, the kube-system/aws-node DaemonSet must be deleted or, at least, prevented from running. To achieve this in a non-destructive way, run the following command:
kubectl -n kube-system patch daemonset aws-node --type='strategic' -p='{"spec":{"template":{"spec":{"nodeSelector":{"io.cilium/aws-node-enabled":"true"}}}}}'
Let's use the latest Gateway API Custom Resource Definitions (CRDs). Remember to install them before installing Cilium:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.0.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yamlSupport for the features we are exploring today - and for the 1.0 Gateway API version - was recently added to the main branch of Cilium - to use them, you will need Cilium 1.15.
Use this Cilium CLI command (install the CLI with the instructions here if you've not done so already) for both the EKS and AKS clusters:
cilium install --version 1.15.0-pre.0 --namespace kube-system --set kubeProxyReplacement=true --set gatewayAPI.enabled=true --set azure.resourceGroup="${AZURE_RESOURCE_GROUP}"Expect this output:
Auto-detected Kubernetes kind: AKS
ℹ️ Using Cilium version 1.15.0-pre.0
Auto-detected cluster name: nicovibert-4665
✅ Derived Azure subscription ID subscription-id from subscription cilium-dev
✅ Detected Azure AKS cluster in BYOCNI mode (no CNI plugin pre-installed)
Auto-detected kube-proxy has been installedRemember that the kubeProxyReplacement feature (KPR) is required for the Cilium Gateway API. Note that, even when creating the cluster in BYOCNI mode, kubeProxy was deployed in the cluster.
You can use this new option instead of skipping its deployment, as it is no longer needed once Cilium is deployed in KPR mode.
After installation, check the Cilium status with:
cilium status --wait
Expect the status to be:
/¯¯
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode)
\__/¯¯\__/ Hubble Relay: disabled
\__/ ClusterMesh: disabled
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
Containers: cilium Running: 3
cilium-operator Running: 1
Cluster Pods: 5/5 managed by Cilium
Helm chart version: 1.15.0-pre.0
Image versions cilium quay.io/cilium/cilium:v1.15.0-pre.0: 3
cilium-operator quay.io/cilium/operator-generic:v1.15.0-pre.0: 1The status for both Cilium and Operator should be OK. Now that our environment is ready, let's go ahead and deploy our demo app and our Gateway before deploying the HTTPRoutes to illustrate the use cases we will cover in this blog post.
We will use this YAML manifest I have put on a GitHub repo.
This configuration is a subset of configs used in the Gateway API conformance tests. Remember that one of the benefits of the Gateway API project is consistency. To ensure that each Gateway API provides a consistent user experience, each implementation is tested against a set of conformance tests that creates a series of Gateways and Routes and tests that the implementation matches the API specification.
Deploy this demo app:
kubectl apply -f https://raw.githubusercontent.com/nvibert/gateway-api-samples/main/Rewrite%2C%20Redirect%2C%20Mirror/demo-app.yamlExpect this output:
service/infra-backend-v1 created
deployment.apps/infra-backend-v1 created
service/infra-backend-v2 created
deployment.apps/infra-backend-v2 createdWe are re-using the same Gateway configuration we used in the previous blog post. It's a simple Gateway configuration (it's simply listening for HTTP traffic):
cat << 'EOF' > gateway.yaml
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: cilium-gw
spec:
gatewayClassName: cilium
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Same
EOF
kubectl apply -f gateway.yamlExpect this output:
gateway.gateway.networking.k8s.io/cilium-gw createdLet's double-check that the Gateway has been installed and received an IP address. Remember that the IP address allocation is done automatically for you in public clouds. In contrast, you would need to use MetalLB or Cilium's own LoadBalancer IP Address Management feature on a private cloud to assign this IP.
kubectl get gateway Expect an outcome such as:
NAME CLASS ADDRESS PROGRAMMED AGE
cilium-gw cilium 20.115.194.177 True 85sLet's save this IP as the $GATEWAY variable.
GATEWAY=$(kubectl get gateway cilium-gw -o jsonpath='{.status.addresses[0].value}')
echo $GATEWAYHTTP Redirect
One common use case for ingress gateways is sending HTTP redirects to clients to tell them that the resource they are trying to access in the cluster has moved to a different location.This is a common requirement for migration, content optimization, SSL/TLS enforcement - the list goes on.
Redirects return HTTP 3XX responses to a client, instructing it to retrieve a different resource. With the Gateway API, we can use redirect filters to substitute various URL components independently, as shown below.
Let's use this HTTPRoute YAML for the four examples in this section. We will review each rule in detail below.
kubectl apply -f https://raw.githubusercontent.com/nvibert/gateway-api-samples/main/Rewrite%2C%20Redirect%2C%20Mirror/http-redirect-route.yamlPath Redirect
In this first example, we will do a simple redirect: we only replace a portion of the URL and redirect them there:
You should see the IP address allocated to the Gateway (such as 20.115.194.177).
The following rule will match traffic to /original-prefix and redirect the client to a different URL:
- matches:
- path:
type: PathPrefix
value: /original-prefix
filters:
- type: RequestRedirect
requestRedirect:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /replacement-prefixLet's try using curl.
curl -l -v http://$GATEWAY/original-prefixNotice we use -l in the curl request to follow the redirects (by default, curl will not follow redirects), and we use the verbose option of curl to see the response headers.
* Trying 20.115.194.177:80...
* Connected to 20.115.194.177 (20.115.194.177) port 80 (#0)
> GET /original-prefix HTTP/1.1
> Host: 20.115.194.177
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 302 Found
< location: http://20.115.194.177:80/replacement-prefix
< date: Tue, 12 Sep 2023 08:51:52 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host 20.115.194.177 left intactThe location is used in Redirect messages to tell the client where to go. As you can see, the client is redirected to http://20.115.194.177:80/replacement-prefix. The prefix was replaced with /original-prefix to /replacement-prefix. Note that we support different models that replace the URL. We can replace just a portion of the path or the entire one.
Host & Path redirects
You can also redirect the client to a different host.
Let's try, with this specification, to redirect the client to example.org:
- matches:
- path:
type: PathPrefix
value: /path-and-host
filters:
- type: RequestRedirect
requestRedirect:
hostname: example.org
path:
type: ReplacePrefixMatch
replacePrefixMatch: /replacement-prefixLet's make HTTP requests to that external address and path:
curl -l -v http://$GATEWAY/path-and-hostExpect an output such as:
* Trying 20.115.194.177:80...
* Connected to 20.115.194.177 (20.115.194.177) port 80 (#0)
> GET /path-and-host HTTP/1.1
> Host: 20.115.194.177
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 302 Found
< location: http://example.org:80/replacement-prefix
< date: Tue, 12 Sep 2023 08:52:26 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host 20.115.194.177 left intactAs you can see, the client is redirected to http://example.org:80/replacement-prefix.
Both the hostname and the path prefix were modified.
Redirect to new prefix and custom status code
Next, you can also modify the status code. By default, as you saw above, the redirect status code is 302. It means that the resources have been moved temporarily.
To indicate that the resources the client is trying to access have moved permanently, you can use the status code 301. You can also combine it with the prefix replacement.
Let's use this example:
- matches:
- path:
type: PathPrefix
value: /path-and-status
filters:
- type: RequestRedirect
requestRedirect:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /replacement-prefix
statusCode: 301Try to access this URL:
curl -l -v http://$GATEWAY/path-and-statusExpect an output such as:
* Trying 20.115.194.177:80...
* Connected to 20.115.194.177 (20.115.194.177) port 80 (#0)
> GET /path-and-status HTTP/1.1
> Host: 20.115.194.177
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< location: http://20.115.194.177:80/replacement-prefix
< date: Tue, 12 Sep 2023 08:52:33 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host 20.115.194.177 left intactNote the status code returned is 301 Moved Permanently and the client is redirected to http://20.115.194.177:80/replacement-prefix.
Redirect to new prefix and custom status code
Finally, we can use the Gateway API to impose tighter security controls. You can redirect the client to use HTTPS instead of HTTP, changing the scheme used by the client.
Look at the last line in this specification:
- matches:
- path:
type: PathPrefix
value: /scheme-and-host
filters:
- type: RequestRedirect
requestRedirect:
hostname: example.org
scheme: "https"Let's try it.
curl -l -v http://$GATEWAY/scheme-and-hostExpect an output such as:
* Trying 20.115.194.177:80...
* Connected to 20.115.194.177 (20.115.194.177) port 80 (#0)
> GET /scheme-and-host HTTP/1.1
> Host: 20.115.194.177
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 302 Found
< location: https://example.org:443/scheme-and-host
< date: Tue, 12 Sep 2023 09:09:20 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host 20.115.194.177 left intactAs you can see, the client initially tried to connect via HTTP and is redirected to https://example.org:443/scheme-and-host:
HTTP Rewrite
Sometimes, we don't need redirecting the client to a different URL. Instead, we can rewrite components of a client request.
Let's explore this with a couple of examples. Create the HTTPRoute by applying the manifest below:
kubectl apply -f https://raw.githubusercontent.com/nvibert/gateway-api-samples/main/Rewrite%2C%20Redirect%2C%20Mirror/http-rewrite-route.yamlThe Gateway will replace the /prefix/one in the URL request to /one.
Let's now check that traffic based on the URL path is proxied and altered by the Gateway API:
curl http://$GATEWAY/prefix/oneThe request is received by an echo server that copies the original request and sends the reply back in the packet's body.
JSON:
{
"path": "/one",
"host": "20.115.194.177",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.1.2"
],
"X-Envoy-External-Address": [
"A.B.C.D"
],
"X-Envoy-Original-Path": [
"/prefix/one"
],
"X-Forwarded-For": [
"A.B.C.D"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"7b09721a-2162-45e0-a0a8-c55829f1603a"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "infra-backend-v1-57dc9df649-lwxc6"
} As you can see, the Gateway changed the original request from /prefix/one to /one.
As we use the Envoy proxy for L7 traffic processing, note that Envoy also adds information about the original path in the packet (see "X-Envoy-Original-Path").
We can also combine this with previous Gateway API features we explored in the previous tutorial. You might want to rewrite traffic and add some headers to it to add some metadata (so that the receiving server can interpret it accordingly).
If you look at the HTTPRoute we've just created, you will see that traffic to /rewrite-path-and-modify-headers will not only be partially rewritten, but some headers can be added, removed, or modified.
curl http://$GATEWAY/prefix/rewrite-path-and-modify-headersExpect an output such as:
JSON:
{
"path": "/prefix",
"host": "20.115.194.177",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.1.2"
],
"X-Envoy-External-Address": [
"A.B.C.D"
],
"X-Envoy-Original-Path": [
"/prefix/rewrite-path-and-modify-headers"
],
"X-Forwarded-For": [
"A.B.C.D"
],
"X-Forwarded-Proto": [
"http"
],
"X-Header-Add": [
"header-val-1"
],
"X-Header-Add-Append": [
"header-val-2"
],
"X-Header-Set": [
"set-overwrites-values"
],
"X-Request-Id": [
"de0055e7-5d91-49fd-98cc-181ed132a08d"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "infra-backend-v1-57dc9df649-lwxc6"
}HTTP Mirroring
In this final example, we will review how to mirror traffic. It has plenty of use cases - monitoring incoming traffic for forensics, analysis, logging, troubleshooting, etc.
With the following rule, we can mirror traffic meant for infra-backend-v1 to infra-backend-v2.
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: request-mirror
spec:
parentRefs:
- name: cilium-gw
rules:
- matches:
- path:
type: PathPrefix
value: /mirror
filters:
- type: RequestMirror
requestMirror:
backendRef:
name: infra-backend-v2
port: 8080
backendRefs:
- name: infra-backend-v1
port: 8080Apply it:
kubectl apply -f https://raw.githubusercontent.com/nvibert/gateway-api-samples/main/Rewrite%2C%20Redirect%2C%20Mirror/http-mirror-route.yamlThe Gateway will forward the traffic infra-backend-v1 as standard but copy it across to infra-backend-v2 (purple arrow in the image below). The response infra-backend-v1 will be processed normally by the Gateway, but the responses infra-backend-v2 will be ignored (red arrow).
Let's request the /mirror path.
curl http://$GATEWAY/mirrorLook at the output below. Traffic was sent to the infra-backend-v1 Service. Was it also mirrored to infra-backend-v2 ? How can we prove it?
JSON:
{
"path": "/mirror",
"host": "20.115.194.177",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.1.2"
],
"X-Envoy-External-Address": [
"A.B.C.D"
],
"X-Forwarded-For": [
"A.B.C.D"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"3caacef1-bbcb-4e01-a088-d3d411bd6dc1"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "infra-backend-v1-57dc9df649-xgvkr"
}The image used by the echo Pods serving this Service is minimal. Instead of trying to install tcpdump Let's use the Kubernetes debug functionality instead, with an image (nicolaka/netshoot) designed to troubleshoot networking issues.
The command below will deploy a container in the same Pod as the infra-backend-v2 Pod and will see the traffic coming in:
BACKEND_POD_NAME=$(kubectl get pods --no-headers=true -o custom-columns=":metadata.name" -l app=infra-backend-v2 | head -n 1)
kubectl debug $BACKEND_POD_NAME -it --image=nicolaka/netshoot -- tcpdump -i eth0Once you run the curl command again from a different terminal; you should see traffic appearing (you may have to run it on a few occasions as traffic will be load-balanced between multiple infra-backend-v2 Pods):
11:30:15.778113 IP 10.244.2.149.59950 > infra-backend-v2-664bffc4-8hbll.3000: Flags [F.], seq 235, ack 627, win 503, options [nop,nop,TS val 4109474445 ecr 389226307], length 0
11:30:15.778233 IP infra-backend-v2-664bffc4-8hbll.3000 > 10.244.2.149.59950: Flags [F.], seq 627, ack 236, win 501, options [nop,nop,TS val 389286309 ecr 4109474445], length 0This shows that the Gateway mirrored traffic to the infra-backend-v2 Service.
Post Demo Clean-Up
After testing, you can clean up your demo environment and remove both the cluster and the resource group with the following commands:
Delete the AKS Cluster:
az aks delete --resource-group "${AZURE_RESOURCE_GROUP}" --name "${NAME}"
az group delete --name "${AZURE_RESOURCE_GROUP}"Delete the EKS Cluster:
eksctl delete cluster cluster2 --region ap-northeast-1Recap
Cilium might be famously known for being a high-performance CNI and its network policy engine, but it is equally capable of performing L7 load-balancing. Unlike my days as a network engineer, when I would have to drive to a data center, lug a load balancer onto the rack, install it, and configure it, with Cilium, I don't have to install anything else—it's just another feature to enable and a YAML manifest to apply.
In this blog post, you will have learned—and hopefully tried alongside me—how Cilium Gateway API can send HTTP redirects, rewrite URL paths, and mirror HTTP traffic. This is another powerful addition to all the other Layer 7 Load Balancing features that Cilium Gateway API already supports.
Do you think this might be useful for your applications? Got any use case in mind? Let me know on Cilium Slack, or you can find me on LinkedIn.
Learn More
- YouTube Playlist of Cilium Gateway API videos
- Cilium Gateway API Lab
- Advanced Gateway API Use Cases Lab
Nico Vibert, Principal Technical Marketing Engineer
Nico Vibert is a Principal Technical Marketing Engineer at Isovalent, the company behind the open-source cloud-native solution Cilium. Prior to joining Isovalent, Nico worked in many different roles - operations and support, design and architecture, and technical pre-sales - at companies such as HashiCorp, VMware, and Cisco. In his current role, Nico focuses primarily on creating content to make networking a more approachable field and regularly speaks at events like KubeCon and Cisco Live. Nico has held over 15 networking certifications, including the Cisco Certified Internetwork Expert CCIE (# 22990). Nico is the co-author of the "Cilium: Up and Running" O'Reilly book and is the Lead Subject Matter Expert on the Cilium Certified Associate (CCA) certification.
Amit Gupta, Senior Technical Marketing Engineer
Amit Gupta is a senior technical marketing engineer at Isovalent, powering eBPF cloud-native networking and security. Amit has 21+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source. He has previously worked with Motorola, Juniper, Avi Networks (acquired by VMware), and Prosimo. He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area outside of work. Amit is an avid runner and cyclist and also spends considerable time helping kids in orphanages.