Tutorial: Cross-Namespace Routing with Cilium Gateway API

The Gateway API is quickly gaining in popularity and features. There were multiple sessions at KubeCon Europe 2023 and the only one I had the chance to attend was massively oversubscribed. As we stated in a previous blog, this is clearly the future of Kubernetes ingress – if not already the present.

But with Kubernetes engineers beginning to operate Gateway API at scale, they are likely to see a sprawl of Gateways being created. It is likely to cost them not only the operational pain in managing them but also a financial tax (especially in cloud environments as a Gateway API per app or per namespace would require a public IP and a cloud load balancer, which are not free resources).

In this tutorial, we will show how users can centralize their Gateway API management and leverage an elegant Gateway API feature that enables namespaces to access a shared Gateway.

If you are new to the Gateway API on Cilium, start with this introductory post on the Gateway API (the “Why” and “What”) and this tutorial (the “How”) on how to deploy and configure it.

You can also head out to our lab pages and try some either of the Gateway API labs. Start with the intro lab before moving on to the advanced use cases lab, where you will be able to test the cross-namespace routing capability featured in this post.

Alternatively, just watch me introduce and demo the feature:

Cross-Namespace Routing

The Gateway API has core support for cross Namespace routing. This is useful when more than one user or team is sharing the underlying networking infrastructure, yet control and configuration must be segmented to minimize access and fault domains.

Gateways and Routes can be deployed into different Namespaces and Routes can attach to Gateways across Namespace boundaries. Gateway and Route attachment is bidirectional – attachment can only succeed if the Gateway owner and Route owner both agree to the relationship.

This effectively creates a handshake between the infra owners and application owners that enables them to independently define how applications are exposed through Gateways.

It results in creating a policy that reduces administrative overhead. App owners can specify which Gateways their apps should use and infra owners can constrain the Namespaces and types of Routes that a Gateway accepts. 

Let’s explore a practical example.

Cross-Namespace Routing at ACME

In this example, we will consider a fictional ACME company and three different business units within ACME. Each of them has its own environment, application and namespace.

• The recruiting team has a public-facing careers app where applicants can submit their CV.
• The product team has a public-facing product app where prospective customers can find out more the ACME product.
• The HR team has an internal-facing hr app storing private employee details.

Each app is deployed in its own Namespace. Because careers and product are both public-facing apps, the Security team approved the use of a shared Gateway API. A benefit of a shared Gateway API is that platform and security teams could control centrally the Gateway API, including its certificate management. As mentioned before, this would also reduce the cost of running Gateway API in the public cloud.

However, the Security team does not want the HR details to be exposed and accessible from outside the cluster and therefore does not approve a HTTPRoute attachment from the hr namespace to the Gateway.

Walkthrough

To walk through how this feature works, I deployed four namespaces, alongside deployments and services fronting these deployments. You can find the manifest here.

root@server:~# kubectl get ns --show-labels | grep -e hr -e infra -e product -e careers 
careers              Active   16m   kubernetes.io/metadata.name=careers,shared-gateway-access=true
hr                   Active   16m   kubernetes.io/metadata.name=hr
infra-ns             Active   16m   kubernetes.io/metadata.name=infra-ns
product              Active   16m   kubernetes.io/metadata.name=product,shared-gateway-access=true
root@server:~# kubectl get svc -A | grep -e echo-hr -e echo-product -e echo-careers 
careers       echo-careers               ClusterIP      10.96.215.227   <none>           9080/TCP                 23m
hr            echo-hr                    ClusterIP      10.96.39.228    <none>           9080/TCP                 23m
product       echo-product               ClusterIP      10.96.242.106   <none>           9080/TCP                 23m
root@server:~# 

Notice that product and careers both have the shared-gateway-access=true label, but hr does not (this will be important later).

Let’s review the Gateway and the HTTPRoutes manifest before I deploy it with kubectl apply:

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: shared-gateway
  namespace: infra-ns
spec:
  gatewayClassName: cilium
  listeners:
  - name: shared-http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            shared-gateway-access: "true"
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: cross-namespace
  namespace: hr
spec:
  parentRefs:
  - name: shared-gateway
    namespace: infra-ns
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /hr
    backendRefs:
    - kind: Service
      name: echo-hr
      port: 9080
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: cross-namespace
  namespace: product
spec:
  parentRefs:
  - name: shared-gateway
    namespace: infra-ns
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /product
    backendRefs:
    - kind: Service
      name: echo-product
      port: 9080
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: cross-namespace
  namespace: careers
spec:
  parentRefs:
  - name: shared-gateway
    namespace: infra-ns
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /careers
    backendRefs:
    - kind: Service
      name: echo-careers
      port: 9080

By now, you should be familiar with the vast majority of the manifest. The HTTPRoutes just route the traffic depending on the path (HTTP traffic to /careers will be proxied to the careers Service over port 9080 and likewise for product and hr).

Let’s look at some of the fields you may not have seen (assuming you read the previous Gateway API tutorial). First, in the Gateway definition, notice it has been deployed in the infra-ns namespace:

metadata:
  name: shared-gateway
  namespace: infra-ns

This section might also look unfamiliar:

    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            shared-gateway-access: "true"

This Gateway uses a Namespace selector to define which HTTPRoutes are allowed to attach. This allows the infrastructure team to constrain who or which apps can use this Gateway by allowlisting a set of Namespaces.

Only Namespaces which are labelled shared-gateway-access: "true" will be able to attach their Routes to the shared Gateway.

In the HTTPRoute definitions, notice how we refer to the shared-gateway in the parentRefs. We specify the Gateway we want to attach to and the Namespace it is in.

Let’s now test the cross-namespace routing. First, let’s fetch the Gateway IP.

root@server:~# GATEWAY=$(kubectl get gateway shared-gateway -n infra-ns -o jsonpath='{.status.addresses[0].value}')
echo $GATEWAY
172.18.255.201

Now, let’s connect to the product and careers Services:

root@server:~# curl -s -o /dev/null -w "%{http_code}" http://$GATEWAY/product
200
root@server:~# curl -s -o /dev/null -w "%{http_code}" http://$GATEWAY/careers
200

Both commands are successful and return a 200 status code.

Let’s try to connect to the hr Service:

root@server:~# curl -s -o /dev/null -w "%{http_code}" http://$GATEWAY/hr
404

It returns a 404. Why?

The HTTPRoute in the hr Namespace with a parentRef for infra-ns/shared-gateway would be ignored by the Gateway because the attachment constraint (Namespace label) was not met.

Let’s verify with the following commands by checking the status of the HTTPRoutes:

root@server:~# echo "Product HTTPRoute Status"
kubectl get httproutes.gateway.networking.k8s.io -n product -o jsonpath='{.items[0].status.parents[0].conditions[0]}' | jq
echo "Careers HTTPRoute Status"
kubectl get httproutes.gateway.networking.k8s.io -n careers -o jsonpath='{.items[0].status.parents[0].conditions[0]}' | jq
echo "HR HTTPRoute Status"
kubectl get httproutes.gateway.networking.k8s.io -n hr -o jsonpath='{.items[0].status.parents[0].conditions[0]}' | jq

Product HTTPRoute Status
{
  "lastTransitionTime": "2023-04-24T10:48:59Z",
  "message": "Accepted HTTPRoute",
  "observedGeneration": 1,
  "reason": "Accepted",
  "status": "True",
  "type": "Accepted",
}
Careers HTTPRoute Status
{
  "lastTransitionTime": "2023-04-24T10:48:59Z",
  "message": "Accepted HTTPRoute",
  "observedGeneration": 1,
  "reason": "Accepted",
  "status": "True",
  "type": "Accepted",
}
HR HTTPRoute Status
{
  "lastTransitionTime": "2023-04-24T10:48:59Z",
  "message": "HTTPRoute is not allowed",
  "observedGeneration": 1,
  "reason": "InvalidHTTPRoute",
  "status": "False",
  "type": "Accepted",
}

As expected, the first two are Accepted HTTPRoute while the last one has been rejected (its status is False and the message is HTTPRoute is not allowed).

This feature provides engineers with multiple options: either have a dedicated Gateway API per Namespace or per app if required or alternatively use shared Gateway API for centralized management and to reduce potential costs.

If you’d like to try this feature, head over to the lab page.

Thanks for reading.