Mixed Routing Mode with Isovalent Cluster Mesh

Users need to reach applications hosted in the cloud or private data centers. Cluster Mesh solves this problem by seamlessly interconnecting clusters. However, one thing remains: Kubernetes clusters often run with different routing modes and encapsulations. This blog will guide you in enabling Isovalent Cluster Mesh across two Kubernetes clusters (GKE Data Plane v1) with different routing modes via Mixed Routing Mode.

What are the various Routing Modes in Cilium?

Cilium can be configured to run in either the native (specific to the cloud provider) routing mode or using a specific encapsulation (GENEVE/VXLAN) when native routing is not available.

Native Routing- The native routing mode leverages the routing capabilities of the network Cilium runs on instead of performing encapsulation. In native routing mode, Cilium will delegate all packets which are not addressed to another local endpoint to the routing subsystem of the Linux kernel. As a result, the network connecting the cluster nodes must be capable of routing PodCIDRs. Cilium automatically enables IP forwarding in the Linux kernel when native routing is configured.

Encapsulation- In this mode, all cluster nodes form a mesh of tunnels using the UDP-based encapsulation protocols VXLAN or Geneve. All traffic between Cilium nodes is encapsulated. Encapsulation relies on normal node to node connectivity. This means that if Cilium nodes can already reach each other, all routing requirements are already met.

What is Cluster Mesh, and why do you need it?

Multi-cluster Kubernetes setups are often adopted for fault isolation, scalability, and geographical distribution. This approach can lead to networking complexities. In this context, traditional networking models struggle with service discovery, network segmentation, policy enforcement, and load balancing across clusters. Additionally, managing security protocols and policies across multiple environments can be challenging due to the distributed nature of services. You can read more about this in detail in a great resource from the Isovalent team about traffic engineering.

Try it out

Start with the Cluster Mesh lab and see how to enable Cluster Mesh in your enterprise environment.

The need?

• Multi-Cluster Networking– Isovalent Cluster Mesh allows you to connect the networks of multiple clusters so that pods in each cluster can discover and access services in all other clusters of the mesh, provided all the clusters run Cilium as their CNI. This effectively joins multiple clusters into a large unified network, regardless of the Kubernetes distribution or location each of them is running.
• High Availability and Fault Tolerance– Isovalent Cluster Mesh enhances your services with high availability and fault tolerance. It supports the operation of Kubernetes clusters in multiple regions or availability zones. If resources become temporarily unavailable, are misconfigured in one cluster, or offline for upgrades, it enables failover to other clusters, ensuring your services remain accessible at all times.
• Transparent Service Discovery- Isovalent Cluster Mesh automates service discovery across your Kubernetes clusters. This means your applications can discover and interact with services, regardless of their cluster, greatly simplifying cross-cluster communication.
• Effortless Pod IP Routing– Isovalent Cluster Mesh can handle Pod IP routing across multiple Kubernetes clusters at native performance. Using either tunneling or direct routing circumvents the need for any gateways or proxies. This allows your pods to communicate seamlessly across clusters, enhancing your microservice architecture’s overall efficiency.
• Shared Services Across Clusters– Isovalent Cluster Mesh enables sharing services such as secrets management, logging, monitoring, or DNS between all clusters. This reduces operational overhead, simplifies management, and maintains isolation between tenant clusters.
• Uniform Network Policy Enforcement– Isovalent Cluster Mesh extends Cilium’s Layer 3-7 network policy enforcement to all clusters in the mesh. It standardizes the application of network policies, ensuring a consistent security approach across your entire Kubernetes deployment, irrespective of the number of clusters involved.

Can we mesh clusters with different routing methods? Absolutely!!

Our engineering team has often been asked, “Can we interconnect multiple clusters configured with different routing modes?” With Mixed Routing Mode:

• Intra-cluster traffic is forwarded according to the mode configured for each given cluster.
• Inter-cluster traffic is forwarded according to the locally configured mode when matching that of the target cluster (for example, native routing if both clusters use native routing, encapsulation if both clusters use encapsulation)
• Traffic defaults to encapsulation when the modes of the source and destination clusters mismatch.
• In other words, mixed routing mode clusters are interconnected through tunnels.

An example configuration for Mixed Mode Routing while setting up clustermesh across two clusters:

# The same tunnel protocol (either vxlan or geneve) needs to be
# configured in all clusters.
tunnelProtocol: vxlan

# Enable mixed routing modes support
enterprise:
  clustermesh:
    mixedRoutingMode:
      enabled: true

 (...)

What is Isovalent Enterprise for Cilium?

Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.

Why Isovalent Enterprise for Cilium?

For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits:

• Advanced network policy: advanced network policy capabilities that enable fine-grained control over network traffic for micro-segmentation and improved security.
• Hubble flow observability + User Interface: real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
• Multi-cluster connectivity via Cluster Mesh: seamless networking and security across multiple cloud providers like AWS, Azure, Google, and on-premises environments.
• Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
• Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making it easy to deploy and manage complex microservices architectures.
• Enterprise-grade support: Enterprise-grade support from Isovalent’s experienced team of experts ensures that issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.

Pre-Requisites

The following prerequisites need to be taken into account before you proceed with this tutorial:

• Two up-and-running Kubernetes clusters. For this tutorial, we will create two Google Kubernetes Clusters.
  • The two clusters were created in two different regions; hence, VPC peering should be set up.
• The following dependencies should be installed:
  • Install gcloud CLI.
  • You should have a Google Cloud subscription.
  • Install kubectl.
  • Install Cilium CLI.
  • Install Helm.
• Cluster Mesh with Mixed Routing mode support requires Isovalent Enterprise for Cilium 1.15 or later.
  • Changing the tunnel protocol of an already configured production cluster setup in encapsulation mode is risky and can cause connectivity disruption. For more information, contact sales@isovalent.com and support@isovalent.com.
  • When planning to mesh existing clusters configured with different routing modes, first enable the Mixed Routing Mode support in all clusters.
    • New and existing connections will work normally during this phase, defaulting to the primary routing mode.
    • Once the feature is enabled in all clusters, you can merge all the clusters with mismatched routing modes.
• The same tunnel protocol must be configured in all clusters.
  • You cannot mesh together a cluster using VXLAN with another using Geneve as a tunneling protocol.
• The IPAM modes supported are
  • Cluster Scope
  • Kubernetes Host Scope
• Users can contact their partner Sales/SE representative(s) at sales@isovalent.com for more detailed insights into the features below and access the requisite documentation.

Creating the GKE clusters

Let’s briefly see the commands to create GKE clusters.

gcloud config set project PROJECT_ID

gcloud container clusters create amitgag-test-7580 --project ########### --zone us-west2-a

kubectl -n kube-system exec ds/cilium -- cilium status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.31 (v1.31.5-gke.1023000) [linux/amd64]
Kubernetes APIs:         ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    False   [eth0   10.168.15.225 fe80::4001:aff:fea8:fe1]
Host firewall:           Disabled
SRv6:                    Disabled
CNI Chaining:            none
CNI Config file:         successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                  Ok   1.15.13-cee.1 (v1.15.13-cee.1-176e3717)
NodeMonitor:             Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok
IPAM:                    IPv4: 4/254 allocated from 10.96.1.0/24,
ClusterMesh:             1/1 remote clusters ready, 0 global-services
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Host Routing:            Legacy
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       32/32 healthy
Proxy Status:            OK, ip 10.96.1.128, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 65536, max 131071
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 3.68   Metrics: Disabled
Encryption:              Disabled
Cluster health:          4/4 reachable   (2025-02-17T09:39:58Z)
Modules Health:          Stopped(0) Degraded(0) OK(14)

kubectl -n kube-system exec ds/cilium -- cilium-health status

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
Probe time:   2025-02-17T09:46:58Z
Nodes:
  amitgag-test-7580/gke-amitgag-test-7580-default-pool-30357df9-4jd9 (localhost):
    Host connectivity to 10.168.15.225:
      ICMP to stack:   OK, RTT=411.35µs
      HTTP to agent:   OK, RTT=226.32µs
    Endpoint connectivity to 10.96.1.91:
      ICMP to stack:   OK, RTT=414.68µs
      HTTP to agent:   OK, RTT=244.94µs
  amitgag-test-20646/gke-amitgag-test-20646-default-pool-ce415cec-kb4n:
    Host connectivity to 10.168.15.228:
      ICMP to stack:   OK, RTT=1.04648ms
      HTTP to agent:   OK, RTT=1.358299ms
    Endpoint connectivity to 10.0.1.243:
      ICMP to stack:   OK, RTT=1.05722ms
      HTTP to agent:   OK, RTT=499.019µs
  amitgag-test-20646/gke-amitgag-test-20646-default-pool-ce415cec-r7gl:
    Host connectivity to 10.168.15.227:
      ICMP to stack:   OK, RTT=1.03942ms
      HTTP to agent:   OK, RTT=307.89µs
    Endpoint connectivity to 10.0.0.184:
      ICMP to stack:   OK, RTT=1.06561ms
      HTTP to agent:   OK, RTT=507.09µs
  amitgag-test-7580/gke-amitgag-test-7580-default-pool-30357df9-q51x:
    Host connectivity to 10.168.15.223:
      ICMP to stack:   OK, RTT=1.06164ms
      HTTP to agent:   OK, RTT=643.93µs
    Endpoint connectivity to 10.96.0.233:
      ICMP to stack:   OK, RTT=1.06049ms
      HTTP to agent:   OK, RTT=724.56µs

What are the Pod IPs across clusters?

Once the GKE clusters are created, you can check that the Pods on the GKE clusters are on distinct IP addresses.

gcloud container clusters describe amitgag-test-7580 --zone=us-west2-a |   grep -e clusterIpv4Cidr
clusterIpv4Cidr: 10.96.0.0/14
  clusterIpv4Cidr: 10.96.0.0/14
  clusterIpv4CidrBlock: 10.96.0.0/14

How can you enable Cluster Mesh with Mixed Routing Mode?

• To set up Cluster Mesh, reach out to sales@isovalent.com to get access to the complete Enterprise documentation.
• Some key pre-requisites to set up Cluster Mesh:
  • A unique Cluster ID and Cluster Name must identify all clusters.
  • Install the cert-manager CRDs and set up the cilium issuer associated with the same Certification Authority in all clusters.
    • It doesn’t have to be via Cert-Manager, but it’s highly recommended, as manual CA cert copying and pasting is error-prone.
• Create a sample yaml file. (Unique per cluster)
  • The yaml configuration file contains the basic properties to set up Cilium, Cluster Mesh, and Hubble.
    • Configures Cilium in CRD identity allocation mode.
    • Enables Hubble and Hubble Relay.
    • Enables the Cluster Mesh API Server and exposes it using a service of Type LoadBalancer. Cloud-provider-specific annotations are added to force the usage of private IP addresses.
    • Enables the automatic generation of the certificates using cert-manager, leveraging the existing cilium Issuer associated with the shared certificate authority.
    • Configures the most granular cross-cluster authentication scheme for improved segregation. 
• Sample configuration file for Cluster Mesh on both the clusters:
  • Cluster-1 will be set up with IPAM=Kubernetes and Routing Mode=native
  • Cluster-2 will be set up with IPAM=cluster-pool and Routing Mode=tunnel with encapsulation=VXLAN.

cluster:
  id: 1
  name: amitgag-test-7580

clustermesh:
  useAPIServer: true

  apiserver:
    service:
      type: LoadBalancer
      annotations:
        networking.gke.io/load-balancer-type: "Internal"

    tls:
      authMode: cluster
      auto:
        enabled: true
        method: certmanager
        certManagerIssuerRef:
          group: cert-manager.io
          kind: Issuer
          name: cilium

hubble:
  enabled: true
  relay:
    enabled: true

  tls:
    auto:
      enabled: true
      method: certmanager
      certManagerIssuerRef:
        group: cert-manager.io
        kind: Issuer
        name: cilium

nodeinit:
  enabled: true
  reconfigureKubelet: true
  removeCbrBridge: true
cni:
  binPath: /home/kubernetes/bin
gke:
  enabled: true

# The nativeRoutingCIDR must include the PodCIDRs of all clusters to
# be connected together. The PodCIDR of a cluster can be retrieved through:
# $ gcloud container clusters describe ${NAME} --zone ${ZONE} --format 'value(clusterIpv4Cidr)'
ipv4NativeRoutingCIDR: 10.96.0.0/14

ipam:
  mode: kubernetes

• To enable support for the Mixed Routing Mode, you would need to use the following helm flag:

# The same tunnel protocol (either vxlan or geneve) needs to be
# configured in all clusters.
tunnelProtocol: vxlan

# Enable mixed routing modes support
enterprise:
  clustermesh:
    mixedRoutingMode:
      enabled: true

 (...)

• Install Isovalent Enterprise for Cilium and connect the clusters using the Cluster Mesh documentation.

How can you verify Cluster Mesh status?

Check the status of the clusters by running cilium clustermesh status on either of the clusters.  If you use a service of type LoadBalancer, it will also wait for the LoadBalancer to be assigned an IP.

cilium clustermesh status --wait

✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ Cluster access information is available:
  - 10.168.15.215:2379
✅ Deployment clustermesh-apiserver is ready
ℹ️  KVStoreMesh is disabled

✅ All 2 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]

🔌 Cluster Connections:
  - amitgag-test-20646: 2/2 configured, 2/2 connected

🔀 Global services: [ min:0 / avg:0.0 / max:0 ]

kubectl exec -n kube-system -ti ds/cilium -- cilium-dbg status --all-clusters

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.31 (v1.31.5-gke.1023000) [linux/amd64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   False   [eth0   10.168.15.225 fe80::4001:aff:fea8:fe1]
Host firewall:          Disabled
SRv6:                   Disabled
CNI Chaining:           none
CNI Config file:        successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium:                 Ok   1.15.13-cee.1 (v1.15.13-cee.1-176e3717)
NodeMonitor:            Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok
IPAM:                   IPv4: 4/254 allocated from 10.96.1.0/24,
ClusterMesh:            1/1 remote clusters ready, 0 global-services
   amitgag-test-20646: ready, 2 nodes, 6 endpoints, 4 identities, 0 services, 1 reconnections (last: 7h24m3s ago)
   └  etcd: 1/1 connected, leases=0, lock leases=1, has-quorum=true: endpoint status checks are disabled, ID: 387b9ba1e51328b5
   └  remote configuration: expected=true, retrieved=true, cluster-id=2, kvstoremesh=false, sync-canaries=true
   └  synchronization status: nodes=true, endpoints=true, identities=true, services=true
IPv4 BIG TCP:            Disabled
IPv6 BIG TCP:            Disabled
BandwidthManager:        Disabled
Host Routing:            Legacy
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       32/32 healthy
Proxy Status:            OK, ip 10.96.1.128, 0 redirects active on ports 10000-20000, Envoy: embedded
Global Identity Range:   min 65536, max 131071
Hubble:                  Ok              Current/Max Flows: 4095/4095 (100.00%), Flows/s: 3.70   Metrics: Disabled
Encryption:              Disabled
Cluster health:          4/4 reachable   (2025-02-17T13:02:58Z)
Modules Health:          Stopped(0) Degraded(0) OK(14)

How can you validate Mixed Routing Mode?

• When Mixed Routing Modes support is enabled, each Cilium agent advertises the supported routing modes through the corresponding CiliumNode resource and equivalent kvstore representation.
• On the first cluster, nodes are configured with native routing as primary routing mode, and VXLAN encapsulation as a fallback.

kubectl get ciliumnode -o custom-columns='NAME:.metadata.name,MODES:.metadata.annotations.routing\.isovalent\.com/supported'
NAME                                               MODES
gke-amitgag-test-7580-default-pool-30357df9-4jd9   native,tunnel/vxlan
gke-amitgag-test-7580-default-pool-30357df9-q51x   native,tunnel/vxlan

• On the second cluster, nodes are configured with VXLAN encapsulation as the primary mode.

kubectl get ciliumnode -o custom-columns='NAME:.metadata.name,MODES:.metadata.annotations.routing\.isovalent\.com/supported'
NAME                                                MODES
gke-amitgag-test-20646-default-pool-ce415cec-kb4n   tunnel/vxlan
gke-amitgag-test-20646-default-pool-ce415cec-r7gl   tunnel/vxlan

Conclusion

The evolution of network architectures poses challenges, and the Isovalent team is here to help you overcome them. Mixed Routing Mode is one such challenge, but as you can see, we can easily overcome it. Hopefully, this post gave you an overview of setting up Isovalent Cilium Enterprise’s Cluster Mesh with Mixed Routing Modes. You can schedule a demo with our experts if you’d like to learn more.