Modern-day enterprise networks are complex and dynamic, designed to support various applications and services while ensuring security, scalability, and reliability. Kubernetes has become a go-to platform for enterprises looking to modernize their application deployment and management processes. With Multi-Cloud and Hybrid Cloud support, Enterprises can leverage multiple cloud providers or combine on-premises and cloud resources, optimizing costs and performance based on specific needs. Now imagine a team creating a sandbox environment where some of the Pods appear to have the same IP address that Pods have on another cloud provider, leading to massive network outages and application downtime. Isovalent Enterprise for Cilium offers a unique headway into this problem by connecting multiple Kubernetes clusters or swarms of clusters across cloud providers, hybrid cloud set-ups, and mitigating the overlapping Pod IP addressing problem. This tutorial will guide you through setting up Isovalent Cilium Enterprise’s Cluster Mesh with overlapping Pod CIDR. What leads to this scenario in an Enterprise Network? Overlapping IP addresses occurs when identical IP ranges are allocated across different networks or applications, causing massive communication hurdles. Here are the key causes: Merger and Acquisition When organizations merge, they may have existing networks with overlapping CIDR blocks. Integrating these networks without proper planning can lead to conflicts. Dynamic Environment In dynamic environments (e.g., cloud-native applications), rapid resource provisioning can lead to overlapping CIDRs if not managed carefully, especially in auto-scaling or microservices architectures. Multiple Cloud Providers Using different cloud providers without a unified IP addressing strategy can lead to conflicts when the same CIDR ranges are employed across providers. Legacy Systems Legacy systems may have fixed IP addresses or ranges that conflict with newer allocations, especially if the legacy systems are poorly documented. Suboptimal IP Address Management (IPAM) Failure to use robust IP address management tools can result in misallocations and overlaps, particularly in large or complex networks. Testing Environments Duplicate IP ranges may be used in test or staging environments that mirror production setups, leading to conflicts when integrating these environments. How does Isovalent address this issue? You can minimize these interruptions by using Isovalent Enterprise for Cilium’s Cluster Mesh support for Overlapping Pod CIDR or deploying an Egress Gateway. Let’s look at the Overlapping Pod CIDR support from Isovalent. How do packets traverse a ClusterMesh with Overlapping IP CIDRs? The following diagram provides an overview of the packet flow of Cluster Mesh with overlapping Pod CIDR support. Inter-cluster communication must be performed in this mode through Global or Phantom service. Global Service– Isovalent can load balance traffic to Pods across all clusters in a Cluster Mesh. This is achieved by using global services. A global service is a service that is created with the same spec in each cluster and annotated with service.cilium.io/global: "true". Global service will load balance across all available backends in all clusters by default. Phantom Service– Global services mandate that an identical service be present in each cluster from which the service is accessed. Phantom services lift this requirement, allowing a given service to be accessed from remote clusters even if it is not there. A phantom service is a LoadBalancer service associated with at least one VIP and annotated with service.isovalent.com/phantom: "true". This makes the phantom service LoadBalancer IP address accessible from all clusters in the Cluster Mesh. Source IP addresses and identities are preserved for cross-cluster communication. When the traffic crosses the cluster boundary, the source IP address is translated to Node IP. For Intra-Cluster communication, the source IP address is preserved. Intra-cluster communication via a service leads to a destination IP translation. What is Isovalent Enterprise for Cilium? Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns. Why Isovalent Enterprise for Cilium? For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits: Advanced network policy: advanced network policy capabilities that enable fine-grained control over network traffic for micro-segmentation and improved security. Hubble flow observability + User Interface: real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management. Multi-cluster connectivity via Cluster Mesh: seamless networking and security across multiple cloud providers like AWS, Azure, Google, and on-premises environments. Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments. Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making it easy to deploy and manage complex microservices architectures. Enterprise-grade support: Enterprise-grade support from Isovalent’s experienced team of experts ensures that issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments. What is supported alongside Overlapping IP CIDR? Pre-Requisites The following prerequisites need to be taken into account before you proceed with this tutorial: Two up-and-running Kubernetes clusters. For this tutorial, we will create two Azure Kubernetes Service clusters using the network-plugin as BYOCNI: Bring your Own CNI The AKS clusters can be created without kube-proxy. The following dependencies should be installed: Install Azure CLI. You should have an Azure Subscription. Install kubectl. Install Cilium CLI. Install Helm. Cluster Mesh with Overlapping PodCIDR requires Isovalent Enterprise for Cilium 1.13 or later. Users can contact their partner Sales/SE representative(s) at sales@isovalent.com for more detailed insights into the features below and access the requisite documentation. Creating the AKS clusters Let’s briefly see the commands to create AKS clusters with the network plugin BYOCNI. What are the Pod IPs across clusters? Once the AKS clusters are created, you can check that the Pods on the AKS clusters are on the same IP addresses. Cluster-1 Cluster-2 How can you Peer the AKS clusters? Use VNet peering to peer the AKS clusters across the two chosen regions. This step only needs to be done in one direction. The connection will automatically be established in both directions. Login to the Azure Portal Click Home Click Virtual Network Select the respective Virtual Network Click Peerings Click Add Give the local peer a name Select “Allow cluster1 to access cluster2” Give the remote peer a name Select the virtual network deployment model as “Resource Manager” Select the subscription Select the virtual network of the remote peer Select “Allow cluster2 to access cluster1” Click Add How can you enable Cluster Mesh with overlapping IP CIDR? To set up cluster-mesh, reach out to sales@isovalent.com to get access to the complete Enterprise documentation. Some key pre-requisites to set up Cluster Mesh: A unique Cluster ID and Cluster Name must identify all clusters. All clusters must be configured with the same datapath mode, in other words, either native routing or encapsulation (using the same encapsulation protocol). Overlapping PodCIDR mandates tunneling. The two clusters should be set up with Kube-Proxy Replacement set to true. Install the cert-manager CRDs and set up the cilium issuer associated with the same Certification Authority in all clusters. It doesn’t have to be via Cert-Manager, but it’s highly recommended, as manual CA cert copying and pasting is error-prone. Create a sample yaml file. (Unique per cluster) The yaml configuration file contains the basic properties to set up Cilium, Cluster Mesh, and Hubble. Configures Cilium in CRD identity allocation mode. Enables Hubble and Hubble Relay. Enables the Cluster Mesh API Server and exposes it using a service of Type LoadBalancer. Cloud-provider-specific annotations are added to force the usage of private IP addresses. Enables the automatic generation of the certificates using cert-manager, leveraging the existing cilium Issuer associated with the shared certificate authority. Configures the most granular cross-cluster authentication scheme for improved segregation. Sample configuration file for Cluster Mesh: To enable support for the overlapping Pod CIDR feature, you would need to use the following helm flag: How can you verify Cluster Mesh status? Check the status of the clusters by running cilium clustermesh status on either of the clusters. If you are using a service of type LoadBalancer then this will also wait for the LoadBalancer to be assigned an IP. Cluster-1 Cluster-2 What’s the status of the clustermesh-api pod? The Cluster Mesh API Server contains an etcd instance to keep track of the cluster’s state. The state from multiple clusters is never mixed. Cilium agents in other clusters connect to the Cluster Mesh API Server to watch for changes and replicate the multi-cluster state into their cluster. Access to the Cluster Mesh API Server is protected using TLS certificates. Access from one cluster to another is always read-only, ensuring failure domains remain unchanged. A failure in one cluster never propagates to other clusters. Ensure that the clustermesh-api pod is running on both clusters. How can you test Inter-Cluster Service communication? Let’s deploy a client on Cluster- Deploy a Deployment and Global Service on both clusters. Check connectivity. Notice that when the backend of a local cluster is selected, the response shows the original source IP of the Pod. Check connectivity again. Notice that when the backend of a remote cluster is selected, the response shows the IP of the node on which the source Pod is running. How can you use L3/L4 policies in an Overlapping Pod CIDR scenario? When using Cilium, endpoint IP addresses are irrelevant when defining security policies. Instead, you can use the labels assigned to the Pods to define security policies. The policies will be applied to the right Pods based on the labels, irrespective of where or when they run within the cluster. The layer 3 policy establishes the base connectivity rules regarding which endpoints can talk to each other. The layer 4 policy can be specified independently or independently in addition to the layer 3 policies. It restricts an endpoint’s ability to emit and/or receive packets on a particular port using a protocol. A cilium network policy is always only applied to a single cluster and not automatically synced to the remote one if Cluster Mesh is enabled. Let’s deploy an egress policy on cluster 1. Let’s deploy an ingress policy on cluster 2. Check connectivity again. You will notice that the pod cannot connect to local backends but will connect only to remote backends. Troubleshooting Cluster Mesh Issues You can use the following commands to troubleshoot Cluster Mesh-related deployments. Once the clusters are connected via Cluster Mesh, you can check the health of Nodes from either cluster. Notice Nodes for both clusters are displayed. Check the service endpoints from cluster 1; the remote endpoints are marked as preferred with the @2 suffix that denotes the Cluster-ID of the cluster. Notice the httpbin-service has cluster-IP 10.11.42.230, and the remote backend Pods in cluster-2 are the preferred Pods with IPs 10.10.0.252@2 and 10.10.4.129@2. Verify whether Cilium agents are successfully connected to all remote clusters. Multiple causes can prevent Cilium agents (or KVStoreMesh, when enabled) from correctly connecting to the remote etcd cluster, being it the sidecar instance part of the clustermesh-apiserver, or a separate etcd cluster when Cilium operates in KVStore mode. Cilium features an automatic cilium-dbg troubleshoot Cluster Mesh command, which performs automatic checks to validate DNS resolution, network connectivity, mTLS authentication, etcd authorization, and more and reports the output in a user-friendly format. Conclusion The evolution of network architectures poses challenges, and the Isovalent team is here to help you overcome them. Overlapping addresses is one such challenge, but as you can see, we can easily overcome it. Hopefully, this post gave you an overview of setting up Isovalent Cilium Enterprise’s Cluster Mesh with overlapping Pod CIDR. You can schedule a demo with our experts if you’d like to learn more. Try it out Start with the Cluster Mesh lab and see how to enable Cluster Mesh in your enterprise environment. Further reading Deploying a Cluster Mesh in Azure Marketplace using Isovalent Enterprise for Cilium Support for Overlapping IP CIDR in 1.13 release blog announcement Isovalent and Azure Kubernetes Service Topology Aware Routing, Canary Testing with Cluster Mesh