A network migration is a daunting task that most network engineers will eventually have to face in their career. I remember having once to plan a major network migration program. It was a Herculean challenge that took months to complete and while it was eventually successful, we ran into many hiccups along the way (I may have caused a minor routing loop on a major service provider network when making a BGP change back in 2011 - oops!). I am not sure whether the era of cloud native has made it any easier: on one hand, we now have programmable networks, documented APIs and we have adopted a culture of automation that should make large-scale migration far more repeatable and consistent. On the other hand, we've added multiple layers of networking - Linux networking stack, container networking, virtual network overlay - to the already complex physical network. But whether you are looking at migrating to a new data center network or to a different Container Network Interface (CNI), the challenges remain the same: How can we migrate from one network to another, with minimal disruption? How can do ensure connectivity between workloads on the old network and the new network during the transition? How can we repeat the migration process across a number of network nodes? In this blog post, we will look at answering these questions and explore how we can elegantly migrate from any CNI to Cilium. We will consider various migration approaches and we will walkthrough a migration from Flannel to Cilium, using a recently-released feature that will make migrations easier for users. While this blog post focuses on migrating from Flannel, that the same approach should apply to other CNIs too. Note we won't explore why you should move to Cilium but if you're not convinced yet, I suggest you head out to our networking, security and observability pages to learn more. CNI Migration Considerations and Approaches Before we talk about CNI migration, we should review what a CNI does and how it actually works. When the kubelet creates a Pod’s sandbox, the CNI specified in the configuration file located in the /etc/cni/net.d/ directory is called. The CNI will handle the networking for a Pod - including: allocating an IP address, creating & configuring a network interface, and (potentially) establishing an overlay network. When migrating CNIs, there are several approaches with pros and cons. Migration Approaches The ideal scenario would be to build a brand new cluster and to migrate workloads (ideally, using a GitOps approach). But this can involve a lot of preparation and potential disruptions. Another method consists in reconfiguring /etc/cni/net.d/ to point to Cilium. However, any existing Pods will still have been configured by the old network plugin and any new Pods will be configured by the newer CNI. To complete the migration, all Pods on the cluster that are configured by the old CNI must be recycled in order to be a member of the new CNI. A naive approach to migrating a CNI would be to reconfigure all nodes with a new CNI and then gradually restart each node in the cluster, thus replacing the CNI when the node is brought back up and ensuring that all pods are part of the new CNI. This simple migration, while effective, comes at the cost of disrupting cluster connectivity during the rollout. Unmigrated and migrated nodes would be split in to two “islands” of connectivity, and pods would be randomly unable to reach one-another until the migration is complete. In this blog post, you will learn about a new hybrid approach. Hybrid Migration Mode Cilium supports a hybrid mode, where two separate overlays are established across the cluster. While Pods on a given node can only be attached to one network, they have access to both Cilium and non-Cilium Pods while the migration is taking place. That's as long as Cilium and the existing network use a separate IP range. Migration Overview The migration process utilizes the per-node configuration feature to selectively enable Cilium CNI. This allows for a controlled rollout of Cilium without disrupting existing workloads. Cilium will be installed, first, in a mode where it establishes an overlay but does not provide CNI networking for any pods. Then, individual nodes will be migrated. In summary, the process looks like: Prepare the cluster and install Cilium in “secondary” mode. Cordon, drain, migrate, and reboot each node. Remove the existing network provider. (Optional) Reboot each node again. Requirements This approach to our migration requires the following: A new, distinct Cluster CIDR for Cilium to use. Use of the Cluster Pool IPAM mode. A distinct network overlay, either a different protocol (Geneve instead of VXLAN for example) or port. An existing network plugin that uses the Linux routing stack, such as Flannel or Calico. Let's now go through a migration. Step 1 - Check the existing cluster First, let's have a look at migrating away from Flannel. Flannel is a very popular and simple CNI with widespread adoption in home lab environments. It has however limited routing and security features (it does not support the use of Network Policies, does not support Ingress/Gateway API, does not benefit from the performance gains from eBPF, etc...). Let's first look at our Kubernetes cluster (deployed via kind). It's made up of two worker nodes and one control plane node. Flannel is deployed and running with no issues: Let's check the PodCIDR (the IP address range from which the Pods will pick up an IP from) on each node. It's from the 10.244.0.0/16 range - take note of this as this will be important later. Just to illustrate the connectivity during the migration process, we've deployed a Deployment of 10 nginx Pods. The ten Pods have been distributed across both worker nodes. Flannel would have allocated IP addresses from the PodCIDRs of the nodes where the Pods are deployed. Let's verify that: Step 2 - Prepare for the Migration First, we need to select a new CIDR for Pods. It must be distinct from all other CIDRs in use and choosing a different CIDR will enable us to maintain connectivity during the migration. For kind clusters, the default is 10.244.0.0/16 and is the one in use as we saw earlier. So, for this example, we will use 10.245.0.0/16. Next, we need to select a different encapsulation protocol (Geneve instead of VXLAN for example) or a distinct encapsulation port. For this example, we will use VXLAN with a non-default port of 8473 (the default is 8472). We will now create a Cilium configuration file that we will use during the installation of Cilium. The Cilium configuration file will based on a combination of the parameters below (defined in the Helm configuration file values-migration.yaml below) and parameters based on your own environment. Let's review some of the key parameters first: This is there to prevent Cilium from restarting Pods that are not being managed by Cilium (we don't want to disrupt the Pods that are managed by Flannel and not by Cilium). As highlighted earlier, this setting here specifies the different encapsulation port for VXLAN. The first setting above temporarily skips writing the CNI configuration (customConf: true). This is to prevent Cilium from taking over immediately. Note the customConf will be switched it back to the default false at the end of the migration. The second setting above will prevent the CNI configuration file and plugin binaries to be removed which is recommended during the migration (uninstall: false). As highlighted earlier, we recommend the use of cluster-pool IPAM mode and a distinct PodCIDR during the migration. The above disables the enforcement of network policy until the migration is completed. We will enforce network policies post-migration. This flag should route traffic via host stack to provide connectivity during the migration. We will verify during the migration that Flannel-managed pods and Cilium-managed pods have connectivity. We now need to use these settings and apply them to your own specific environment. For this, let's use the Cilium CLI. We saw in a previous tutorial how cilium-cli can be used to install Cilium. In this instance, we will use it to auto-detect settings specific to the underlying cluster platform (kind in this particular post but could be minikube, GKE, AKS, EKS, etc...) and use helm to install Cilium. With the following command, we can: Create a new Helm values file called values-initial.yaml Pull from values-migration.yaml the non-default values Fill in the missing values through the use of the helm-auto-gen-values flag Let's review the created file. It is a combination of the the values pulled from the values-migration.yaml file and the one auto-generated by the Cilium CLI. Step 3 - Install Cilium as a second overlay Let's now install Cilium using helm and the values we have just generated. At this point, we have a cluster with Cilium installed and an overlay established, but no Pods managed by Cilium itself. Let's verify this with the cilium status command. Note that none of the 13 Pods are currently managed by Cilium. That's to be expected. You can also confirm this by checking the CNI Configuration on the node: As you can see, the Cilium CNI configuration file has not been written in yet. Step 4 - Deploy the Cilium Node Config To migrate gradually and to minimize the disruption during the migration, we are going to be using a new feature introduced in Cilium 1.13: the CiliumNodeConfig object. The Cilium agent process supports setting configuration on a per-node basis instead of constant configuration across the cluster. This allows overriding the global Cilium config for a node or set of nodes. It is managed by CiliumNodeConfig objects. A CiliumNodeConfig object consists of a set of fields and a label selector. The label selector defines to which nodes the configuration applies. Let's now create a per-node config that will instruct Cilium to “take over” CNI networking on the node. Initially, this will not apply to any nodes. As you can see in the spec.nodeSelector section, the CiliumNodeConfig only applies to nodes with the io.cilium.migration/cilium-default: "true" label. We will gradually migrate nodes by applying the label to each node, one by one. Once the node is reloaded, the custom Cilium configuration will be applied, the CNI configuration will be written and the CNI functionality will be enabled. Step 5 - Start the Migration Remember that we deployed 10 replicas of an nginx image earlier. You should see Pods spread across both worker nodes. Cordon and Drain the Node It is recommended to always cordon and drain at the beginning of the migration process, so that end-users are not impacted by any potential issues. Let's remind ourselves the differences between "cordon" and "drain": Cordoning a node will prevent new Pods from being scheduled on the node. Draining a node will gracefully evict all the running Pods from the node. This ensures that the Pods are not abruptly terminated and that their workload is gracefully handled by other available nodes. Let's get started with kind-worker: To show that the node has been cordoned off, let's scale the deployment to 12 from 10 with the following command: As you can see, no new nginx instance is deployed on kind-worker as it's cordoned off (that's why we have 7 Pods on kind-worker2 and 5 on kind-worker). Let's now drain the node. Note that we use the ignore-daemonset flag as several DaemonSets are still required to run. You should know that, when we drain a node, the node is automatically cordoned. We did it first in this instance to provide clarity in the migration process. Let's verify no Pods are running on the drained node. The 12 pods are all running on kind-worker2. We can now label the node: this causes the CiliumNodeConfig to apply to this node. Let's restart Cilium on the node. That will trigger the creation of CNI configuration file. Finally, we can reboot the node. As we are using Kind, simulating a node reboot is as simple as restarting the Docker container. Let's take another look at the CNI configuration file: Note how there is now a Cilium configuration file present! Let's deploy a Pod and verify that Cilium allocates the IP to the Pod. Remember that we rolled out Cilium in cluster-scope IPAM mode where Cilium assigns per-node PodCIDRs to each node and allocates IPs on each node. The Cilium operator will manage the per-node PodCIDRs via the CiliumNode resource. The following command will check the CiliumNode resource and will show us the Pod CIDRs used to allocate IP addresses to the pods: Let's verify that, when we deploy a Pod on the migrated node, that the Pod picks an IP from the Cilium CIDR. The command below deploys a temporary Pod on the node and outputs the Pod's IP details (filtering on the Cilium Pod CIDR 10.245). Note we use the toleration to override the cordon. As you can see, the temporary Pod picks up an IP from the new range. Let's test connectivity between Pods on the existing overlay and the new Cilium-overlay. Let's first get the IP of one of the NGINX pod that was initially deployed. This Pod should still be on the Flannel network. This command will spin up a temporary container on the Cilium-managed network that will connect with curl to one of the nginx pods. We use grep to filter the response so that we only see the response code. As the HTTP response code is a successful 200, we've just established that we have successful connectivity during the migration! We can finally uncordon the migrated node with: Step 6 - Repeat for the next node(s) We can now proceed to the migration of the next worker node. Let's cordon and drain the node: Let's verify no Pods are running on the drained node (they should have been recreated over on the already-migrated node and should be all on the 10.245 IP range): The drained Pods are restarted on the already-migrated nodes. It's therefore no surprise that Cilium is now managing most of the Pods. We can now label the node, restart Cilium on it, reboot the node and uncordon it like we did earlier. You can now repeat the same process on the other nodes until the cluster is completed migrated. At the end, the status of Cilium should be OK and all pods should be managed by Cilium: Step 7 - Clean-up post-migration Now the migration has been completed, let's update the Cilium configuration to support Network Policies and remove the previous network plugin. Now that Cilium is healthy, let's update the Cilium configuration. First, let's create the right configuration file. Again, we are using the cilium-cli to generate an updated Helm config file. As you can see from checking the differences between the two files, we are only changing three parameters. We are: Enabling Cilium to write the CNI configuration file. Enabling Cilium to restart unmanaged Pods. Enabling Network Policy Enforcement. Let's apply it: Let's remove Flannel as it is no longer needed: And we are done! Conclusion Migrating CNIs is not a task most users look forward to but we think this new method will give users the option to gracefully migrate their clusters to Cilium. We also think the experience can be even further improved by leveraging the new CRD and building some tooling around it to facilitate the migration for some of the larger clusters. We would love your feedback - you can find us on the Cilium Slack channel!