Kubernetes doesn’t provide a network interface system by default. Instead, network plugins provide this functionality. AWS suggests several alternate CNI plugins with Cilium and Isovalent Enterprise for Cilium as popular alternatives. Until recently, EKS clusters would boot with the default networking add-ons and would require users to take some additional steps to install these alternative plug-ins. With the recently announced ability to create an EKS cluster minus the default networking add-on (AWS VPC CNI), you can install CNI plugins like Isovalent Enterprise for Cilium with zero effort. This blog shows how to initially deploy an EKS cluster without a preinstalled CNI plugin and then add Cilium as the CNI plugin. Why should you disable the add-ons? Every EKS cluster automatically comes with default networking add-ons, including Amazon VPC CNI, CoreDNS, and Kube-Proxy, providing critical functionality that enables pod and service operations for EKS clusters. Many EKS users often choose to use Cilium and Isovalent Enterprise for Cilium to remove Kube-Proxy and instead use the full powers of eBPF. However, installing Cilium or any 3rd party networking plugin would have required you to add taints to your nodegroups or patch the aws daemonset (see below). As you will see in the tutorial below, disabling the add-ons at bootup greatly simplifies the installation and operation of Cilium and Isovalent Enterprise for Cilium. What is Isovalent Enterprise for Cilium? Isovalent Enterprise for Cilium is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns. Why Isovalent Enterprise for Cilium on EKS? For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits: Advanced network policy: Isovalent Enterprise for Cilium provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy. These capabilities enable fine-grained control over network traffic for micro-segmentation and improved security. Hubble flow observability + User Interface: Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management. Multi-cluster connectivity via Cluster Mesh: Isovalent Enterprise for Cilium provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments. Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments. Service Mesh: Isovalent Enterprise for Cilium provides sidecar-free, seamless service-to-service communication and advanced load balancing, making it easy to deploy and manage complex microservices architectures. Enterprise-grade support: Isovalent Enterprise for Cilium includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments. Prerequisites The following prerequisites need to be taken into account before you proceed with this tutorial: Access to AWS. Create a new account for free. The Cilium operator requires the following EC2 privileges to perform ENI creation and IP allocation. Ensure you have a cluster IAM role if you’re going to create your cluster with eksctl. To ensure Cilium works properly in an environment that requires firewall rules to enable connectivity, you must add the respective firewall rules. Install kubectl Install Helm Install eksctl (make sure the version is 0.186.0 and higher) Install awscli Cilium CLI (optional): Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s). How can I disable the add-ons? Every EKS cluster automatically comes with default networking add-ons (see below) that provide critical functionality that enables pod and service operations for EKS clusters. Amazon VPC CNI CoreDNS, and Kube-Proxy You now have the option to create an EKS cluster where you can skip these add-ons during cluster creation. This is possible in the following ways: AWS Console AWS CLI API‘s This tutorial will refer to the eksctl method of creating the EKS cluster minus these add-ons by creating an EKS cluster, installing Isovalent Enterprise for Cilium, and then adding a managed node group. Points to consider You might think, why can’t you create an EKS cluster with a managed group in one go? As you can see below, if you try to create an EKS cluster without the add-ons and try to add a managed group during cluster creation, that is not supported, as the cluster is being created without AWS-VPC-CNI. How do you create an EKS cluster (with no add-ons)? As listed in the pre-requisites section, you need to ensure that the eksctl version is 0.186.0 or higher. Create a ClusterConfig file to create an EKS cluster (e.g., default-ipv4-no-addons.yaml) For this tutorial, we will be disabling Kube-Proxy AWS-VPC-CNI Create the cluster Check the status of the Pods. The cluster would be running just the core-dns. Pods are pending, which is fine, as no CNI runs on the cluster. How can I deploy Isovalent for Cilium as the CNI on the EKS cluster? With the EKS cluster up and running, you can install Isovalent Enterprise for Cilium via two methods: Helm values and repo can be obtained by contacting sales or support@isovalent.com You can also install Isovalent Enterprise for Cilium from the AWS marketplace on this newly created EKS cluster. Once the Helm details have been obtained, ensure you have the EKS cluster Kubernetes API URL and port details. Specifying this is necessary as the EKS cluster is created explicitly without setting up Kube-Proxy. Consequently, no component is currently doing the cluster’s internal L4 load balancing. These details can be retrieved via kubectl. The Cilium agent needs to be made aware of this information with the following configuration values when installing Cilium: Check the status of the Pods. Notice that no Kube-Proxy Pods have been created anymore. We are adding a node group for this tutorial after the EKS cluster creation. Note- You no longer have to add any taints for Cilium, which was required previously to ensure application. Pods will only be scheduled once Cilium is ready to manage them. Create the node group. Check the status of the nodes and make sure they are in a “Ready” state. Check the status of the Pods. Check if any AWS daemonsets are running on the EKS cluster. Validate Cilium version Validate health check cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking and connectivity. You can check node-to-node health with cilium-health status: Cilium Connectivity Test (Optional) The Cilium connectivity test deploys a series of services and deployments, and CiliumNetworkPolicy will use various connectivity paths to connect. Connectivity paths include with and without service load-balancing and various network policy combinations. Conclusion Hopefully, this post gave you a good overview of deploying an EKS cluster without a preinstalled CNI plugin and then adding Isovalent Enterprise for Cilium as the CNI plugin. You can schedule a demo with our experts if you’d like to learn more. Try it out Choose your way to explore Cilium with our Cilium learning tracks focusing on features important to engineers using Cilium in cloud environments. Cilium comes in different flavors, whether using GKE, AKS, or EKS; not all these features will apply to each managed Kubernetes Service. However, it should give you a good idea of features relevant to operating Cilium in cloud environments. Suggested Reading Partners: AWS and Isovalent bring Cilium to Enterprise Installing Isovalent from the AWS marketplace for EKS or EKS-A for your hybrid, multi-cloud, or on-prem environments. Enabling Enterprise features using Isovalent from the AWS marketplace. Isovalent Enterprise for Cilium on the AWS marketplace. Deep dive into Cilium and EKS-Anywhere. Amazon introduces cluster creation flexibility.