Security is at the forefront of most cloud architects' minds. Engineers often turn to isolated and private environments in businesses with stringent security requirements. Due to the nature of the cloud, where resources are commonly shared, options to cordon off your environments are limited. In the case of Azure Kubernetes Service users, there is, however, a robust option: private AKS clusters. A private cluster means the managed API server created has internal IP addresses, and your Kubernetes nodes talk to it over your VNet. Private AKS cluster operators will want to benefit from Isovalent Enterprise for Cilium's zero-trust tooling to optimize their security posture. This tutorial will guide you in creating a private AKS cluster with Azure CNI Powered by Cilium and upgrading to Isovalent Enterprise for Cilium. What is Isovalent Enterprise for Cilium? Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns. Why Isovalent Enterprise for Cilium? For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, "Isovalent Enterprise for Cilium” is recommended with the following benefits: Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security. Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management. Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments. Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments. Service Mesh: Isovalent Cilium Enterprise provides seamless service-to-service communication that’s sidecar-free and advanced load balancing, making it easy to deploy and manage complex microservices architectures. Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments. Pre-Requisites The following prerequisites need to be taken into account before you proceed with this tutorial: You should have an Azure Subscription. Install kubectl Install Cilium CLI. Users can contact their partner Sales/SE representative(s) at sales@isovalent.com for more detailed insights into the features below and access the requisite documentation and Cilium CLI software images. Azure CLI version 2.48.1 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI. Permission to configure Azure Firewall. Familiarity with Azure Hub and Spoke Architecture. VNet peering is in place across both VNet’s. You must ensure that the following network and FQDN/application rules are required for AKS cluster extensions to work seamlessly. In the case of a private AKS cluster, configure rules to access an Azure container registry behind a firewall. The endpoints could be: REST endpoints. Dedicated data endpoints. Example: arcmktplaceprod.westeurope.data.azurecr.io Registry FQDN’s. Note– this is a mandatory requirement as the images for the cilium-* and cilium-operator-* pods are pulled from these Microsoft repositories. Ensure you have enough quota resources to create an AKS cluster. Go to the Subscription blade, navigate to “Usage + Quotas,” and make sure you have enough quota for the following resources: -Regional vCPUs -Standard Dv4 Family vCPUs What does our deployment consist of? In this deployment example, you will create an AKS cluster in a Spoke VNet and an Azure Firewall in the Hub VNet. Using a private link, you can broadcast the AKS control plane's private IP address, which runs in an Azure-managed virtual network, into an IP address that is part of your AKS subnet. To execute all Kubernetes and Azure CLI-related commands, you will create a Jumpbox VM in the Hub VNet. You will also set up the Azure Firewall to lock down ingoing and outgoing traffic from a Kubernetes cluster in Azure. Creating the Azure Resources Set the subscription Choose the subscription you want to use if you have multiple Azure subscriptions. Replace SubscriptionName with your subscription name. You can also use your subscription ID instead of your subscription name. AKS Resource Group creation Create a Resource Group for the Kubernetes Resources Create a Resource Group for the Hub and VNet Resources AKS Network creation Create a virtual network for the hub. Create a subnet in the hub VNet. Create a subnet for the Jumpbox host in the hub VNet Create a virtual network for the spoke. Create an ingress subnet in the spoke VNet Create an AKS subnet in the spoke VNet VNet Peering Peer the hub and spoke VNet's. Hub to Spoke Spoke to Hub Azure Firewall An open question that Azure users have is how to lock down ingoing to and outgoing traffic from a Kubernetes cluster in Azure. The Azure Firewall is one option that can be set up relatively easily but is not documented in detail. Enable the firewall extension. Configure a public IP for the Azure Firewall Create the Azure Firewall To observe outbound network traffic, you can set up azure monitor and configure a log-analytics workplace. Note- this is optional. Create a User-defined Route Create a user-defined route that will force all traffic from the AKS subnet to the internal IP of the Azure firewall. Check if the route table has been created successfully. Create Exception Rules Create the necessary exception rules for the AKS-required network dependencies to ensure the worker nodes can set their system time and Ubuntu updates (optional) and retrieve the AKS kube-system container images from the Microsoft Container Registry. Note- The exception rules carry a wildcard "*" entry for source and destination, which can be carefully looked into in the respective source(s) from which traffic will be allowed and similarly in the respective destination (s). These firewall rules are examples; you can work around them after checking with your network and system administrators. Now you have created the virtual networks and subnets, the peering between them, and a route from the AKS subnet to the internet going through the Azure firewall that only allows the required Azure Kubernetes Services dependencies. Enable Private Link in the AKS cluster. An AKS cluster, by default, will attach a public IP to the standard load balancer. In this case, we want to prevent this by setting the outboundType to userDefinedRouting in our deployment, configuring a private cluster with a managed identity, and assigning the correct permissions. Create a managed Identity Create an AKS cluster with Azure CNI powered by Cilium in overlay mode. Output truncated: Once the AKS cluster is created, you need to create a link in your DNS zone to the hub VNet to ensure that your jumpbox can correctly resolve the DNS name of your private link-enabled cluster using kubectl from the jumpbox. This will ensure that your jumpbox can resolve the private IP of the API server using Azure DNS. Create a Jumpbox VM Deploy an Ubuntu Jumpbox VM into the JumpboxSubnet so that you can issue all Azure CLI and kubectl commands for the private cluster. Login to the Jumpbox VM and install Azure CLI and kubectl. Make sure you set the correct subscription. Issue kubectl commands to see the status of all the pods and nodes in your private AKS cluster running Azure CNI powered by Cilium in overlay mode. Upgrade to Isovalent Enterprise for Cilium Upgrade your cluster to Isovalent Enterprise for Cilium. For this tutorial, we will be upgrading the cluster using az cli. Note- Refer to this section to obtain the values for plan-name, plan-product & plan-publisher Issue kubectl commands to see the status of all the pods and nodes in your private AKS cluster running Isovalent Enterprise for Cilium in overlay mode. Validation: Isovalent Enterprise for Cilium Cluster status check Check the status of the nodes and make sure they are in a “Ready” state Validate Cilium version Check the version of cilium with cilium version: Cilium Health Check cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking and connectivity. You can check node-to-node health with cilium-health status: Cilium Connectivity Test (Optional) The Cilium connectivity test deploys a series of services and deployments, and CiliumNetworkPolicy will use various connectivity paths to connect. Connectivity paths include with and without service load-balancing and various network policy combinations. Note- This test is optional since it requires access to outbound IPs like 1.1.1.1 and 1.0.0.1 on port 80 and 443. If administrators want to run the test, they must add rules in the Azure Firewall to enable outbound traffic toward the IPs. You can also limit these tests to an internal IP, like the firewall internal IP in this case, which has access to the internet and can override cilium connectivity test with the below option: When you run the cilium connectivity test command, cilium creates deployments and pods. The pods' images are downloaded from specific URLs that must be allowed in the Azure Firewall. Failing to add them in the Azure Firewall leads to the pods being in Error: ImagePullBackOff state. Before adding the rule in Azure Firewall, the state of the pods Describe the pod(s) to see the URL(s) that are being blocked Create an exception rule in Azure Firewall to allow the URL's Once the exception rule is added, we can see that the pods are up and running. You will also observe that outbound traffic to 1.0.0.1 and 1.1.1.1 on port 80 and 443 is blocked. You can create a network rule in Azure Firewall for the DNS-based tests to pass. The cilium connectivity test was run for all of the above scenarios, and the tests were passed successfully. A truncated output for one such test result is added. Output Truncated: Conclusion Hopefully, this post gave you a good overview of deploying a private AKS cluster running Isovalent Enterprise for Cilium in a hub and spoke deployment model with Azure Firewall. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel. Further Reading Isovalent and Azure Kubernetes Service Enabling Enterprise features for Isovalent in Azure Kubernetes Service (AKS) Upgrade to cilium in Azure Azure and Isovalent main partner page