Isovalent Enterprise for Cilium can now be installed on Azure Kubernetes Service clusters using Azure Linux as the host Operating system. In this tutorial you will learn how to:
- Install AKS clusters running Azure CNI powered by Cilium with Azure Linux.
- Migrate your existing clusters on Azure CNI powered by Cilium from Ubuntu to Azure Linux
- Upgrade your clusters from Azure CNI powered by Cilium running Azure Linux to Isovalent Enterprise for Cilium.
What is Isovalent Enterprise for Cilium?
Azure Kubernetes Service (AKS) uses Cilium natively wherein AKS combines the robust control plane of Azure CNI with the data plane of Cilium to provide high-performance networking and security. Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.
How can you deploy Isovalent Enterprise for Cilium?
Isovalent Enterprise for Cilium is available in the Azure Marketplace. It can also be deployed using Azure Resource Manager (ARM) Templates and Azure CLI.
What is Azure Linux?
Microsoft announced the General Availability for Azure Linux Container Host in May 2023. Azure Linux is a lightweight operating system, containing only the packages needed for a cloud environment. Azure Linux can be customized through custom packages and tools, to fit the requirements of your application. Azure Kubernetes Services is one such application that does Production-grade container orchestration as an option for container hosting. The Azure Linux container host for AKS is an open-source Linux distribution created by Microsoft, and it’s available as a container host on Azure Kubernetes Service (AKS).
Why Azure Linux as the host OS?
A popular question you would ask is why choose Azure Linux as the host OS:
- Optimized to run in Azure. Built verified and digitally signed by Microsoft .
- Supply chain security.
- Smaller and leaner Linux to reduce footprint, surface attack area, & optimize performance.
- Operational consistency across Edge to Cloud.
- Rigorous validation and testing of packages and images on AKS infrastructure.
Pre-Requisites
The following prerequisites need to be taken into account before you proceed with this tutorial.
- Azure CLI version 2.48.1 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
- If using ARM templates or the REST API, the AKS API version must be 2022–09–02-preview or later.
- You should have an Azure Subscription.
- Install kubectl.
- Install Cilium CLI.
- Install Helm.
- To install Isovalent Enterprise for Cilium, contact sales@isovalent.com or support@isovalent.com
- Ensure you have enough quota resources to create an AKS cluster. Go to the Subscription blade, navigate to “Usage + Quotas”, and make sure you have enough quota for the following resources:
-Regional vCPUs
-Standard Dv4 Family vCPUs - You can choose regions where the respective quotas are available and not strictly follow the regions picked up during this tutorial.
Limitations with Azure Linux Container Host
- Azure Linux cannot yet be deployed through the Azure Portal.
- Azure Linux doesn’t support AppArmor. Support for SELinux can be manually configured.
- Creating an AKS cluster on Isovalent Enterprise for Cilium with Azure Linux as the host OS will be available in a future release.
Installing Azure Linux on Azure Kubernetes Service Clusters
We will be covering the following combinations of how to install and migrate AKS clusters with Azure Linux.
| Network Plugin | Default Nodepool OS (during AKS cluster creation) | Additional Nodepool OS (after AKS cluster creation) | Migration from Ubuntu to Azure Linux |
| Azure CNI (Powered by Cilium)-Overlay Mode | Azure Linux | Azure Linux | N.A |
| Azure CNI (Powered by Cilium)-Overlay Mode | Ubuntu | Azure Linux | Yes |
| Azure CNI (Powered by Cilium)-Dynamic IP Allocation Mode | Azure Linux | Azure Linux | N.A |
| Azure CNI (Powered by Cilium)-Dynamic IP Allocation Mode | Ubuntu | Azure Linux | Yes |
| Azure CNI (Powered by Cilium)-Overlay Mode to Isovalent Enterprise for Cilium | Azure Linux | N.A | N.A |
| Bring your own CNI (BYOCNI) | Azure Linux | Azure Linux | N.A |
| Bring your own CNI (BYOCNI) | Ubuntu | Azure Linux | Yes |
- N.A= Not Applicable
- BYOCNI (Azure Linux) and BYOCNI (Ubuntu) have also been tested and validated. If you would like to get more information about them; you can get in touch with sales@isovalent.com and support@isovalent.com
Choosing the IMU for a Product?- Installation, Migration or Upgrade
You can take a look at this flowchart and then decide whether you would like to do:
- A greenfield installation of your AKS cluster with Azure Linux
- Upgrade/Migrate your existing AKS clusters from Ubuntu to Azure Linux
Scenario 1: AKS cluster on Azure CNI powered by Cilium in (Overlay mode) with Azure Linux
AKS Resource Group Creation
Create a Resource Group
AKS Cluster creation
Create a cluster with Azure CNI Powered by Cilium with network-plugin as Azure, network-plugin-mode as Overlay, and network-dataplane as Cilium
Set the Subscription
If you have multiple Azure subscriptions, choose the subscription you want to use.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal and browse to Kubernetes Services> select the respective Kubernetes service that was created ( AKS Cluster) and click on connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Add nodepool with OS-type as AzureLinux
Add an Azure Linux node pool to your existing cluster.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System . Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added node and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Scenario 2: AKS cluster on Azure CNI powered by Cilium (Overlay Mode) with Ubuntu (Migration to Azure Linux)
AKS Resource Group Creation
Create a Resource Group
AKS Cluster creation
Create a cluster with Azure CNI Powered by Cilium with network-plugin as Azure, network-plugin-mode as Overlay, and network-dataplane as Cilium.
Set the Subscription
If you have multiple Azure subscriptions, choose the subscription you want to use.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal and browse to Kubernetes Services> select the respective Kubernetes service that was created ( AKS Cluster) and click on connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘Ubuntu’ as the host OS.
Add nodepool with OS-type as AzureLinux
Add an Azure Linux node pool to your existing cluster.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System . Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Migrate the default nodes to Azure Linux
You can migrate the default nodes that are created while creating the AKS cluster and are running Ubuntu as the host OS. This is optional and if not required you can skip this step. Migration is a 3-part process:
Cordon the existing Nodes (Default)
Cordoning marks specified nodes as unschedulable and prevents any more pods from being added to the nodes.
First, obtain the names of the nodes you’d like to cordon with kubectl get nodes:
Next, using kubectl cordon <node-names>, specify the desired nodes in a space-separated list:
Check the status of the nodes that are being cordoned:
Drain the existing nodes (Default)
To successfully drain nodes and evict running pods, ensure that any PodDisruptionBudgets (PDBs) allow for at least 1 pod replica to be moved at a time, otherwise, the drain/evict operation will fail. To check this, you can run kubectl get pdb -A, and make sure ALLOWED DISRUPTIONS is at least 1 or higher.
Draining nodes will cause pods running on them to be evicted and recreated on the other, schedulable nodes.
To drain nodes, use kubectl drain <node-names> --ignore-daemonsets --delete-emptydir-data, again using a space-separated list of node names:
Note- Using --delete-emptydir-data is required to evict the AKS-created coredns and metrics-server pods. If this flag isn’t used, an error is expected.
Remove the existing nodes (Default)
To remove the existing nodes use the az aks delete command. The final result is the AKS cluster having a single Azure Linux node pool with the desired SKU size and all the applications and pods properly running.
Check the status of the nodes to ensure that the default node has been deleted and the additional node running AzureLinux is in a ‘Ready’ state:
Scenario 3: AKS cluster on Azure CNI powered by Cilium (Dynamic IP mode) with Azure Linux
AKS Resource Group Creation
Create a Resource Group
AKS Network creation
Create a virtual network with a subnet for nodes and a subnet for pods and retrieve the subnetID
AKS Cluster creation
Create an AKS cluster referencing the node subnet using –vnet-subnet-id and the pod subnet using –pod-subnet-id. Make sure to use the argument –network-plugin as azure and network-dataplane as cilium.
Set the Subscription
If you have multiple Azure subscriptions, choose the subscription you want to use.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal and browse to Kubernetes Services> select the respective Kubernetes service that was created ( AKS Cluster) and click on connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Add nodepool with OS-type as AzureLinux
Add an Azure Linux node pool to your existing cluster. In the case of Azure CNI (Dynamic IP allocation), you need to add a new subnet for pods and nodes in addition to what was created originally at the time of the AKS cluster creation.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System . Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added node and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Scenario 4: AKS cluster on Azure CNI powered by Cilium (Dynamic IP mode) with Ubuntu (Migration to Azure Linux)
AKS Resource Group Creation
Create a Resource Group
AKS Network creation
Create a virtual network with a subnet for nodes and a subnet for pods and retrieve the subnetID
AKS Cluster creation
Create an AKS cluster referencing the node subnet using –vnet-subnet-id and the pod subnet using –pod-subnet-id. Make sure to use the argument –network-plugin as azure and network-dataplane as cilium.
Set the Subscription
If you have multiple Azure subscriptions, choose the subscription you want to use.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Set the Kubernetes Context
Log in to the Azure portal and browse to Kubernetes Services> select the respective Kubernetes service that was created ( AKS Cluster) and click on connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.
Cluster Status Check
Check the status of the nodes and make sure they are in a ‘Ready’ state and are running ‘Ubuntu’ as the host OS.
Add nodepool with OS-type as AzureLinux
Add an Azure Linux node pool to your existing cluster. In the case of Azure CNI (Dynamic IP allocation), you need to add a new subnet for pods and nodes in addition to what was created originally at the time of the AKS cluster creation.
Note- When adding a new Azure Linux node pool, you need to add at least one as --mode System . Otherwise, AKS will not allow you to delete your existing node pool.
Cluster Status Check
Check the status of the newly added node and make sure they are in a ‘Ready’ state and are running ‘CBL-Mariner/Linux’ as the host OS.
Migrate the default nodes to Azure Linux
You can migrate the default nodes that are created while creating the AKS cluster and are running Ubuntu as the host OS. This is optional and if not required you can skip this step. Migration is a 3-part process:
Cordon the existing Nodes (Default)
Cordoning marks specified nodes as unschedulable and prevents any more pods from being added to the nodes.
First, obtain the names of the nodes you’d like to cordon with kubectl get nodes:
Next, using kubectl cordon <node-names>, specify the desired nodes in a space-separated list:
Check the status of the nodes that are being cordoned:
Drain the existing nodes (Default)
To successfully drain nodes and evict running pods, ensure that any PodDisruptionBudgets (PDBs) allow for at least 1 pod replica to be moved at a time, otherwise, the drain/evict operation will fail. To check this, you can run kubectl get pdb -A, and make sure ALLOWED DISRUPTIONS is at least 1 or higher.
Draining nodes will cause pods running on them to be evicted and recreated on the other, schedulable nodes.
To drain nodes, use kubectl drain <node-names> --ignore-daemonsets --delete-emptydir-data, again using a space-separated list of node names:
Note- Using --delete-emptydir-data is required to evict the AKS-created coredns and metrics-server pods. If this flag isn’t used, an error is expected.
Remove the existing nodes (Default)
To remove the existing nodes use the az aks delete command. The final result is the AKS cluster having a single Azure Linux node pool with the desired SKU size and all the applications and pods properly running.
Check the status of the nodes to ensure that the default node has been deleted and the additional node running AzureLinux is in a ‘Ready’ state:
Scenario 5: AKS cluster on Isovalent Enterprise for Cilium with Azure Linux
Note- You can upgrade your existing clusters as described in Scenarios 1 to 4 to Isovalent Enterprise for Cilium through Azure Marketplace and we have chosen one of those options to highlight the upgrade process. The steps for upgrading all the 4 scenarios are the same.
You can follow this blog and the steps outlined to upgrade an existing AKS cluster to Isovalent Enterprise for Cilium. Make sure you take care of the prerequisites.
- In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
- Type ‘Isovalent’ In the search window and select the offer.
- On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
- Select the resource group in which the cluster exists that we will be upgraded.
- Click Create New Dev Cluster, select ‘No’ and click Next: Cluster Details.
- As ‘No’ was selected, this will result in an upgrade of an already existing cluster in that region
- The name for the AKS cluster will be auto-populated by clicking on the drop-down selection.
- Click ‘Next: Review + Create’ Details.
- Once Final validation is complete, click ‘Create’
- When the application is deployed, the portal will show ‘Your deployment is complete’, along with details of the deployment.
- Verify that the nodes are running Azure Linux. Click > Resource Groups> Kubernetes Services> Select the AKS cluster> Nodepools
How to upgrade Azure Linux Container Host Nodes?
The Azure Linux Container Host ships updates through Updated Azure Linux node images
Note- Make sure you have an AKS cluster either running Azure Linux or migrated to Azure Linux by following the steps outlined in the previous sections.
Manually upgrade your cluster
To manually upgrade the node-image on a cluster:
Validation- Isovalent Enterprise for Cilium
Validate the version of Isovalent Enterprise for Cilium
Check the version of Isovalent Enterprise for Cilium with cilium version:
Cilium Health Check
cilium-health is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. You can check node-to-node health with cilium-health status:
Cilium Connectivity Test
The Cilium connectivity test deploys a series of services, deployments, and CiliumNetworkPolicy which will use various connectivity paths to connect. Connectivity paths include with and without service load-balancing and various network policy combinations.
Cilium connectivity test was run for all of the above scenarios and the tests passed successfully. Adding a truncated output for one such test result.
Caveats/ Troubleshooting
- If you are adding a nodepool with network plugins Azure CNI Dynamic IP or Azure CNI powered by Cilium and a different/new subnet for both pods and nodes has not been added you will observe this error.
- If you are deleting a nodepool in any of the above scenarios that have been explained, ensure that there is one nodepool that was created with
--mode Systemelse you will observe this error.
Conclusion
Hopefully, this post gave you a good overview of how to install or migrate your existing or new AKS clusters running Azure CNI powered by Cilium with Azure Linux; and also upgrade to Isovalent Enterprise for Cilium. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel.
Try it out
- Azure CNI powered by Cilium.
- Isovalent Enterprise for Cilium on the Azure marketplace.
Further Reading
- Tutorial on installing Isovalent Enterprise for Cilium in Azure
- Enabling Enterprise features on Isovalent Enterprise from Cilium in Azure
- Tutorial on installing an AKS cluster running Azure CNI powered by Cilium
- Upgrade to cilium in Azure
- Azure and Isovalent main partner page
Amit Gupta is a Senior Technical Marketing Engineer at Isovalent that is powering eBPF cloud-native networking and security. Amit has 20+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source and has worked in the past with companies like Motorola, Juniper, Avi Networks (acquired by VMware), and Prosimo. He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area in his time outside of work. Amit is an avid runner and cyclist and also spends a considerable amount of time helping kids in orphanages.
