Cilium in Azure Kubernetes Service (AKS)

A few weeks ago, we announced that Isovalent Enterprise for Cilium is now available to Azure Kubernetes Service (AKS) customers as a one-click upgrade in the Microsoft Azure Marketplace and provide the full set of advanced Cilium features available to Azure customers. Today, in this tutorial, you will learn how to deploy Isovalent Enterprise for Cilium on your AKS cluster using Azure Marketplace, Azure CLI, and Azure Resource Manager (ARM) templates. The tutorial will walk through the installation of a new cluster and how to upgrade an existing AKS cluster running Azure CNI powered by Cilium to Isovalent Enterprise for Cilium.

What is Isovalent Enterprise for Cilium?

Isovalent Enterprise for Cilium is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.

Why Isovalent Enterprise for Cilium?

For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits:

• Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security.
• Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
• Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments.
• Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
• Service Mesh: Isovalent Cilium Enterprise provides seamless service-to-service communication that’s sidecar-free and advanced load balancing, making it easy to deploy and manage complex microservices architectures.
• Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.

This offering brings complete flexibility regarding access to Cilium features while retaining the advantageous ease of use and seamless integration with Azure. There are three ways to deploy Isovalent Enterprise for Cilium:

• Azure Marketplace
• Azure ARM Template
• Azure CLI

Why Azure Marketplace ?

Azure Marketplace is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. In Azure Marketplace, you can find, try, buy, and deploy the software and services needed to build new solutions and manage your cloud infrastructure. The catalog includes solutions for different industries and technical areas, free trials, and consulting services from Microsoft partners. Included among these solutions are Kubernetes application-based container offers. These offers contain applications that are meant to run on Kubernetes clusters such as Azure Kubernetes Service (AKS).

Pre-Requisites

The following prerequisites need to be taken into account before you proceed with this tutorial:

• An Azure account with an active subscription- Create an account for free
• AKS Cluster is up and running with Azure CNI powered by Cilium using either of the two options:
  • The cluster can be created by Assigning IP addresses from an overlay network
  • The cluster can be created by Assigning IP addresses from a virtual network
• Azure CLI version 2.41.0 or later. Run az --version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
• If using ARM templates or the REST API, the AKS API version must be 2022-09-02-preview or later.
• The kubectl command line tool is installed on your device. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.26, you can use kubectl version 1.25, 1.26, or 1.27 with it. To install or upgrade kubectl, see Installing or updating kubectl.
• Isovalent Enterprise for Cilium as a marketplace extension is currently supported only in the following regions.
• Isovalent Enterprise for Cilium cannot be installed on AKS clusters created with Service Principals.
  • It would be best if you upgraded your AKS clusters to managed identities.
• You must ensure that the following network and FQDN/application rules are required for AKS cluster extensions to work seamlessly.
• In the case of a private AKS cluster, configure rules to access an Azure container registry behind a firewall. The endpoints could be:
  • REST endpoints.
  • Dedicated data endpoints.
    • Example: arcmktplaceprod.centralindia.data.azurecr.io
  • Registry FQDN’s.
• If you are upgrading your AKS cluster from Azure CNI powered by Cilium to Isovalent Enterprise for Cilium and the cluster has a lot of configuration, make sure you get in touch with Isovalent support as they can assist you by setting “AgentOperationTimeoutInMinutes” to reasonable value for the upgrade to go through.
• New namespace being created:
  • When an AKS cluster is either being created or upgraded to Isovalent Enterprise for Cilium, it results in the creation of an additional namespace azure-extensions-usage-system. Make sure that your Azure Policy whitelists this namespace.
  • When hubble is installed, it creates pods in the kube-system namespace by default. If you have an SIEM like Microsoft Defender, you can see an error similar to “New container ‘hubble-relay’ in the kube-system namespace detected.”
  • It is an ideal installation process for Cilium and Hubble components. With the default alerting rules, these errors are expected. Cilium and Hubble’s components use best practices to run within the kube-system namespace within the cluster.
  • Adding an exception for such events on Microsoft Defender’s alerting rules can help mitigate the errors. For such modifications, you can get in touch with Microsoft Support. This is true for any alternative External Attack Surface Management tools.

Register resource providers

Before you deploy a container offer, you must register with Microsoft.ContainerService and Microsoft.KubernetesConfiguration providers on your subscription by using the az provider register command:

az provider register --namespace Microsoft.ContainerService --wait
az provider register --namespace Microsoft.KubernetesConfiguration --wait

Create a new AKS cluster with Isovalent Enterprise for Cilium

• In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
• You can search for an offer or publisher directly by name or browse all offers. To find Kubernetes application offers, on the left side, under Categories, select Containers.

• Type “Isovalent” in the search window and select the offer.

• On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
• Select the respective subscription in which the new AKS cluster needs to be created.
• Select the resource group to deploy the cluster in. If a resource group doesn’t exist, click “Create New”.
• Click Create New Dev Cluster, select “Yes,” and click Next: Cluster Details.

• Provide a name for the AKS cluster and click “Next: Review + Create”

• Once Final validation is complete, click “Create”

• When the application is deployed, the portal will show “Your deployment is complete,” along with deployment details.

• Check the extensions installed on the cluster from Azure Portal. On the AKS cluster, navigate to the “Extensions + applications” menu to verify the same.

• Verify the deployment by using the following command to list the extensions that are running on your cluster:

az k8s-extension show --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

• Log in to the Azure portal, browse Kubernetes Services, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the individual Kubernetes context.

az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
az aks get-credentials --resource-group <resourcegroup-name> --name <clustername>

• Validate the version of Isovalent Enterprise for Cilium on your newly created cluster

#kubectl --namespace=kube-system exec -i -t ds/cilium  -- cilium version

Client: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64
Daemon: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64

Note– cee-indicates Cilium Enterprise Edition, the Enterprise Version of Cilium offered by Isovalent installed on the cluster.

Upgrade an existing cluster

Note-

• You can upgrade an existing AKS cluster running Azure CNI powered by Cilium in the same region from where the upgrade is being attempted. The cluster being upgraded can be running Azure CNI powered by Cilium using either of the two options:
  • The cluster can be created by Assigning IP addresses from an overlay network
  • The cluster can be created by Assigning IP addresses from a virtual network
• The version on your cluster can be verified by:

kubectl --namespace=kube-system exec -i -t ds/cilium  -- cilium version

Client: 1.12.5 701acde56b 2022-12-15T16:03:30-08:00 go version go1.18.9 linux/amd64
Daemon: 1.12.5 701acde56b 2022-12-15T16:03:30-08:00 go version go1.18.9 linux/amd64

Preparing for the upgrade

This section will guide you through the steps required to upgrade an existing AKS cluster running Azure CNI powered by Cilium to Isovalent Enterprise for Cilium.

• In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
• You can search for an offer or publisher directly by name or browse all offers. To find Kubernetes application offers, on the left side, under Categories, select Containers.

• Type “Isovalent” In the search window and select the offer.

• On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
• Select the resource group in which the cluster exists that we will be upgraded.
• Click Create New Dev Cluster, select “No,” and click Next: Cluster Details.

• As “No” was selected, this will result in an upgrade of an already existing cluster in that region
• The name for the AKS cluster will be auto-populated by clicking on the drop-down selection.
• Click “Next: Review + Create” Details.

• Once Final validation is complete, click “Create”

• When the application is deployed, the portal will show “Your deployment is complete,” along with deployment details.

• Check the extensions installed on the cluster from Azure Portal. On the AKS cluster, navigate to the “Extensions + applications” menu to verify the same.

• Verify the deployment by using the following command to list the extensions that are running on your cluster:

az k8s-extension show --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

• Log in to the Azure portal, browse to Kubernetes Services, select the respective Kubernetes service created ( AKS Cluster), and click connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.

az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
az aks get-credentials --resource-group <resourcegroup-name> --name <clustername>

• Validate the version of Isovalent Enterprise for Cilium on your newly upgraded cluster

kubectl --namespace=kube-system exec -i -t ds/cilium  -- cilium version

Client: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64
Daemon: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64

Note– cee-indicates Cilium Enterprise Edition, the Enterprise Version of Cilium offered by Isovalent to which the cluster was upgraded.

Manage the offer lifecycle

For lifecycle management, an Azure Kubernetes offer is represented as a cluster extension for AKS. For more information, see Cluster extensions for AKS.

Purchasing an offer from Azure Marketplace creates a new extension instance on your AKS cluster. View the extension instance from the cluster by using the following command:

az k8s-extension show --name <extension-name> --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters

Note– extension-name, in this case, is cilium

Remove an offer

You can delete a purchased plan for an Azure container offer by deleting the extension instance on the cluster. For example:

az k8s-extension delete --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

Installation via Azure Resource Manager (ARM) Template

Many teams have adopted agile development methods since moving to the cloud. These teams iterate quickly. They need to repeatedly deploy their solutions to the cloud and know their infrastructure is reliable.

You can use Azure Resource Manager (ARM) templates to implement infrastructure as code for your Azure solutions. The template is a JavaScript Object Notation (JSON) file that defines the infrastructure and configuration for your project. The template uses declarative syntax, which lets you state what you intend to deploy without writing the sequence of programming commands to create it. In the template, you specify the resources to deploy and the properties for those resources.

To deploy a Kubernetes application programmatically through ARM templates, a user needs to select the Kubernetes application and settings, generate an ARM template, accept legal terms and conditions, and finally deploy the ARM template.

Pre-Requisites

The following prerequisites need to be taken into account before you proceed with this tutorial:

• An Azure account with an active subscription- Create an account for free
• AKS Cluster is up and running with Azure CNI powered by Cilium using either of the two options:
  • The cluster can be created by Assigning IP addresses from an overlay network
  • The cluster can be created by Assigning IP addresses from a virtual network
• Azure CLI version 2.41.0 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
• Templates are JavaScript Object Notation (JSON) files. To create templates, you need a good JSON editor. We recommend Visual Studio Code with the Azure Resource Manager Tools extension. If you need to install these tools, see Quickstart: Create ARM templates with Visual Studio Code.
• The kubectl command line tool is installed on your device. The version can be the same as or up to one minor version earlier or later than the Kubernetes version of your cluster. For example, if your cluster version is 1.26, you can use kubectl version 1.25, 1.26, or 1.27 with it. To install or upgrade kubectl, see Installing or updating kubectl.
• You must ensure that the following network and FQDN/application rules are required for AKS cluster extensions to work seamlessly.
• Isovalent Enterprise for Cilium cannot be installed on AKS clusters created with Service Principals.
  • It would be best if you upgraded your AKS clusters to managed identities.
• In the case of a private AKS cluster, configure rules to access an Azure container registry behind a firewall. The endpoints could be:
  • REST endpoints.
  • Dedicated data endpoints.
    • Example: arcmktplaceprod.centralindia.data.azurecr.io
  • Registry FQDN’s.
• If you are upgrading your AKS cluster from Azure CNI powered by Cilium to Isovalent Enterprise for Cilium and the cluster has a lot of configuration, make sure you get in touch with Isovalent support as they can assist you by setting “AgentOperationTimeoutInMinutes” to reasonable value for the upgrade to go through.
• New namespace being created:
  • When an AKS cluster is either being created or upgraded to Isovalent Enterprise for Cilium, it results in the creation of an additional namespace azure-extensions-usage-system. Make sure that your Azure Policy whitelists this namespace.
  • When hubble is installed, it creates pods in the kube-system namespace by default. If you have an SIEM like Microsoft Defender, you can see an error similar to “New container ‘hubble-relay’ in the kube-system namespace detected.”
  • It is an ideal installation process for Cilium and Hubble components. With the default alerting rules, these errors are expected. Cilium and Hubble’s components use best practices to run within the kube-system namespace within the cluster.
  • Adding an exception for such events on Microsoft Defender’s alerting rules can help mitigate the errors. For such modifications, you can get in touch with Microsoft Support. This is true for any alternative External Attack Surface Management tools.

Register resource providers

Before you deploy a container offer, you must register the Microsoft.ContainerService and Microsoft.KubernetesConfiguration providers on your subscription by using the az provider register command:

az provider register --namespace Microsoft.ContainerService --wait
az provider register --namespace Microsoft.KubernetesConfiguration --wait

Select Kubernetes Application

First, you must select the Kubernetes application you want to deploy/upgrade in the Azure portal.

• In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
• You can search for an application or publisher directly by name or browse all applications. To find Kubernetes application applications, on the left side, under Categories, select Containers.

Generate ARM Template

• Type “Isovalent” in the search window and select the application.

• On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
• Select the respective subscription in which the existing AKS cluster has already been created with Azure CNI powered by Cilium.
• Select the resource group to deploy the cluster in.
• Click Create New Dev Cluster, select “No,” and click Next: Cluster Details.

Note- In case you would like to create a new AKS cluster, you need to click on the resource group and then select “Create New” under Resource Group. Also, if you would like to create a new cluster with Isovalent Enterprise for Cilium, you need to select the option “Yes” at the “Create new dev cluster” step.

• The AKS cluster name will be available through the drop-down and click on “Next: Review + Create”

• Once Final validation is complete, click > “View Automation Template” > click “Download”

• If all the validations have passed, you’ll see the ARM template in the editor.

• Download the ARM template and save it to a location on your computer.
• You need to note the following values, which will be used later on in this tutorial
  • Plan-publisher
  • Plan-offerID and
  • Plan-name
• You need to exit out of the Azure portal screen and not Click on Create in the UI workflow as ARM templates will now take over the part of installing the “Cilium” extension and enabling the Enterprise features.

Editing the json file

• Unzip the ARM template file, which will include a parameters.json file, and edit the following fields:
  • ClusterResourceName– The description should be the name of the AKS cluster that is either being created or an existing AKS cluster running on Azure CNI powered by Cilium.
  • createNewCluster- The default value is false, meaning an existing AKS cluster is running on Azure CNI powered by Cilium, which will be upgraded to Isovalent Enterprise for Cilium. The user can turn it to True if they are creating a new cluster on Isovalent Enterprise for Cilium.
  • extensionResourceName– The description should be the cilium extension and not be edited.
  • Location- A description of the location of the AKS resource should be the location where the AKS resource was created/ upgraded.

Accept terms and agreements

Before deploying a Kubernetes application, you need to accept its terms and agreements. To do so, use Azure CLI or Azure PowerShell. For this tutorial, we will be using Azure CLI to deploy the ARM template.

Be sure to use the values you copied before downloading the ARM template for plan-publisher, plan-offerID, and plan-name in your command.

az vm image terms accept --offer <Product ID> --plan <Plan ID> --publisher <Publisher ID>

Values.yaml to enable Enterprise Features

• To enable Enterprise features, you can create a sample yaml file as shown in the example below:

namespace: kube-system
hubble.enabled: true
hubble.relay.enabled: true
encryption.enabled: true
encryption.type: wireguard
kubeProxyReplacement: strict
k8sServicePort: <API_SERVER_PORT>
k8sServiceHost: <API_SERVER_IP>
l7Proxy : false

Setting Enterprise Values for the ARM template

• You can download yq and inject custom helm values into the ARM template, as Azure ARM templates follow a key:value pair syntax notation.
  • Users can use a tool of their choice as well to inject helm values. 

yq -o=json '.resources[1].properties.configurationSettings |= load("./values.yaml")' template.json

• The output of this will be a json file which will be used to deploy the ARM template. Sample json file would look like as shown below which you have to copy.

Note– The output below is truncated.

...
      "name": "[parameters('extensionResourceName')]",
      "plan": {
        "name": "[variables('plan-name')]",
        "product": "[variables('plan-offerID')]",
        "publisher": "[variables('plan-publisher')]"
      },
      "properties": {
        "autoUpgradeMinorVersion": true,
        "configurationProtectedSettings": {},
        "configurationSettings": {
          "namespace": "kube-system",
          "hubble.enabled": true,
          "hubble.relay.enabled": true,
          "encryption.enabled"  : true,
          "encryption.type"     : "wireguard",
          "l7Proxy"             : false,
		"kubeProxyReplacement": "strict",
		"k8sServiceHost"	  : "API_SERVER_IP",
		"k8sServicePort"	  : "API_SERVER_PORT"
        },
        "extensionType": "[variables('clusterExtensionTypeName')]",
        "releaseTrain": "[variables('releaseTrain')]"
      },
      "scope": "[concat('Microsoft.ContainerService/managedClusters/', parameters('clusterResourceName'))]",
      "type": "Microsoft.KubernetesConfiguration/extensions"
    }
  ],
  "variables": {
    "clusterExtensionTypeName": "Isovalent.CiliumEnterprise.One",
    "plan-name": "isovalent-cilium-enterprise-base-edition",
    "plan-offerID": "isovalent-cilium-enterprise",
    "plan-publisher": "isovalentinc3222233121323",
    "releaseTrain": "stable"
  }
}

Deploy ARM Template

• To start working with ARM templates, sign in with your Azure credentials in Azure CLI.

az login

• Choose the subscription you want to use if you have multiple Azure subscriptions.
  • Replace SubscriptionName with your subscription name.
  • You can also use your subscription ID instead of your subscription name.

az account set --subscription SubscriptionName

• To deploy the template, use either Azure CLI or Azure PowerShell. Use the resource group you created while creating the cluster on Azure CNI powered by Cilium.
  • Template-File type is “json”.

templateFile="{provide-the-path-to-the-template-file}"
az deployment group create \
  --name <deployment-name> \
  --resource-group <resource-group name> \
  --template-file $templateFile

• The deployment command returns results. Look for ProvisioningState to see whether the deployment succeeded.

Note– The output below is truncated.

...

"provisioningState": "Succeeded",
    "templateHash": "93xxxxxxxxxxxx9951",
    "templateLink": null,
    "timestamp": "2023-06-15T17:46:08.011055+00:00",
    "validatedResources": null

Verify Deployment

• You can verify the deployment by exploring the resource group from the Azure portal.
• Sign in to the Azure portal.
• From the left menu, select Resource Groups.
• Check the box to the left of the resource group in which the AKS cluster resides.
• Select the resource group you used in the earlier procedure.

• Check the extensions installed on the cluster from Azure Portal. Go to the “Extensions + applications” menu on the AKS cluster to verify the settings.

• Optional– Log in to the Azure portal and browse to Kubernetes Services> select the respective Kubernetes service that was created ( AKS Cluster) and click on connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.

az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx

az aks get-credentials --resource-group <resourcegroup-name> --name <clustername>

• Verify the deployment by using the following command to list the extension and features that were enabled on your cluster after the successful deployment of the ARM template:

az k8s-extension show --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

{
  "aksAssignedIdentity": {
    "principalId": "9b9b5044-57c6-49c8-923a-0953b0cadfd8",
    "tenantId": null,
    "type": null
  },
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {
    "encryption.enabled": "true",
    "encryption.type": "wireguard",
    "hubble.enabled": "true",
    "hubble.relay.enabled": "true",
    "k8sServiceHost": "testciliumextension-7aqazoma.hcp.centralindia.azmk8s.io",
    "k8sServicePort": "443",
    "kubeProxyReplacement": "strict",
    "l7Proxy": "false",
    "namespace": "kube-system"
  },
  "currentVersion": "1.0.0",
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "Isovalent.CiliumEnterprise.One",
  "id": "/subscriptions/###################################/resourceGroups/testciliumextension/providers/Microsoft.ContainerService/managedClusters/testciliumextension/providers/Microsoft.KubernetesConfiguration/extensions/cilium",
  "identity": null,
  "isSystemExtension": false,
  "name": "cilium",
  "packageUri": null,
  "plan": {
    "name": "isovalent-cilium-enterprise-base-edition",
    "product": "isovalent-cilium-enterprise",
    "promotionCode": null,
    "publisher": "isovalentinc3222233121323",
    "version": null
  },
  "provisioningState": "Succeeded",
  "releaseTrain": "stable",
  "resourceGroup": "testciliumextension",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2023-07-12T17:32:56.255587+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2023-07-12T17:42:43.573078+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": null
}

• Validating the version of Isovalent Enterprise for Cilium on your newly created cluster

Client: 1.12.11-cee.1 88bf67bd 2023-06-15T03:12:05+00:00 go version go1.18.10 linux/amd64
Daemon: 1.12.11-cee.1 88bf67bd 2023-06-15T03:12:05+00:00 go version go1.18.10 linux/amd64

Installation via Azure CLI

A user can create extension instances in an AKS cluster, setting required and optional parameters including options related to updates and configurations. You can also view, list, update, and delete extension instances.

Pre-Requisites

The following prerequisites need to be taken into account before you proceed with this tutorial:

• An Azure subscription.
• While creating the extension- Users need to make sure that they have an AKS Cluster up and running with Azure CNI powered by Cilium
  • The cluster can be created by Assigning IP addresses from an overlay network
  • The cluster can be created by Assigning IP addresses from a virtual network
• While updating the extension- Users need to make sure that they have an AKS Cluster up and running with Isovalent Enterprise for Cilium from the Azure marketplace.
• Isovalent Enterprise for Cilium as a marketplace extension is currently supported only in the following regions.
• You need to ensure that the following network and FQDN/application rules are required for AKS cluster extensions to work seamlessly.
• Isovalent Enterprise for Cilium cannot be installed on AKS clusters created with Service Principals.
  • You need to upgrade your AKS clusters to managed identities.
• In the case of a private AKS cluster, configure rules to access an Azure container registry behind a firewall. The endpoints could be:
  • REST endpoints.
  • Dedicated data endpoints.
    • Example: arcmktplaceprod.centralindia.data.azurecr.io
  • Registry FQDN’s.
• If you are upgrading your AKS cluster from Azure CNI powered by Cilium to Isovalent Enterprise for Cilium and the cluster has a lot of configuration, make sure you get in touch with Isovalent support as they can assist you by setting “AgentOperationTimeoutInMinutes” to reasonable value for the upgrade to go through.
• New namespace being created:
  • When an AKS cluster is either being created or upgraded to Isovalent Enterprise for Cilium, it results in the creation of an additional namespace azure-extensions-usage-system. Make sure that your Azure Policy whitelists this namespace.
  • When hubble is installed, it creates pods in the kube-system namespace by default. If you have a SIEM like Microsoft Defender, you can see an error similar to “New container ‘hubble-relay’ in the kube-system namespace detected.”
  • It is an ideal installation process for Cilium and Hubble components. With the default alerting rules, these errors are expected. Cilium and Hubble’s components use best practices to run within the kube-system namespace within the cluster.
  • Adding an exception for such events on Microsoft Defender’s alerting rules can help mitigate the errors. For such modifications, you can get in touch with Microsoft Support. This is true for any alternative External Attack Surface Management tools.
• The Microsoft.ContainerService and Microsoft.KubernetesConfiguration Resource providers must be registered with your subscription. 
• To register these providers, run the following command:

az provider register --namespace Microsoft.ContainerService --wait 

az provider register --namespace Microsoft.KubernetesConfiguration --wait

• The latest version of the k8s-extension Azure CLI extensions. Install the extension by running the following command:

az extension add --name k8s-extension

• If the extension is already installed, make sure you’re running the latest version by using the following command:

az extension update --name k8s-extension

Select the Kubernetes application

You must select the Kubernetes application they want to deploy in the Azure portal. You also need to copy some of the details for later use.

• In the Azure portal, go to the Marketplace page.
• Select the Kubernetes application.
• Select the required plan.
• Select the Create button.
• Fill out all the application (extension) details.
• In the Review + Create tab, select Download a template for automation. You’ll see the ARM template in the editor if all the validations are passed.
• Examine the ARM template:
  • In the variables section, copy the plan-name, plan-publisher, plan-offerID, and clusterExtensionTypeName values for later use.
  • In the resource Microsoft.KubernetesConfiguration/extensions‘ section, copy the configurationSettings section for later use.

Accept terms and agreements

• Before deploying a Kubernetes application, you must accept its terms and agreements. To do so, use Azure CLI or Azure PowerShell. In this section, we will use Azure CLI to deploy the ARM template.
• Be sure to use the values you copied before downloading the ARM template for plan-publisher, plan-offerID, and plan-name in your command.

az vm image terms accept --offer <Product ID> --plan <Plan ID> --publisher <Publisher ID>

Setting the correct Context

Log in to the Azure portal and browse to Kubernetes Services> select the respective Kubernetes service created ( AKS Cluster) and click on connect. This will help you connect to your AKS cluster and set the respective Kubernetes context.

az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx

az aks get-credentials --resource-group <resourcegroup-name> --name <clustername>

Creating the extension

You can create a new extension instance with k8s-extension create passing in values for the mandatory parameters.

az k8s-extension create --name cilium --extension-type Isovalent.CiliumEnterprise.One --scope cluster --cluster-name ciliumossazmktplace  --resource-group ciliumossazmktplace --cluster-type managedClusters --plan-name isovalent-cilium-enterprise-base-edition --plan-product isovalent-cilium-enterprise --plan-publisher isovalentinc3222233121323

Update the extension

You can update an existing extension (Cilium) installed on an AKS cluster with Isovalent Enterprise for Cilium.

• Listing an existing extension

az k8s-extension show -c <cluster-name> -t managedClusters -g <resource-group> -n cilium

Enabling Enterprise Features

• You will be prompted for a prompt while enabling these features, select Yes.

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings namespace=kube-system 

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings hubble.enabled=true 

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings hubble.relay.enabled=true

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings encryption.enabled=true

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings encryption.type=wireguard

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings l7Proxy=false

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings kubeProxyReplacement=strict

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings k8sServicePort=<API_SERVER_PORT>

az k8s-extension update -c <cluster-name> -t managedClusters -g <resource-group> -n cilium --configuration-settings k8sServiceHost=<API_SERVER_IP>

Verifying the extension

Verify the k8s-extension by using the following command to list the extensions and features that were enabled on your cluster using Azure CLI.

az k8s-extension show -c <cluster-name> -t managedClusters -g <resource-group> -n cilium

{
  "aksAssignedIdentity": {
    "principalId": "9b9b5044-57c6-49c8-923a-0953b0cadfd8",
    "tenantId": null,
    "type": null
  },
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {
    "encryption.enabled": "true",
    "encryption.type": "wireguard",
    "hubble.enabled": "true",
    "hubble.relay.enabled": "true",
    "k8sServiceHost": "testciliumextension-7aqazoma.hcp.centralindia.azmk8s.io",
    "k8sServicePort": "443",
    "kubeProxyReplacement": "strict",
    "l7Proxy": "false",
    "namespace": "kube-system"
  },
  "currentVersion": "1.0.0",
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "Isovalent.CiliumEnterprise.One",
  "id": "/subscriptions/###################################/resourceGroups/testciliumextension/providers/Microsoft.ContainerService/managedClusters/testciliumextension/providers/Microsoft.KubernetesConfiguration/extensions/cilium",
  "identity": null,
  "isSystemExtension": false,
  "name": "cilium",
  "packageUri": null,
  "plan": {
    "name": "isovalent-cilium-enterprise-base-edition",
    "product": "isovalent-cilium-enterprise",
    "promotionCode": null,
    "publisher": "isovalentinc3222233121323",
    "version": null
  },
  "provisioningState": "Succeeded",
  "releaseTrain": "stable",
  "resourceGroup": "testciliumextension",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2023-07-12T17:32:56.255587+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2023-07-12T17:42:43.573078+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": null
}

Validation-Isovalent Enterprise for Cilium

Cluster and Cilium Health Check

• Check the nodes’ status and ensure they are in a “Ready” state

kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-87984109-vmss000000   Ready    agent   45m   v1.25.6
aks-nodepool1-87984109-vmss000001   Ready    agent   45m   v1.25.6

• cilium-health , which is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking connectivity. Use cilium-health to get visibility into the overall health of the cluster’s networking connectivity.

cilium-health status
Probe time:   2023-05-25T09:31:52Z
Nodes:
  aks-nodepool1-87984109-vmss000000 (localhost):
    Host connectivity to 10.10.0.5:
      ICMP to stack:   OK, RTT=124.002µs
      HTTP to agent:   OK, RTT=470.504µs
  aks-nodepool1-87984109-vmss000001:
    Host connectivity to 10.10.0.4:
      ICMP to stack:   OK, RTT=2.274623ms
      HTTP to agent:   OK, RTT=3.199333ms

Cilium Connectivity Test

• Run cilium connectivity test (an automated test that checks that Cilium has been deployed correctly and tests intra-node connectivity, inter-node connectivity, and network policies) to verify that everything is working as expected.

cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [ciliumossazmktplace] Creating namespace cilium-test for connectivity check...
✨ [ciliumossazmktplace] Deploying echo-same-node service...
✨ [ciliumossazmktplace] Deploying DNS test server configmap...
✨ [ciliumossazmktplace] Deploying same-node deployment...
✨ [ciliumossazmktplace] Deploying client deployment...
✨ [ciliumossazmktplace] Deploying client2 deployment...
✨ [ciliumossazmktplace] Deploying echo-other-node service...
✨ [ciliumossazmktplace] Deploying other-node deployment...
⌛ [ciliumossazmktplace] Waiting for deployments [client client2 echo-same-node] to become ready...
⌛ [ciliumossazmktplace] Waiting for deployments [echo-other-node] to become ready...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/client-7b78db77d5-2tgfk to appear...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/client2-78f748dd67-tnzcw to appear...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client-7b78db77d5-2tgfk to reach DNS server on cilium-test/echo-same-node-85bc9b6b56-hrvbg pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client2-78f748dd67-tnzcw to reach DNS server on cilium-test/echo-same-node-85bc9b6b56-hrvbg pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client-7b78db77d5-2tgfk to reach DNS server on cilium-test/echo-other-node-cd69fcf6b-x8hpl pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client2-78f748dd67-tnzcw to reach DNS server on cilium-test/echo-other-node-cd69fcf6b-x8hpl pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client-7b78db77d5-2tgfk to reach default/kubernetes service...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client2-78f748dd67-tnzcw to reach default/kubernetes service...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/echo-other-node-cd69fcf6b-x8hpl to appear...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-85bc9b6b56-hrvbg to appear...
⌛ [ciliumossazmktplace] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [ciliumossazmktplace] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.5:32244 (cilium-test/echo-other-node) to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.5:31540 (cilium-test/echo-same-node) to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.4:32244 (cilium-test/echo-other-node) to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.4:31540 (cilium-test/echo-same-node) to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.12.7
🏃 Running tests...
[=] Test [no-policies]
............................
[=] Test [no-policies-extra]
........
[=] Test [allow-all-except-world]
..............
[=] Test [client-ingress]
..
[=] Test [all-ingress-deny]
........
[=] Test [all-egress-deny]
................
[=] Test [all-entities-deny]
........
[=] Test [cluster-entity]
..
[=] Test [host-entity]
....
[=] Test [echo-ingress]
....
[=] Test [client-ingress-icmp]
..
[=] Test [client-egress]
....
[=] Test [client-egress-expression]
....
[=] Test [client-egress-to-echo-service-account]
....
[=] Test [to-entities-world]
......
[=] Test [to-cidr-1111]
....
[=] Test [echo-ingress-from-other-client-deny]
......
[=] Test [client-ingress-from-other-client-icmp-deny]
......
[=] Test [client-egress-to-echo-deny]
......
[=] Test [client-ingress-to-echo-named-port-deny]
....
[=] Test [client-egress-to-echo-expression-deny]
....
[=] Test [client-egress-to-echo-service-account-deny]
....
[=] Test [client-egress-to-cidr-deny]
....
[=] Test [client-egress-to-cidr-deny-default]
....

[=] Skipping Test [health]
[=] Test [echo-ingress-l7]
............
[=] Test [echo-ingress-l7-named-port]
............
[=] Test [client-egress-l7-method]
............
[=] Test [client-egress-l7]
..........
[=] Test [client-egress-l7-named-port]
..........

[=] Skipping Test [client-egress-l7-tls-deny-without-headers]

[=] Skipping Test [client-egress-l7-tls-headers]
[=] Test [dns-only]
..........
[=] Test [to-fqdns]
........

✅ All 31 tests (230 actions) successful, 3 tests skipped, 0 scenarios skipped.

Troubleshooting

You can always refer to this article on how to troubleshoot errors that occur when you deploy cluster extensions for Azure Kubernetes Service (AKS).

Conclusion

Hopefully, this post gave you a good overview of deploying Isovalent Enterprise for Cilium in your AKS clusters. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel.

Try it out

• Isovalent Enterprise for Cilium on the Azure marketplace.

Further Reading

• Tutorial on installing an AKS cluster running Azure CNI powered by Cilium
• Azure and Isovalent main partner page