Tutorial: Deploying Isovalent Enterprise for Cilium from Azure Marketplace

With Azure CNI Powered by Cilium, AKS is now natively powered by Cilium. Azure CNI Powered by Cilium combines the robust control plane of Azure CNI with the dataplane of Cilium to provide high-performance networking and security.

AKS customers will also benefit from a seamless one-click upgrade experience from Azure CNI Powered by Cilium to the full Isovalent Enterprise for Cilium platform. The enterprise platform will be available in the Azure Container Marketplace and makes the full set of advanced Cilium features available to Azure customers. This includes security and governance controls, extended network capabilities, Timescape, the full set of Isovalent Tetragon Enterprise features, and more!

The tight integration into the Azure platform simplifies operations by enabling auto-upgrades and natively integrating into the Azure ecosystem for SIEM export, monitoring, and governance control. The unified billing experience will eliminate management overhead. Finally, the support collaboration will maximize the reliability and customer experience of the platform.

Azure Marketplace is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. In Azure Marketplace, you can find, try, buy, and deploy the software and services that you need to build new solutions and manage your cloud infrastructure. The catalog includes solutions for different industries and technical areas, free trials, and consulting services from Microsoft partners.

Included among these solutions are Kubernetes application-based container offers. These offers contain applications that are meant to run on Kubernetes clusters such as Azure Kubernetes Service (AKS).

In this tutorial, you’ll learn how to deploy Isovalent Enterprise for Cilium on your AKS cluster from Azure Marketplace on a new cluster and also upgrade an existing cluster from an AKS cluster running Azure CNI powered by Cilium to Isovalent Enterprise for Cilium.

What is Isovalent Enterprise for Cilium

Isovalent Enterprise for Cilium is an enterprise-grade, hardened distribution of the open-source Cilium project. It provides advanced networking, security, and observability capabilities that enable organizations to achieve compliance, simplify multi-cloud connectivity, implement zero-trust security principles, and gain security observability in Kubernetes and other cloud-native infrastructure.

Cilium can run natively in any Kubernetes environment, operate as a virtual appliance as a transit gateway, or run as an agent on virtual machines and servers. Connectivity is provided at the networking (L3-L4) and service mesh level (L7). Across all Cilium layers, Cilium provides extensive observability functionality with integrations for Prometheus, Splunk, Grafana, Elastic, fluent, and OpenTelemetry.

Why Isovalent Enterprise for Cilium

For enterprise customers requiring support and/or usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium Base Edition” is recommended.

This offering brings complete flexibility in terms of access to Cilium features while retaining the advantageous ease of use and integration with Azure seamlessly.

Limitations

This feature is currently supported only in the following regions:

East US, EastUS2EUAP, West US, Central US, West Central US, South Central US, East US2, West US2, West Europe, North Europe, Canada Central, South East Asia, Australia East, Central India, Japan East, Korea Central, UK South, UK West, Germany West Central, France Central, East Asia, West US3, Norway East, South African North, North Central US, Australia South East, Switzerland North, Japan West, South India

Prerequisites

AKS Cluster is up and running with Azure CNI powered by Cilium
Azure CLI version 2.41.0 or later. Run az --version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
If using ARM templates or the REST API, the AKS API version must be 2022-09-02-preview or later.

Register resource providers

Before you deploy a container offer, you must register the Microsoft.ContainerService and Microsoft.KubernetesConfiguration providers on your subscription by using the az provider register command:

az provider register --namespace Microsoft.ContainerService --wait
az provider register --namespace Microsoft.KubernetesConfiguration --wait

Select and deploy a Kubernetes offer

Note- There are three ways to deploy a Kubernetes offer:

This document will discuss the method to deploy a Kubernetes offering from the Azure Marketplace.

In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
You can search for an offer or publisher directly by name, or you can browse all offers. To find Kubernetes application offers, on the left side under Categories select Containers.

In the search window type “Isovalent” and select the offer.

On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
Select the respective subscription in which the new AKS cluster needs to be created.
Select the resource group to deploy the cluster in. If a resource group doesn’t exist, click on “Create New”.
Click on Create New Dev Cluster, select “Yes” and click on Next: Cluster Details.

Provide a name for the AKS cluster and click on “Next: Review + Create”

Once Final validation is complete, click on “Create”

When the application is deployed, the portal will show “Your deployment is complete”, along with details of the deployment.

End users can also check the extensions installed on the cluster from Azure Portal. On the AKS cluster, users can navigate to “Extensions + applications (Preview)” menu to verify the same.

Verify the deployment by using the following command to list the extensions that are running on your cluster:

az k8s-extension show --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

Log in to the Azure portal, browse to Kubernetes Services, select the respective Kubernetes service created ( AKS Cluster), and click on connect. This will help end users connect to their AKS cluster and set the individual Kubernetes context.

az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
az aks get-credentials --resource-group <resourcegroup-name> --name <clustername>

Validating the version of Cilium on your newly created cluster

#kubectl --namespace=kube-system exec -i -t ds/cilium  -- cilium version

Client: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64
Daemon: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64

Note– cee-indicates Cilium Enterprise Edition which is the Enterprise Version of Cilium offered by Isovalent that was installed on the cluster.

Upgrade an existing cluster

Note-

AKS Cluster is up and running with Azure CNI powered by Cilium
An existing AKS cluster running Azure CNI powered by Cilium in the same region from where the upgrade is being attempted.
The version on your cluster can be verified:

kubectl --namespace=kube-system exec -i -t ds/cilium  -- cilium version

Client: 1.12.5 701acde56b 2022-12-15T16:03:30-08:00 go version go1.18.9 linux/amd64
Daemon: 1.12.5 701acde56b 2022-12-15T16:03:30-08:00 go version go1.18.9 linux/amd64

Preparing for the upgrade

This section will guide you through the steps required to upgrade an existing AKS cluster running Azure CNI powered by Cilium to Isovalent Enterprise for Cilium.

In the Azure portal, search for Marketplace on the top search bar. In the results, under Services, select Marketplace.
You can search for an offer or publisher directly by name, or you can browse all offers. To find Kubernetes application offers, on the left side under Categories select Containers.

In the search window type “Isovalent” and select the offer.

On the Plans + Pricing tab, select an option. Ensure that the terms are acceptable, and then select Create.
Select the resource group in which the cluster is existing that we will be upgrading.
Click on Create New Dev Cluster, select “No” and click on Next: Cluster Details.

As “No” was selected, this will result in an upgrade of an already existing cluster in that region
The name for the AKS cluster will be auto-populated by clicking on the drop-down selection.
Click on “Next: Review + Create” Details.

Once Final validation is complete, click on “Create”

When the application is deployed, the portal will show “Your deployment is complete”, along with details of the deployment.

End users can also check the extensions installed on the cluster from Azure Portal. On the AKS cluster, users can navigate to “Extensions + applications (Preview)” menu to verify the same.

Verify the deployment by using the following command to list the extensions that are running on your cluster:

az k8s-extension show --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

Log in to the Azure portal and browse to Kubernetes Services, select the respective Kubernetes service that was created ( AKS Cluster), and click on connect. This will help end users connect to their AKS cluster and set the respective Kubernetes context.

az account set --subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
az aks get-credentials --resource-group <resourcegroup-name> --name <clustername>

Validating the version of Cilium on your newly upgraded cluster

kubectl --namespace=kube-system exec -i -t ds/cilium  -- cilium version

Client: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64
Daemon: 1.12.7-cee.1 7e62255 2023-02-13T16:43:36+00:00 go version go1.18.10 linux/amd64

Note– cee-indicates Cilium Enterprise Edition which is the Enterprise Version of Cilium offered by Isovalent that the cluster was upgraded to.

Basic checks

Let’s start by checking the status of the nodes.

kubectl get nodes
NAME                                STATUS   ROLES   AGE   VERSION
aks-nodepool1-87984109-vmss000000   Ready    agent   45m   v1.25.6
aks-nodepool1-87984109-vmss000001   Ready    agent   45m   v1.25.6

Let’s also check the node-to-node health with cilium-health status

cilium-health status
Probe time:   2023-05-25T09:31:52Z
Nodes:
  aks-nodepool1-87984109-vmss000000 (localhost):
    Host connectivity to 10.10.0.5:
      ICMP to stack:   OK, RTT=124.002µs
      HTTP to agent:   OK, RTT=470.504µs
  aks-nodepool1-87984109-vmss000001:
    Host connectivity to 10.10.0.4:
      ICMP to stack:   OK, RTT=2.274623ms
      HTTP to agent:   OK, RTT=3.199333ms

We can even run a cilium connectivity test (an automated test that checks that Cilium has been deployed correctly and tests intra-node connectivity, inter-node connectivity, and network policies) to verify that everything is working as expected.

cilium connectivity test
ℹ️  Monitor aggregation detected, will skip some flow validation steps
✨ [ciliumossazmktplace] Creating namespace cilium-test for connectivity check...
✨ [ciliumossazmktplace] Deploying echo-same-node service...
✨ [ciliumossazmktplace] Deploying DNS test server configmap...
✨ [ciliumossazmktplace] Deploying same-node deployment...
✨ [ciliumossazmktplace] Deploying client deployment...
✨ [ciliumossazmktplace] Deploying client2 deployment...
✨ [ciliumossazmktplace] Deploying echo-other-node service...
✨ [ciliumossazmktplace] Deploying other-node deployment...
⌛ [ciliumossazmktplace] Waiting for deployments [client client2 echo-same-node] to become ready...
⌛ [ciliumossazmktplace] Waiting for deployments [echo-other-node] to become ready...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/client-7b78db77d5-2tgfk to appear...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/client2-78f748dd67-tnzcw to appear...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client-7b78db77d5-2tgfk to reach DNS server on cilium-test/echo-same-node-85bc9b6b56-hrvbg pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client2-78f748dd67-tnzcw to reach DNS server on cilium-test/echo-same-node-85bc9b6b56-hrvbg pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client-7b78db77d5-2tgfk to reach DNS server on cilium-test/echo-other-node-cd69fcf6b-x8hpl pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client2-78f748dd67-tnzcw to reach DNS server on cilium-test/echo-other-node-cd69fcf6b-x8hpl pod...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client-7b78db77d5-2tgfk to reach default/kubernetes service...
⌛ [ciliumossazmktplace] Waiting for pod cilium-test/client2-78f748dd67-tnzcw to reach default/kubernetes service...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/echo-other-node-cd69fcf6b-x8hpl to appear...
⌛ [ciliumossazmktplace] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-85bc9b6b56-hrvbg to appear...
⌛ [ciliumossazmktplace] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [ciliumossazmktplace] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.5:32244 (cilium-test/echo-other-node) to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.5:31540 (cilium-test/echo-same-node) to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.4:32244 (cilium-test/echo-other-node) to become ready...
⌛ [ciliumossazmktplace] Waiting for NodePort 10.10.0.4:31540 (cilium-test/echo-same-node) to become ready...
ℹ️  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠️  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4245: connect: connection refused"
ℹ️  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ️  Cilium version: 1.12.7
🏃 Running tests...
[=] Test [no-policies]
............................
[=] Test [no-policies-extra]
........
[=] Test [allow-all-except-world]
..............
[=] Test [client-ingress]
..
[=] Test [all-ingress-deny]
........
[=] Test [all-egress-deny]
................
[=] Test [all-entities-deny]
........
[=] Test [cluster-entity]
..
[=] Test [host-entity]
....
[=] Test [echo-ingress]
....
[=] Test [client-ingress-icmp]
..
[=] Test [client-egress]
....
[=] Test [client-egress-expression]
....
[=] Test [client-egress-to-echo-service-account]
....
[=] Test [to-entities-world]
......
[=] Test [to-cidr-1111]
....
[=] Test [echo-ingress-from-other-client-deny]
......
[=] Test [client-ingress-from-other-client-icmp-deny]
......
[=] Test [client-egress-to-echo-deny]
......
[=] Test [client-ingress-to-echo-named-port-deny]
....
[=] Test [client-egress-to-echo-expression-deny]
....
[=] Test [client-egress-to-echo-service-account-deny]
....
[=] Test [client-egress-to-cidr-deny]
....
[=] Test [client-egress-to-cidr-deny-default]
....

[=] Skipping Test [health]
[=] Test [echo-ingress-l7]
............
[=] Test [echo-ingress-l7-named-port]
............
[=] Test [client-egress-l7-method]
............
[=] Test [client-egress-l7]
..........
[=] Test [client-egress-l7-named-port]
..........

[=] Skipping Test [client-egress-l7-tls-deny-without-headers]

[=] Skipping Test [client-egress-l7-tls-headers]
[=] Test [dns-only]
..........
[=] Test [to-fqdns]
........

✅ All 31 tests (230 actions) successful, 3 tests skipped, 0 scenarios skipped.

Manage the offer lifecycle

For lifecycle management, an Azure Kubernetes offer is represented as a cluster extension for AKS. For more information, see Cluster extensions for AKS.

Purchasing an offer from Azure Marketplace creates a new instance of the extension on your AKS cluster. You can view the extension instance from the cluster by using the following command:

az k8s-extension show --name <extension-name> --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters

Note– extension-name in this case is cilium

Remove an offer

You can delete a purchased plan for an Azure container offer by deleting the extension instance on the cluster. For example:

az k8s-extension delete --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters -n cilium

Conclusion

Hopefully, this post gave you a good overview of how and why you would deploy Isovalent Enterprise for Cilium on your AKS clusters from the Azure marketplace.

If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel.

Try it Out