Cilium’s popularity as a CNI has been such that users have enabled it not just on cloud providers but have also chosen Cilium as the default CNI in their bare-metal and sandbox environments. On the data plane side, Cilium offers breakthrough features and enhancements. Still, on the control plane side, it becomes imperative that a solution is available wherein users can manage their infrastructure with a single pane of glass. Azure Arc combined with Isovalent Enterprise for Cilium provides a combined view for on-premise and cloud providers. This tutorial teaches you to manage multiple Kubernetes clusters running Isovalent Enterprise for Cilium with Azure Arc.
What is Isovalent Enterprise for Cilium?
Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.
What is Azure Arc?
Azure Arc-enabled Kubernetes allows you to attach Kubernetes clusters running anywhere so that you can manage and configure them in Azure. By managing your Kubernetes resources in a single control plane, you can enable a more consistent development and operation experience to run cloud-native apps anywhere and on any Kubernetes platform. Salient features of Azure Arc include:
- Azure Arc-enabled Kubernetes provides a centralized, consistent control plane to manage policy, governance, and security across Kubernetes clusters in different environments.
- When the Azure Arc agents are deployed to the cluster, an outbound connection to Azure is initiated, using industry-standard SSL to secure data in transit.
- Once clusters are connected to Azure, they’re represented as their resources in Azure Resource Manager, and they can be organized using resource groups and tagging.
Why Isovalent Enterprise for Cilium and Azure Arc?
You get the best out of both offerings with Isovalent Enterprise for Cilium and Azure Arc.
- Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security.
- Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
- Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments.
- Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
- Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making it easy to deploy and manage complex microservices architectures.
- Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.
How can you deploy Isovalent Enterprise for Cilium?
In this tutorial, you will install Isovalent Enterprise for Cilium on Kubernetes clusters using Helm. To obtain the helm values to install Isovalent Enterprise for Cilium and access the Enterprise documentation, you need to reach out to sales@isovalent.com and support@isovalent.com
Pre-Requisites
The following prerequisites must be considered before you proceed with this tutorial.
- Azure CLI version 2.48.1 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
- You should have an Azure Subscription.
- To enable Azure Arc on your Kubernetes clusters, you can install the following dependencies on your development machine or create a VM in the respective environment.
- Install kubectl.
- Install docker.
- Install Cilium CLI.
- Install Helm.
- Install eksctl.
- Install gcloud CLI.
- Install eksctl-anywhere.
- Install AWS CLI.
- An up-and-running Kubernetes cluster. If you don’t have one, you can create a cluster using one of these options:
- Kubernetes in Docker (KIND)
- k3s
- GKE
- EKS
- EKS-Anywhere
- AKS
- Create a Kubernetes cluster using Docker for Mac or Windows
- Self-managed Kubernetes cluster using Cluster API
- At least 850 MB free for the Arc agents that will be deployed on the cluster, and the capacity to use approximately 7% of a single CPU.
- The latest version of connectedk8s Azure CLI extension is installed by running the following command:
Azure Arc Requirements
You need to take care of specific requirements for your Kubernetes cluster to work with Azure Arc:
Create Kubernetes clusters
You can install Azure Arc on any Kubernetes cluster distribution as it applies to your use case. These clusters can be created from your local machine or a VM in the respective resource group/VPC/VNet in the respective cloud distribution. In this tutorial, we will be installing Azure Arc on the following distributions:
- AKS
- EKS
- EKS-Anywhere
- GKE
- Kind
- k3s
Install Isovalent Enterprise for Cilium
To obtain the helm values to install Isovalent Enterprise for Cilium and access the Enterprise documentation, you need to reach out to sales@isovalent.com and support@isovalent.com
Providers for Azure Arc-enabled Kubernetes
Note- The steps below are valid for all the distributions.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
Register providers for Azure Arc-enabled Kubernetes
- Enter the following commands:
- Monitor the registration process. Registration may take up to 10 minutes.
Create a Service Principal
For Azure Arc, you need to create an identity (user or service principal) that can be used to log in to Azure CLI and connect your cluster to Azure Arc. This step is optional if it has already been created previously.
Create a resource group
Create a Resource Group
Connect an existing Kubernetes cluster
Option 1- Deploying the Azure Arc agents to the cluster using az connectedk8s
extension
The command below deploys the Azure Arc agents to the cluster and installs Helm v. 3.6.3 to the .azure
folder of the deployment machine. This Helm 3 installation is only used for Azure Arc, and it doesn’t remove or change any previously installed versions of Helm on the machine.
Option 2- Deploying the Azure Arc agents to the cluster using the Azure portal
- Login to the Azure portal
- Click > Home > Azure Arc> Add a Kubernetes Cluster with Azure Arc
- Click > Next
- Select the subscription and resource group where the kubernetes cluster was created in the previous step.
- Select the already created resource group ( see above) and select the connectivity method as “Public endpoint”
- Click > Next
- You can provide tags (optional)
- Click > Next
- You can now copy or download this script onto the Kubernetes cluster and run it.
- Sample output when the script is run from the Kubernetes cluster
- Once the cluster is connected to Azure, click > close
Verify cluster connection
You can verify the cluster connection by running this command:
View Azure Arc agents for Kubernetes
Azure Arc-enabled Kubernetes deploys several agents into the azure-arc
namespace.
- View the deployments and pods using:
Cluster management via Azure Arc
Now that your cluster(s) is connected to Azure, you can view it from the Azure portal. In this case, multiple clusters can now be managed from the Azure portal via Azure Arc.
- Click > Home > Azure Arc > Kubernetes Clusters
Securely connect to an on-premises Kubernetes Cluster with Azure Arc
You can give users access using RBAC (Role-based access control) and let them connect to the Kubernetes cluster through Azure Arc.
Create a User on the Kubernetes Cluster
- To authorize a user to access the Kubernetes cluster with the
kubeconfig
file pointing to theapiserver
of your Kubernetes cluster, run this command to create a service account. This example creates the service account in the default namespace, but you can substitute any other namespace fordefault
.
- Create ClusterRoleBinding to grant this service account the appropriate permissions on the cluster. If you used a different namespace in the first command, substitute it here for
default
.
- Create a service account token:
- Get the token to output to the console
Access the Kubernetes Cluster in the Azure Portal with Azure Arc
When you open the Azure Arc resource in the Azure Portal and go to any Kubernetes resources pane, you will see a message that you must sign in to view the Kubernetes resources. Sign in to view your Kubernetes resources.
Paste the previously created token into the text box and click Sign in. Now, you should see the resources of the Kubernetes cluster.
Access the Kubernetes cluster from your local machine with Azure Arc
Using the Azure Portal to access the Kubernetes cluster is nice, but many users are used to using kubectl. You can access the Kubernetes cluster from your local machine using the Azure CLI command.
Replace <TOKEN> with the previously created token. You can use this command on any computer if the Azure CLI is installed. The command downloads the Kubernetes config file, sets the context, and creates a proxy connection to the Kubernetes cluster through Azure Arc.
After the connection is established, open a new terminal window and use kubectl as you are used to.
Monitor a Kubernetes Cluster with Azure Monitor and Azure Arc
Azure Arc allows you to project your on-premises Kubernetes cluster into Azure. Doing so enables you to manage the cluster from Azure with tools such as Azure Monitor.
Enable Azure Monitor
- Click Azure Arc> Kubernetes Clusters > (name of cluster) > Insights > Configure Monitoring
- Click Configure Monitoring and choose a Log Analytics workspace. This will create a new Log Analytics Workspace for the metrics and logs of the extensions. You can also use an existing Work Analytics Workspace.
Note- This usually takes 5-10 minutes before insights show up.
Create Dashboards in the Azure Portal
After you install the extension, it collects metric information and sends it to Azure. This allows you to use Azure Monitor the same way you would use Azure VMs.
Click > Azure Arc > Kubernetes clusters > Open Azure Arc > (name of cluster) > Insights >
For even more insight into your cluster or pods, open the Metrics pane in Azure Arc. There, you can create charts and display useful information. The following screenshot shows a chart that displays the pod count and the CPU percentage used for all nodes.
Click > Azure Arc > Kubernetes clusters > Open Azure Arc > (name of cluster) > Metrics >
Validation-Isovalent Enterprise for Cilium
Run cilium connectivity test
(an automated test that checks that Cilium has been deployed correctly and tests intra-node connectivity, inter-node connectivity, and network policies) to verify that everything is working as expected.
Output truncated:
Conclusion
Hopefully, this post gave you a good overview of integrating an existing Kubernetes cluster running Isovalent Enterprise for Cilium with Azure Arc. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel.
Further Reading
- Isovalent Enterprise for Cilium
- Azure and Isovalent main partner page
- Enabling Enterprise features in AKS with Isovalent Enterprise for Cilium
- Try Cilium using our interactive labs
Amit Gupta is a senior technical marketing engineer at Isovalent, powering eBPF cloud-native networking and security. Amit has 21+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source. He has previously worked with Motorola, Juniper, Avi Networks (acquired by VMware), and Prosimo. He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area outside of work. Amit is an avid runner and cyclist and also spends considerable time helping kids in orphanages.