Cilium's popularity as a CNI has been such that users have enabled it not just on cloud providers but have also chosen Cilium as the default CNI in their bare-metal and sandbox environments. On the data plane side, Cilium offers breakthrough features and enhancements. Still, on the control plane side, it becomes imperative that a solution is available wherein users can manage their infrastructure with a single pane of glass. Azure Arc combined with Isovalent Enterprise for Cilium provides a combined view for on-premise and cloud providers. This tutorial teaches you to manage multiple Kubernetes clusters running Isovalent Enterprise for Cilium with Azure Arc. What is Isovalent Enterprise for Cilium? Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns. What is Azure Arc? Azure Arc-enabled Kubernetes allows you to attach Kubernetes clusters running anywhere so that you can manage and configure them in Azure. By managing your Kubernetes resources in a single control plane, you can enable a more consistent development and operation experience to run cloud-native apps anywhere and on any Kubernetes platform. Salient features of Azure Arc include: Azure Arc-enabled Kubernetes provides a centralized, consistent control plane to manage policy, governance, and security across Kubernetes clusters in different environments. When the Azure Arc agents are deployed to the cluster, an outbound connection to Azure is initiated, using industry-standard SSL to secure data in transit. Once clusters are connected to Azure, they’re represented as their resources in Azure Resource Manager, and they can be organized using resource groups and tagging. Why Isovalent Enterprise for Cilium and Azure Arc? You get the best out of both offerings with Isovalent Enterprise for Cilium and Azure Arc. Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security. Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management. Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments. Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments. Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making it easy to deploy and manage complex microservices architectures. Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments. How can you deploy Isovalent Enterprise for Cilium? In this tutorial, you will install Isovalent Enterprise for Cilium on Kubernetes clusters using Helm. To obtain the helm values to install Isovalent Enterprise for Cilium and access the Enterprise documentation, you need to reach out to sales@isovalent.com and support@isovalent.com Pre-Requisites The following prerequisites must be considered before you proceed with this tutorial. Azure CLI version 2.48.1 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI. You should have an Azure Subscription. To enable Azure Arc on your Kubernetes clusters, you can install the following dependencies on your development machine or create a VM in the respective environment. Install kubectl. Install docker. Install Cilium CLI. Install Helm. Install eksctl. Install gcloud CLI. Install eksctl-anywhere. Install AWS CLI. An up-and-running Kubernetes cluster. If you don’t have one, you can create a cluster using one of these options: Kubernetes in Docker (KIND) k3s GKE EKS EKS-Anywhere AKS Create a Kubernetes cluster using Docker for Mac or Windows Self-managed Kubernetes cluster using Cluster API At least 850 MB free for the Arc agents that will be deployed on the cluster, and the capacity to use approximately 7% of a single CPU. The latest version of connectedk8s Azure CLI extension is installed by running the following command: Azure Arc Requirements You need to take care of specific requirements for your Kubernetes cluster to work with Azure Arc: System requirements Networking requirements Connectivity modes Create Kubernetes clusters You can install Azure Arc on any Kubernetes cluster distribution as it applies to your use case. These clusters can be created from your local machine or a VM in the respective resource group/VPC/VNet in the respective cloud distribution. In this tutorial, we will be installing Azure Arc on the following distributions:AKS EKS EKS-Anywhere GKE Kind k3s Install Isovalent Enterprise for Cilium To obtain the helm values to install Isovalent Enterprise for Cilium and access the Enterprise documentation, you need to reach out to sales@isovalent.com and support@isovalent.com Providers for Azure Arc-enabled Kubernetes Note- The steps below are valid for all the distributions. Set the Subscription Choose the subscription you want to use if you have multiple Azure subscriptions. Replace SubscriptionName with your subscription name. You can also use your subscription ID instead of your subscription name. Register providers for Azure Arc-enabled Kubernetes Enter the following commands: Monitor the registration process. Registration may take up to 10 minutes. Create a Service Principal For Azure Arc, you need to create an identity (user or service principal) that can be used to log in to Azure CLI and connect your cluster to Azure Arc. This step is optional if it has already been created previously. Create a resource group Create a Resource Group Connect an existing Kubernetes cluster Option 1- Deploying the Azure Arc agents to the cluster using az connectedk8s extension The command below deploys the Azure Arc agents to the cluster and installs Helm v. 3.6.3 to the .azure folder of the deployment machine. This Helm 3 installation is only used for Azure Arc, and it doesn't remove or change any previously installed versions of Helm on the machine. Option 2- Deploying the Azure Arc agents to the cluster using the Azure portal Log in to the Azure portal Click > Home > Azure Arc> Add a Kubernetes Cluster with Azure Arc Click > Next Select the subscription and resource group where the Kubernetes cluster was created in the previous step. Select the already created resource group ( see above) and select the connectivity method as “Public endpoint.” Click > Next You can provide tags (optional) Click > Next You can now copy or download this script onto the Kubernetes cluster and run it. Sample output when the script is run from the Kubernetes cluster Once the cluster is connected to Azure, click > close Verify cluster connection You can verify the cluster connection by running this command: View Azure Arc agents for Kubernetes Azure Arc-enabled Kubernetes deploys several agents into the azure-arc namespace. View the deployments and pods using: Cluster management via Azure Arc Now that your cluster(s) is connected to Azure, you can view it from the Azure portal. In this case, multiple clusters can now be managed from the Azure portal via Azure Arc. Click > Home > Azure Arc > Kubernetes Clusters Securely connect to an on-premises Kubernetes Cluster with Azure Arc You can give users access using RBAC (Role-based access control) and let them connect to the Kubernetes cluster through Azure Arc. Create a User on the Kubernetes Cluster To authorize a user to access the Kubernetes cluster with the kubeconfig file pointing to the apiserver of your Kubernetes cluster, run this command to create a service account. This example creates the service account in the default namespace, but you can substitute any other namespace for default. Create ClusterRoleBinding to grant this service account the appropriate permissions on the cluster. If you used a different namespace in the first command, substitute it here for default. Create a service account token: Get the token to output to the console Access the Kubernetes Cluster in the Azure Portal with Azure Arc When you open the Azure Arc resource in the Azure Portal and go to any Kubernetes resources pane, you will see a message that you must sign in to view the Kubernetes resources. Sign in to view your Kubernetes resources. Paste the previously created token into the text box and click Sign in. Now, you should see the resources of the Kubernetes cluster. Access the Kubernetes cluster from your local machine with Azure Arc Using the Azure Portal to access the Kubernetes cluster is nice, but many users are used to using kubectl. You can access the Kubernetes cluster from your local machine using the Azure CLI command. Replace <TOKEN> with the previously created token. You can use this command on any computer if the Azure CLI is installed. The command downloads the Kubernetes config file, sets the context, and creates a proxy connection to the Kubernetes cluster through Azure Arc. After the connection is established, open a new terminal window and use kubectl as you are used to. Monitor a Kubernetes Cluster with Azure Monitor and Azure Arc Azure Arc allows you to project your on-premises Kubernetes cluster into Azure. Doing so enables you to manage the cluster from Azure with tools such as Azure Monitor. Enable Azure Monitor Click Azure Arc> Kubernetes Clusters > (name of cluster) > Insights > Configure Monitoring Click Configure Monitoring and choose a Log Analytics workspace. This will create a new Log Analytics Workspace for the metrics and logs of the extensions. You can also use an existing Work Analytics Workspace. Note- This usually takes 5-10 minutes before insights show up. Create Dashboards in the Azure Portal After you install the extension, it collects metric information and sends it to Azure. This allows you to use Azure Monitor the same way you would use Azure VMs. Click > Azure Arc > Kubernetes clusters > Open Azure Arc > (name of cluster) > Insights > For even more insight into your cluster or pods, open the Metrics pane in Azure Arc. There, you can create charts and display useful information. The following screenshot shows a chart that displays the pod count and the CPU percentage used for all nodes. Click > Azure Arc > Kubernetes clusters > Open Azure Arc > (name of cluster) > Metrics > Deploy configurations using GitOps Azure Arc uses Flux under the covers as the mechanism for continuous delivery. Install the latest k8s-configuration and k8s-extension CLI extension packages. Once the extension is installed, configure the local Kubernetes cluster with GitOps using a public Git repository hosted on GitHub. The repository mentioned in the above command contains a manifest file to deploy an Nginx server on a Kubernetes cluster. We configure this manifest in GitOps in our Azure Arc-enabled Kubernetes cluster so that it deploys automatically to our local Kubernetes cluster. The Flux agent can monitor this repository for changes and adapt the cluster for any changes in the file. With our configuration created and installed, we can verify that new Nginx pods are deployed on your local Kubernetes cluster. As soon as GitOps is configured with an Azure Arc-enabled Kubernetes cluster, it will pull the configurations from the public Git repository. This can also be viewed in the Azure Portal. Can GitOps reconfigure deployments in the local Kubernetes clusters for changes to the manifest file in the repository? Change the replicas in the manifest file from 3 to 5 locally, and we will observe (in a few minutes) that the Flux agent running in the local Kubernetes cluster will pick these configuration changes from the repository and begin another replica deployment. Validation: Isovalent Enterprise for Cilium Run cilium connectivity test (an automated test that checks that Cilium has been deployed correctly and tests intra-node connectivity, inter-node connectivity, and network policies) to verify that everything is working as expected. Output truncated: Conclusion Hopefully, this post gave you a good overview of integrating an existing Kubernetes cluster running Isovalent Enterprise for Cilium with Azure Arc. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel. Further Reading Isovalent Enterprise for Cilium Azure and Isovalent main partner page Enabling Enterprise features in AKS with Isovalent Enterprise for Cilium Try Cilium using our interactive labs