Cilium's popularity as a CNI has been such that users have enabled it not just on cloud providers but have also chosen Cilium as the default CNI in their bare-metal and sandbox environments. On the data plane side, Cilium offers breakthrough features and enhancements. Still, on the control plane side, it becomes imperative that a solution is available wherein users can manage their infrastructure with a single pane of glass. Azure Arc combined with Isovalent Enterprise for Cilium provides a combined view for on-premise and cloud providers. This tutorial teaches you to manage multiple Kubernetes clusters running Isovalent Enterprise for Cilium with Azure Arc.
What is Isovalent Enterprise for Cilium?
Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.
What is Azure Arc?
Azure Arc-enabled Kubernetes allows you to attach Kubernetes clusters running anywhere so that you can manage and configure them in Azure. By managing your Kubernetes resources in a single control plane, you can enable a more consistent development and operation experience to run cloud-native apps anywhere and on any Kubernetes platform. Salient features of Azure Arc include:
- Azure Arc-enabled Kubernetes provides a centralized, consistent control plane to manage policy, governance, and security across Kubernetes clusters in different environments.
- When the Azure Arc agents are deployed to the cluster, an outbound connection to Azure is initiated, using industry-standard SSL to secure data in transit.
- Once clusters are connected to Azure, they’re represented as their resources in Azure Resource Manager, and they can be organized using resource groups and tagging.
Why Isovalent Enterprise for Cilium and Azure Arc?
You get the best out of both offerings with Isovalent Enterprise for Cilium and Azure Arc.
- Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security.
- Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
- Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments.
- Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
- Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making it easy to deploy and manage complex microservices architectures.
- Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.
How can you deploy Isovalent Enterprise for Cilium?
In this tutorial, you will install Isovalent Enterprise for Cilium on Kubernetes clusters using Helm. To obtain the helm values to install Isovalent Enterprise for Cilium and access the Enterprise documentation, you need to reach out to sales@isovalent.com and support@isovalent.com
Pre-Requisites
The following prerequisites must be considered before you proceed with this tutorial.
- Azure CLI version 2.48.1 or later. Run az –version to see the currently installed version. If you need to install or upgrade, see Install Azure CLI.
- You should have an Azure Subscription.
- To enable Azure Arc on your Kubernetes clusters, you can install the following dependencies on your development machine or create a VM in the respective environment.
- Install kubectl.
- Install docker.
- Install Cilium CLI.
- Install Helm.
- Install eksctl.
- Install gcloud CLI.
- Install eksctl-anywhere.
- Install AWS CLI.
- An up-and-running Kubernetes cluster. If you don’t have one, you can create a cluster using one of these options:
- Kubernetes in Docker (KIND)
- k3s
- GKE
- EKS
- EKS-Anywhere
- AKS
- Create a Kubernetes cluster using Docker for Mac or Windows
- Self-managed Kubernetes cluster using Cluster API
- At least 850 MB free for the Arc agents that will be deployed on the cluster, and the capacity to use approximately 7% of a single CPU.
- The latest version of connectedk8s Azure CLI extension is installed by running the following command:
az extension add --name connectedk8sAzure Arc Requirements
You need to take care of specific requirements for your Kubernetes cluster to work with Azure Arc:
Create Kubernetes clusters
You can install Azure Arc on any Kubernetes cluster distribution as it applies to your use case. These clusters can be created from your local machine or a VM in the respective resource group/VPC/VNet in the respective cloud distribution. In this tutorial, we will be installing Azure Arc on the following distributions:AKS
- EKS
- EKS-Anywhere
- GKE
- Kind
- k3s
AKS
- Follow the prerequisites to create an AKS cluster. Brief steps are outlined below.
clusterName="nwpluginazurecnioverlay"resourceGroup="nwpluginazurecnioverlay"location="westcentralus"az group create --name nwpluginazurecnioverlay --location westcentralusaz aks create -n $clusterName -g $resourceGroup --location $location --network-plugin azure --network-plugin-mode overlay --pod-cidr 192.168.0.0/16az aks get-credentials --resource-group nwpluginazurecnioverlay --name nwpluginazurecnioverlay
- Check that all the pods are up and running.
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES azure-arc cluster-metadata-operator-55d7754dcb-r5n7g 2/2 Running 0 121m 10.0.0.175 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc clusterconnect-agent-55cff6cb6c-fm2r5 3/3 Running 1 (121m ago) 121m 10.0.0.204 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc clusteridentityoperator-5b57955cf-mvdbr 2/2 Running 0 121m 10.0.1.211 aks-azurecilium-29703265-vmss000001 <none> <none> azure-arc config-agent-55f49f9c6b-z76d6 2/2 Running 0 121m 10.0.0.33 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc controller-manager-84bc58855d-9g2k6 2/2 Running 0 121m 10.0.0.26 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc extension-events-collector-68fb565d69-w9dgv 2/2 Running 0 121m 10.0.0.120 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc extension-manager-5f5f94cc75-q6kfs 3/3 Running 0 121m 10.0.1.136 aks-azurecilium-29703265-vmss000001 <none> <none> azure-arc flux-logs-agent-865c554867-vjtlj 1/1 Running 0 121m 10.0.0.137 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc kube-aad-proxy-5bd88bfc5b-fsdgd 2/2 Running 0 121m 10.0.1.121 aks-azurecilium-29703265-vmss000001 <none> <none> azure-arc logcollector-769866f87b-dqwn6 1/1 Running 0 121m 10.0.0.220 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc metrics-agent-7597589fcd-clmlz 2/2 Running 0 121m 10.0.0.246 aks-azurecilium-29703265-vmss000000 <none> <none> azure-arc resource-sync-agent-8cff88d7b-qjg5m 2/2 Running 0 121m 10.0.0.38 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system azure-cns-9f6cj 1/1 Running 0 149m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system azure-cns-hptpx 1/1 Running 0 149m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system azure-ip-masq-agent-kffhl 1/1 Running 0 149m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system azure-ip-masq-agent-wpfnq 1/1 Running 0 149m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system cilium-dszfg 1/1 Running 0 145m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system cilium-operator-6b8448d64d-f42zr 1/1 Running 0 145m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system cilium-operator-6b8448d64d-zndqm 1/1 Running 0 145m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system cilium-qzjn4 1/1 Running 0 145m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system cloud-node-manager-8cl2f 1/1 Running 0 149m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system cloud-node-manager-m8dqt 1/1 Running 0 149m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system coredns-789789675-mdzmd 1/1 Running 0 144m 10.0.1.89 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system coredns-789789675-xngd2 1/1 Running 0 144m 10.0.0.76 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system coredns-autoscaler-649b947bbd-97v8g 1/1 Running 0 143m 10.0.1.248 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system csi-azuredisk-node-dphtk 3/3 Running 0 149m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system csi-azuredisk-node-hl77j 3/3 Running 0 149m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system csi-azurefile-node-krj2x 3/3 Running 0 149m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system csi-azurefile-node-nwzrn 3/3 Running 0 149m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system konnectivity-agent-5c44d98d75-dqkjk 1/1 Running 0 113m 10.0.1.170 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system konnectivity-agent-5c44d98d75-qp6bg 1/1 Running 0 113m 10.0.0.253 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system kube-proxy-s2dgh 1/1 Running 0 149m 192.168.10.5 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system kube-proxy-zzb6m 1/1 Running 0 149m 192.168.10.4 aks-azurecilium-29703265-vmss000001 <none> <none> kube-system metrics-server-5955767688-29kbq 2/2 Running 0 143m 10.0.0.142 aks-azurecilium-29703265-vmss000000 <none> <none> kube-system metrics-server-5955767688-g9htc 2/2 Running 0 143m 10.0.1.99 aks-azurecilium-29703265-vmss000001 <none> <none>- Follow the prerequisites to create an AKS cluster. Brief steps are outlined below.
EKS
- Follow the prerequisites to create an EKS cluster. Brief steps are outlined below.
- The EKS Managed Nodegroups must be properly tainted to ensure applications pods are properly managed by Cilium:
- managedNodeGroups should be tainted with node.cilium.io/agent-not-ready=true:NoExecute to ensure application pods will only be scheduled once Cilium is ready to manage them.
- Create a cluster config file to create an EKS cluster. Sample cluster config file:
- The EKS Managed Nodegroups must be properly tainted to ensure applications pods are properly managed by Cilium:
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: cluster1 region: ap-southeast-2 managedNodeGroups: - name: ng-1 desiredCapacity: 2 privateNetworking: true # taint nodes so that application pods are # not scheduled/executed until Cilium is deployed. # Alternatively, see the note below. taints: - key: "node.cilium.io/agent-not-ready" value: "true" effect: "NoExecute"- Create an EKS cluster
eksctl create cluster -f <filename>.yaml
- Cilium will manage ENIs instead of VPC CNI, so the aws-node DaemonSet has to be patched to prevent conflict behavior.
kubectl -n kube-system patch daemonset aws-node --type='strategic' -p='{"spec":{"template":{"spec":{"nodeSelector":{"io.cilium/aws-node-enabled":"true"}}}}}'
- Check that all the pods are up and running.
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES azure-arc cluster-metadata-operator-5c947c4764-x4x9d 2/2 Running 0 4d 192.168.125.249 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> azure-arc clusterconnect-agent-6c4589b45f-t4k66 3/3 Running 0 4d 192.168.165.78 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> azure-arc clusteridentityoperator-84fc7cc84b-lkm8k 2/2 Running 0 4d 192.168.162.181 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> azure-arc config-agent-67876558f9-xg2x4 2/2 Running 0 4d 192.168.117.96 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> azure-arc controller-manager-6d8588f6b-2xhnt 2/2 Running 0 4d 192.168.165.252 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> azure-arc extension-events-collector-6d4d98bbd-x25bq 2/2 Running 0 4d 192.168.186.70 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> azure-arc extension-manager-5c4d98b7d4-2qr8d 3/3 Running 0 4d 192.168.108.83 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> azure-arc flux-logs-agent-7c49966559-dwjx2 1/1 Running 0 4d 192.168.186.145 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> azure-arc kube-aad-proxy-6c5c8f9f96-xxmxz 2/2 Running 0 4d 192.168.120.78 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> azure-arc logcollector-774d4f95b-9xpdm 1/1 Running 0 4d 192.168.108.154 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> azure-arc metrics-agent-548bb444f4-h454w 2/2 Running 0 4d 192.168.189.88 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> azure-arc resource-sync-agent-85cddb5fc6-s7nqp 2/2 Running 0 4d 192.168.122.116 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> kube-system cilium-6lrpc 1/1 Running 0 4d 192.168.188.226 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> kube-system cilium-operator-6f76bbbdd8-d8rq6 1/1 Running 0 4d 192.168.188.226 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> kube-system cilium-operator-6f76bbbdd8-rnhpn 1/1 Running 0 4d 192.168.117.188 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> kube-system cilium-wrhdc 1/1 Running 0 4d 192.168.117.188 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> kube-system coredns-58fc594bdd-7pdgm 1/1 Running 0 4d 192.168.104.131 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> kube-system coredns-58fc594bdd-zp89g 1/1 Running 0 4d 192.168.168.41 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none> kube-system kube-proxy-2lprt 1/1 Running 0 4d 192.168.117.188 ip-192-168-117-188.ap-southeast-2.compute.internal <none> <none> kube-system kube-proxy-hltbw 1/1 Running 0 4d 192.168.188.226 ip-192-168-188-226.ap-southeast-2.compute.internal <none> <none>- Follow the prerequisites to create an EKS cluster. Brief steps are outlined below.
EKS-Anywhere
- Follow the prerequisites to create an EKS-Anywhere (Docker-based) cluster. Brief steps are outlined below.
- Generate a sample cluster_name yaml file.
CLUSTER_NAME=mgmt
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
--provider docker > $CLUSTER_NAME.yaml
- Sample
cluster_nameyaml file would like:
cat mgmt.yaml apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: Cluster metadata: name: mgmt spec: clusterNetwork: cniConfig: cilium: {} pods: cidrBlocks: - 192.168.0.0/16 services: cidrBlocks: - 10.96.0.0/12 controlPlaneConfiguration: count: 1 datacenterRef: kind: DockerDatacenterConfig name: mgmt externalEtcdConfiguration: count: 1 kubernetesVersion: "1.25" managementCluster: name: mgmt workerNodeGroupConfigurations: - count: 1 name: md-0--- apiVersion: anywhere.eks.amazonaws.com/v1alpha1 kind: DockerDatacenterConfig metadata: name: mgmt spec: {}---- Create a cluster using the
$CLUSTER_NAME.yamlfile from above- eksctl anywhere create cluster -f $CLUSTER_NAME.yaml
- Set KUBECONFIG environment variable
- export KUBECONFIG=/home/ubuntu/mgmt/mgmt-eks-a-cluster.kubeconfig
- Check that all the pods are up and running.
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES azure-arc cluster-metadata-operator-5c947c4764-85wpz 2/2 Running 0 4d 192.168.1.13 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc clusterconnect-agent-6c4589b45f-57c8w 3/3 Running 0 4d 192.168.1.88 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc clusteridentityoperator-6d67448c8-flbwj 2/2 Running 0 4d 192.168.1.98 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc config-agent-7b4dfd9dc-tswpm 2/2 Running 0 4d 192.168.1.20 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc controller-manager-6d8588f6b-gnhp9 2/2 Running 0 4d 192.168.1.79 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc extension-events-collector-74899b879b-gcgct 2/2 Running 0 4d 192.168.1.58 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc extension-manager-6966664764-xnjrs 3/3 Running 0 4d 192.168.1.40 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc flux-logs-agent-7c49966559-9pwcv 1/1 Running 0 4d 192.168.1.50 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc kube-aad-proxy-6c5c8f9f96-6bhqv 2/2 Running 0 4d 192.168.1.46 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc logcollector-774d4f95b-692k5 1/1 Running 0 4d 192.168.1.215 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc metrics-agent-548bb444f4-bcmt8 2/2 Running 0 4d 192.168.1.102 mgmt-md-0-6c9f655895-pj9qr <none> <none> azure-arc resource-sync-agent-85cddb5fc6-r4ttw 2/2 Running 0 4d 192.168.1.18 mgmt-md-0-6c9f655895-pj9qr <none> <none> cert-manager cert-manager-7bc7f646f6-fgpc4 1/1 Running 0 4d 192.168.1.115 mgmt-md-0-6c9f655895-pj9qr <none> <none> cert-manager cert-manager-cainjector-c9dcfc65f-t75lk 1/1 Running 0 4d 192.168.1.182 mgmt-md-0-6c9f655895-pj9qr <none> <none> cert-manager cert-manager-webhook-59b7d96854-4zcfv 1/1 Running 0 4d 192.168.1.131 mgmt-md-0-6c9f655895-pj9qr <none> <none> kube-system cilium-hsx7v 1/1 Running 0 4d 172.18.0.5 mgmt-qg4dr <none> <none> kube-system cilium-operator-5966986c94-m9wjs 1/1 Running 0 4d 172.18.0.5 mgmt-qg4dr <none> <none> kube-system cilium-operator-5966986c94-pd2fx 1/1 Running 0 4d 172.18.0.6 mgmt-md-0-6c9f655895-pj9qr <none> <none> kube-system cilium-t8m82 1/1 Running 0 4d 172.18.0.6 mgmt-md-0-6c9f655895-pj9qr <none> <none> kube-system coredns-84c5c98b7f-792q6 1/1 Running 0 4d 192.168.0.192 mgmt-qg4dr <none> <none> kube-system coredns-84c5c98b7f-t9sf4 1/1 Running 0 4d 192.168.0.91 mgmt-qg4dr <none> <none> kube-system hubble-relay-5d5bcdc6b4-bz8sg 1/1 Running 0 4d 192.168.1.158 mgmt-md-0-6c9f655895-pj9qr <none> <none> kube-system hubble-ui-869b75b895-2864b 2/2 Running 0 4d 192.168.1.67 mgmt-md-0-6c9f655895-pj9qr <none> <none> kube-system kube-apiserver-mgmt-qg4dr 1/1 Running 0 4d 172.18.0.5 mgmt-qg4dr <none> <none> kube-system kube-controller-manager-mgmt-qg4dr 1/1 Running 0 4d 172.18.0.5 mgmt-qg4dr <none> <none> kube-system kube-proxy-9wscb 1/1 Running 0 4d 172.18.0.5 mgmt-qg4dr <none> <none> kube-system kube-proxy-vhxm6 1/1 Running 0 4d 172.18.0.6 mgmt-md-0-6c9f655895-pj9qr <none> <none> kube-system kube-scheduler-mgmt-qg4dr 1/1 Running 0 4d 172.18.0.5 mgmt-qg4dr <none> <none>- Upgrade to Isovalent Enterprise for Cilium.
- Before you install Isovalent Enterprise for Cilium, make sure that you delete a few accounts, secrets, clusterrolebinding, and, clusterroles. This will be fixed in an upcoming release.
kubectl delete serviceaccount cilium --namespace kube-system
kubectl delete serviceaccount cilium-operator --namespace kube-system
kubectl delete secret hubble-ca-secret --namespace kube-system
kubectl delete secret hubble-server-certs --namespace kube-system
kubectl delete configmap cilium-config --namespace kube-system
kubectl delete clusterrole cilium
kubectl delete clusterrolebinding cilium
kubectl delete clusterrolebinding cilium-operator
kubectl delete secret cilium-ca --namespace kube-system
kubectl delete service hubble-peer --namespace kube-system
kubectl delete daemonset cilium --namespace kube-system
kubectl delete deployment cilium-operator --namespace kube-system
kubectl delete clusterrole cilium-operator
- Before you install Isovalent Enterprise for Cilium, make sure that you delete a few accounts, secrets, clusterrolebinding, and, clusterroles. This will be fixed in an upcoming release.
GKE
- Follow the prerequisites to create a GKE cluster. Brief steps are outlined below.
export PROJECT='<project-name>'export NAME='$(whoami)-test-$RANDOM'export ZONE='us-west2-a' export OWNER='$(whoami)'export EXPIRY='$(date "+%Y-%m-%d")'export LABELS='usage=dev-$(whoami),owner=${OWNER},expiry=${EXPIRY}'export COST_REDUCTION=(--num-nodes 2 --machine-type e2-custom-2-4096 --disk-type pd-standard --disk-size 10GB --preemptible)gcloud container clusters create '${NAME}' --project '${PROJECT}' --zone '${ZONE}' --labels '${LABELS}' --image-type COS_CONTAINERD '${COST_REDUCTION[@]}'gcloud container clusters get-credentials '${NAME}' --project '${PROJECT}' --zone '${ZONE}'
- The cluster should be created with the
taint node.cilium.io/agent-not-ready=true:NoExecuteusing--node-taintsoption. - Check that all the pods are up and running.
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES azure-arc cluster-metadata-operator-55d7754dcb-nwzhk 2/2 Running 0 12h 10.36.0.11 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc clusterconnect-agent-55cff6cb6c-v8gls 3/3 Running 0 12h 10.36.0.15 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc clusteridentityoperator-78c784df6c-kqs78 2/2 Running 0 12h 10.36.0.2 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc config-agent-b77f8ff94-d8t9b 2/2 Running 1 12h 10.36.0.13 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc config-agent-b77f8ff94-kh47b 2/2 Running 0 14m 10.36.1.13 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> azure-arc controller-manager-84bc58855d-6thxm 2/2 Running 0 12h 10.36.0.6 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc extension-events-collector-74dfd97d7c-8lrn7 2/2 Running 0 12h 10.36.0.10 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc extension-manager-f5cf5b77f-5s95v 3/3 Running 396 (26s ago) 12h 10.36.0.5 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc flux-logs-agent-865c554867-5gvkr 1/1 Running 0 12h 10.36.0.8 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc kube-aad-proxy-5bd88bfc5b-td8fr 2/2 Running 0 25m 10.36.0.16 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc logcollector-769866f87b-lv82t 1/1 Running 0 25m 10.36.0.116 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> azure-arc metrics-agent-7597589fcd-m8mtj 2/2 Running 0 16m 10.36.1.132 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> azure-arc resource-sync-agent-8cff88d7b-9kf4d 2/2 Running 0 25m 10.36.0.39 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> gmp-system alertmanager-0 2/2 Running 0 12h 10.36.0.20 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> gmp-system collector-gvzfr 2/2 Running 0 16m 10.36.1.2 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> gmp-system collector-rfp4r 2/2 Running 0 12h 10.36.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> gmp-system gmp-operator-66474c4fd-t984m 1/1 Running 0 12h 10.36.0.12 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> gmp-system rule-evaluator-696469b6-jrtr7 2/2 Running 12 (2m36s ago) 12h 10.36.0.16 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system ama-logs-bhvn6 3/3 Running 1 (11m ago) 15m 10.36.1.103 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system ama-logs-rs-799cf7784-xkf9g 2/2 Running 0 17m 10.36.1.248 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system ama-logs-w4fkt 3/3 Running 0 12h 10.36.0.7 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system cilium-42rzz 1/1 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system cilium-f77jb 1/1 Running 0 16m 10.168.0.3 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system cilium-node-init-2x47x 1/1 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system cilium-node-init-h2r8z 1/1 Running 0 16m 10.168.0.3 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system cilium-operator-6d77c7bddb-ktkzc 1/1 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system event-exporter-gke-794b4c457b-fsm8t 2/2 Running 0 16m 10.36.1.77 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system fluentbit-gke-t9zqn 2/2 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system fluentbit-gke-w7pn7 2/2 Running 0 16m 10.168.0.3 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system gke-metrics-agent-9jcwj 2/2 Running 0 16m 10.168.0.3 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system gke-metrics-agent-ggf9v 2/2 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system konnectivity-agent-5b77b6fcdc-gpbgk 1/1 Running 0 16m 10.36.0.131 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system konnectivity-agent-5b77b6fcdc-gw56h 1/1 Running 0 12h 10.36.0.5 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system konnectivity-agent-autoscaler-5d9dbcc6d8-hvpcw 1/1 Running 0 12h 10.36.0.96 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system kube-dns-649bbf755c-rwhgt 4/4 Running 0 25m 10.36.0.163 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system kube-dns-649bbf755c-s428f 4/4 Running 0 16m 10.36.1.10 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system kube-dns-autoscaler-84b8db4dc7-6h689 1/1 Running 0 12h 10.36.0.17 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system kube-proxy-gke-amit-test-1290-default-pool-d00bcc60-e4nl 1/1 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system kube-proxy-gke-amit-test-1290-default-pool-d00bcc60-lsug 1/1 Running 0 16m 10.168.0.3 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none> kube-system l7-default-backend-594666b966-k2tlw 1/1 Running 0 12h 10.36.0.3 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system metrics-server-v0.5.2-666c796f49-6f72w 2/2 Running 0 12h 10.36.0.18 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system pdcsi-node-gsjq5 2/2 Running 0 12h 10.168.0.4 gke-amit-test-1290-default-pool-d00bcc60-e4nl <none> <none> kube-system pdcsi-node-pp9d7 2/2 Running 0 16m 10.168.0.3 gke-amit-test-1290-default-pool-d00bcc60-lsug <none> <none>- Follow the prerequisites to create a GKE cluster. Brief steps are outlined below.
Kind
- Follow the pre-requisites to create a Kind cluster. Brief steps are outlined below.
- Configuring kind cluster creation is done using a YAML configuration file. This step is necessary to disable the default CNI and replace it with Cilium.
- Create a file based on the following template. It will create a cluster with 3 worker nodes and 1 control-plane node. Give it an arbitrary name
kind-config.yaml.
kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker - role: worker networking: disableDefaultCNI: true- Create a cluster with the configuration defined above in the kind-config.yaml file.
kind create cluster --config=kind-config.yaml
- After a couple of seconds or minutes, a 4-node cluster should be created. Check that all the pods are up and running.
- A new kubectl context (
kind-kind) should be added toKUBECONFIGor, if unset, to${HOME}/.kube/config:kubectl cluster-info --context kind-kind
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system cilium-b8qx8 1/1 Running 0 14m 192.168.49.2 kind <none> <none> kube-system cilium-operator-f5dcdcc8d-fr2dw 1/1 Running 0 14m 192.168.49.2 kind <none> <none> kube-system coredns-5dd5756b68-b4qpn 1/1 Running 0 14m 10.0.0.108 kind <none> <none> kube-system etcd-kind 1/1 Running 0 16m 192.168.49.2 kind <none> <none> kube-system kube-apiserver-kind 1/1 Running 0 16m 192.168.49.2 kind <none> <none> kube-system kube-controller-manager-kind 1/1 Running 0 16m 192.168.49.2 kind <none> <none> kube-system kube-proxy-785pj 1/1 Running 0 16m 192.168.49.2 kind <none> <none> kube-system kube-scheduler-kind 1/1 Running 0 16m 192.168.49.2 kind <none> <none> kube-system storage-provisioner 1/1 Running 1 (15m ago) 16m 192.168.49.2 kind <none> <none>k3s
- Follow the pre-requisites to create a k3s cluster. Brief steps are outlined below.
- Install your k3s cluster with 1 master and 1 worker node (optional).
- Disable support for the default CNI plugin and the built-in network policy enforcer so you can install Cilium on top:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='--flannel-backend=none --disable-network-policy' sh -
- Fetch the token for the worker node from the master node:
cat /var/lib/rancher/k3s/server/node-token
- SSH to the worker node and execute
curl -sfL https://get.k3s.io | K3S_URL=https://<master-node-IP>:6443 K3S_TOKEN=<input token from previous command> sh -
- Set KUBECONFIG environment variable
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
- Check that all the pods are up and running.
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system cilium-operator-89b79bd9f-z7ghb 1/1 Running 0 3d20h 10.0.0.8 k3sazurearcworker <none> <none> kube-system cilium-m468m 1/1 Running 0 3d20h 10.0.0.7 k3sazurearc <none> <none> kube-system svclb-traefik-4f02c92e-2rkmg 2/2 Running 2 (3d20h ago) 3d20h 10.42.0.59 k3sazurearc <none> <none> kube-system coredns-6799fbcd5-5bb9f 1/1 Running 0 3d20h 10.42.0.211 k3sazurearc <none> <none> kube-system traefik-f4564c4f4-x2d7t 1/1 Running 10 (3d20h ago) 3d20h 10.42.0.68 k3sazurearc <none> <none> kube-system metrics-server-67c658944b-wb8bh 1/1 Running 9 (3d20h ago) 3d20h 10.42.0.132 k3sazurearc <none> <none> kube-system cilium-dcvgj 1/1 Running 0 3d20h 10.0.0.8 k3sazurearcworker <none> <none> kube-system svclb-traefik-4f02c92e-zctsz 2/2 Running 2 3d20h 10.42.1.35 k3sazurearcworker <none> <none> kube-system local-path-provisioner-84db5d44d9-wlkjv 1/1 Running 8 3d20h 10.42.1.12 k3sazurearcworker <none> <none> azure-arc extension-events-collector-7d7c5784fb-n5dbj 2/2 Running 0 3d20h 10.42.0.52 k3sazurearc <none> <none> azure-arc extension-manager-8694d47d58-vvbmn 3/3 Running 0 3d20h 10.42.0.133 k3sazurearc <none> <none> azure-arc flux-logs-agent-6f8867dfb5-5wblr 1/1 Running 0 3d20h 10.42.1.118 k3sazurearcworker <none> <none> azure-arc cluster-metadata-operator-669654468-tqngh 2/2 Running 0 3d20h 10.42.1.119 k3sazurearcworker <none> <none> azure-arc controller-manager-54cb466bd6-kr6mp 2/2 Running 0 3d20h 10.42.1.199 k3sazurearcworker <none> <none> azure-arc resource-sync-agent-9cb4d78fb-xp62z 2/2 Running 0 3d20h 10.42.1.236 k3sazurearcworker <none> <none> azure-arc clusteridentityoperator-54b675f998-sm6q8 2/2 Running 0 3d20h 10.42.1.90 k3sazurearcworker <none> <none> azure-arc metrics-agent-6888c65674-wltjt 2/2 Running 0 3d20h 10.42.1.116 k3sazurearcworker <none> <none> azure-arc logcollector-54dc695898-lpwkb 1/1 Running 0 3d20h 10.42.1.23 k3sazurearcworker <none> <none> azure-arc clusterconnect-agent-85b6d95d77-dbkwg 3/3 Running 0 3d20h 10.42.1.67 k3sazurearcworker <none> <none> azure-arc config-agent-bb74fb567-4pblx 2/2 Running 0 3d20h 10.42.0.11 k3sazurearc <none> <none> azure-arc kube-aad-proxy-7769ddc8f6-vfxjt 2/2 Running 0 3d20h 10.42.1.140 k3sazurearcworker <none> <none>
Install Isovalent Enterprise for Cilium
To obtain the helm values to install Isovalent Enterprise for Cilium and access the Enterprise documentation, you need to reach out to sales@isovalent.com and support@isovalent.com
Providers for Azure Arc-enabled Kubernetes
Note- The steps below are valid for all the distributions.
Set the Subscription
Choose the subscription you want to use if you have multiple Azure subscriptions.
- Replace SubscriptionName with your subscription name.
- You can also use your subscription ID instead of your subscription name.
az account set --subscription SubscriptionNameRegister providers for Azure Arc-enabled Kubernetes
- Enter the following commands:
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation- Monitor the registration process. Registration may take up to 10 minutes.
az provider show -n Microsoft.Kubernetes -o table
Namespace RegistrationPolicy RegistrationState
-------------------- -------------------- -------------------
Microsoft.Kubernetes RegistrationRequired Registered
az provider show -n Microsoft.KubernetesConfiguration -o table
Namespace RegistrationPolicy RegistrationState
--------------------------------- -------------------- -------------------
Microsoft.KubernetesConfiguration RegistrationRequired Registered
az provider show -n Microsoft.ExtendedLocation -o table
Namespace RegistrationPolicy RegistrationState
-------------------------- -------------------- -------------------
Microsoft.ExtendedLocation RegistrationRequired RegisteredCreate a Service Principal
For Azure Arc, you need to create an identity (user or service principal) that can be used to log in to Azure CLI and connect your cluster to Azure Arc. This step is optional if it has already been created previously.
az ad sp create-for-rbac -n kindazurearc --role contributor --scopes /subscriptions/##############################
Creating 'contributor' role assignment under scope '/subscriptions/##############################'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
"appId": "##################################",
"displayName": "kindazurearc",
"password": "##############################",
"tenant": "#################################"
}Create a resource group
Create a Resource Group
az group create --name AzureArc --location eastusConnect an existing Kubernetes cluster
Option 1- Deploying the Azure Arc agents to the cluster using az connectedk8s extension
The command below deploys the Azure Arc agents to the cluster and installs Helm v. 3.6.3 to the .azure folder of the deployment machine. This Helm 3 installation is only used for Azure Arc, and it doesn't remove or change any previously installed versions of Helm on the machine.
az connectedk8s connect --name kindazurearc --resource-group AzureArc --location eastus
This operation might take a while...
Downloading kubectl client for first time. This can take few minutes...
Downloading helm client for first time. This can take few minutes...
The required pre-checks for onboarding have succeeded.
Azure resource provisioning has begun.
Azure resource provisioning has finished.
Starting to install Azure arc agents on the Kubernetes cluster.
{
"agentPublicKeyCertificate": "#################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################",
"agentVersion": null,
"connectivityStatus": "Connected",
"distribution": "kind",
"id": "/subscriptions/#############
##########/resourceGroups/AzureArc/providers/Microsoft.Kubernetes/connectedClusters/kindazurearc",
"identity": {
"principalId": "#######################",
"tenantId": "#######################",
"type": "SystemAssigned"
},
"infrastructure": "generic",
"kubernetesVersion": null,
"lastConnectivityTime": null,
"location": "eastus",
"managedIdentityCertificateExpirationTime": null,
"name": "kindazurearc",
"offering": null,
"provisioningState": "Succeeded",
"resourceGroup": "AzureArc",
"systemData": {
"createdAt": "2023-12-20T12:09:07.781678+00:00",
"createdBy": "amit.gupta@isovalent.com",
"createdByType": "User",
"lastModifiedAt": "2023-12-20T12:09:07.781678+00:00",
"lastModifiedBy": "amit.gupta@isovalent.com",
"lastModifiedByType": "User"
},
"tags": {},
"totalCoreCount": null,
"totalNodeCount": null,
"type": "microsoft.kubernetes/connectedclusters"
}Option 2- Deploying the Azure Arc agents to the cluster using the Azure portal
- Log in to the Azure portal
- Click > Home > Azure Arc> Add a Kubernetes Cluster with Azure Arc
- Click > Next
- Select the subscription and resource group where the Kubernetes cluster was created in the previous step.
- Select the already created resource group ( see above) and select the connectivity method as “Public endpoint.”
- Click > Next
- You can provide tags (optional)
- Click > Next
- You can now copy or download this script onto the Kubernetes cluster and run it.
- Sample output when the script is run from the Kubernetes cluster
az connectedk8s connect --name "kindazurearc" --resource-group "kubeadm" --location "eastus" --correlation-id "#################################" --tags "."
This operation might take a while...
The required pre-checks for onboarding have succeeded.
Azure resource provisioning has begun.
Azure resource provisioning has finished.
Starting to install Azure arc agents on the Kubernetes cluster.
{
"agentPublicKeyCertificate": "############################################################################################################################################################################################################################################################################################################################################################################################################################################################################",
"agentVersion": null,
"connectivityStatus": "Connected",
"distribution": "kind",
"id": "/subscriptions/############ ############/resourceGroups/kubeadm/providers/Microsoft.Kubernetes/connectedClusters/kindazurearc",
"identity": {
"principalId": "###################################",
"tenantId": "###################################",
"type": "SystemAssigned"
},
"infrastructure": "generic",
"kubernetesVersion": null,
"lastConnectivityTime": null,
"location": "eastus",
"managedIdentityCertificateExpirationTime": null,
"name": "kindazurearc",
"offering": null,
"provisioningState": "Succeeded",
"resourceGroup": "kubeadm",
"systemData": {
"createdAt": "2023-12-20T15:29:46.743307+00:00",
"createdBy": "amit.gupta@isovalent.com",
"createdByType": "User",
"lastModifiedAt": "2023-12-20T15:29:46.743307+00:00",
"lastModifiedBy": "amit.gupta@isovalent.com",
"lastModifiedByType": "User"
},
"tags": {
".": ""
},
"totalCoreCount": null,
"totalNodeCount": null,
"type": "microsoft.kubernetes/connectedclusters"
}
- Once the cluster is connected to Azure, click > close
Verify cluster connection
You can verify the cluster connection by running this command:
az connectedk8s list --resource-group <resourceGroup> -o table
Name Location ResourceGroup
---------- ---------- ---------------
azureaksvm eastus azureaksvmView Azure Arc agents for Kubernetes
Azure Arc-enabled Kubernetes deploys several agents into the azure-arc namespace.
- View the deployments and pods using:
kubectl get deployments,pods -n azure-arc
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cluster-metadata-operator 1/1 1 1 88s
deployment.apps/clusterconnect-agent 1/1 1 1 88s
deployment.apps/clusteridentityoperator 1/1 1 1 88s
deployment.apps/config-agent 1/1 1 1 88s
deployment.apps/controller-manager 1/1 1 1 88s
deployment.apps/extension-events-collector 1/1 1 1 88s
deployment.apps/extension-manager 1/1 1 1 88s
deployment.apps/flux-logs-agent 1/1 1 1 88s
deployment.apps/kube-aad-proxy 1/1 1 1 88s
deployment.apps/logcollector 1/1 1 1 88s
deployment.apps/metrics-agent 1/1 1 1 88s
deployment.apps/resource-sync-agent 1/1 1 1 88s
NAME READY STATUS RESTARTS AGE
pod/cluster-metadata-operator-55d7754dcb-5f9cm 2/2 Running 0 87s
pod/clusterconnect-agent-55cff6cb6c-xscd9 3/3 Running 0 87s
pod/clusteridentityoperator-69f57bbb8f-zmpms 2/2 Running 0 87s
pod/config-agent-85569c4fc6-hkp48 2/2 Running 0 87s
pod/controller-manager-84bc58855d-bth4t 2/2 Running 0 87s
pod/extension-events-collector-5b647476ff-66fnn 2/2 Running 0 88s
pod/extension-manager-78649745d5-86hj2 3/3 Running 0 88s
pod/flux-logs-agent-865c554867-nr5wk 1/1 Running 0 88s
pod/kube-aad-proxy-5bd88bfc5b-4z5mz 2/2 Running 0 87s
pod/logcollector-769866f87b-npmr6 1/1 Running 0 87s
pod/metrics-agent-7597589fcd-vqsk9 2/2 Running 0 87s
pod/resource-sync-agent-8cff88d7b-gffv9 2/2 Running 0 88sCluster management via Azure Arc
Now that your cluster(s) is connected to Azure, you can view it from the Azure portal. In this case, multiple clusters can now be managed from the Azure portal via Azure Arc.
- Click > Home > Azure Arc > Kubernetes Clusters
Securely connect to an on-premises Kubernetes Cluster with Azure Arc
You can give users access using RBAC (Role-based access control) and let them connect to the Kubernetes cluster through Azure Arc.
Create a User on the Kubernetes Cluster
- To authorize a user to access the Kubernetes cluster with the
kubeconfigfile pointing to theapiserverof your Kubernetes cluster, run this command to create a service account. This example creates the service account in the default namespace, but you can substitute any other namespace fordefault.
kubectl create serviceaccount demo-user -n default
serviceaccount/demo-user created- Create ClusterRoleBinding to grant this service account the appropriate permissions on the cluster. If you used a different namespace in the first command, substitute it here for
default.
kubectl create clusterrolebinding demo-user-binding --clusterrole cluster-admin --serviceaccount default:demo-user- Create a service account token:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: demo-user-secret
annotations:
kubernetes.io/service-account.name: demo-user
type: kubernetes.io/service-account-token
EOFTOKEN=$(kubectl get secret demo-user-secret -o jsonpath='{$.data.token}' | base64 -d | sed 's/$/
/g')- Get the token to output to the console
echo $TOKENAccess the Kubernetes Cluster in the Azure Portal with Azure Arc
When you open the Azure Arc resource in the Azure Portal and go to any Kubernetes resources pane, you will see a message that you must sign in to view the Kubernetes resources. Sign in to view your Kubernetes resources.
Paste the previously created token into the text box and click Sign in. Now, you should see the resources of the Kubernetes cluster.
Access the Kubernetes cluster from your local machine with Azure Arc
Using the Azure Portal to access the Kubernetes cluster is nice, but many users are used to using kubectl. You can access the Kubernetes cluster from your local machine using the Azure CLI command.
az connectedk8s proxy -n <clusterName> -g <resourceGroup> --token <TOKEN>Replace <TOKEN> with the previously created token. You can use this command on any computer if the Azure CLI is installed. The command downloads the Kubernetes config file, sets the context, and creates a proxy connection to the Kubernetes cluster through Azure Arc.
After the connection is established, open a new terminal window and use kubectl as you are used to.
Monitor a Kubernetes Cluster with Azure Monitor and Azure Arc
Azure Arc allows you to project your on-premises Kubernetes cluster into Azure. Doing so enables you to manage the cluster from Azure with tools such as Azure Monitor.
Enable Azure Monitor
- Click Azure Arc> Kubernetes Clusters > (name of cluster) > Insights > Configure Monitoring
- Click Configure Monitoring and choose a Log Analytics workspace. This will create a new Log Analytics Workspace for the metrics and logs of the extensions. You can also use an existing Work Analytics Workspace.
Note- This usually takes 5-10 minutes before insights show up.
Create Dashboards in the Azure Portal
After you install the extension, it collects metric information and sends it to Azure. This allows you to use Azure Monitor the same way you would use Azure VMs.
Click > Azure Arc > Kubernetes clusters > Open Azure Arc > (name of cluster) > Insights >
For even more insight into your cluster or pods, open the Metrics pane in Azure Arc. There, you can create charts and display useful information. The following screenshot shows a chart that displays the pod count and the CPU percentage used for all nodes.
Click > Azure Arc > Kubernetes clusters > Open Azure Arc > (name of cluster) > Metrics >
Deploy configurations using GitOps
Azure Arc uses Flux under the covers as the mechanism for continuous delivery.
Install the latest k8s-configuration and k8s-extension CLI extension packages.
az extension add -n k8s-configuration
az extension add -n k8s-extension- Once the extension is installed, configure the local Kubernetes cluster with GitOps using a public Git repository hosted on GitHub.
az k8s-configuration flux create --name azure-arc-config --cluster-name kindazurearc --resource-group AzureArc --url https://github.com/amitmavgupta/azure-arc-gitops --branch main --scope cluster --cluster-type connectedClusters- The repository mentioned in the above command contains a manifest file to deploy an Nginx server on a Kubernetes cluster. We configure this manifest in GitOps in our Azure Arc-enabled Kubernetes cluster so that it deploys automatically to our local Kubernetes cluster. The Flux agent can monitor this repository for changes and adapt the cluster for any changes in the file.
- With our configuration created and installed, we can verify that new Nginx pods are deployed on your local Kubernetes cluster.
kubectl get pods -n nginx-webserver
NAME READY STATUS RESTARTS AGE
nginx-deployment-6cfb64b7c5-bbpqq 1/1 Running 0 27m
nginx-deployment-6cfb64b7c5-c5nxx 1/1 Running 0 27m
nginx-deployment-6cfb64b7c5-l5kdb 1/1 Running 0 27m- As soon as GitOps is configured with an Azure Arc-enabled Kubernetes cluster, it will pull the configurations from the public Git repository. This can also be viewed in the Azure Portal.
- Can GitOps reconfigure deployments in the local Kubernetes clusters for changes to the manifest file in the repository?
- Change the replicas in the manifest file from 3 to 5 locally, and we will observe (in a few minutes) that the Flux agent running in the local Kubernetes cluster will pick these configuration changes from the repository and begin another replica deployment.
kubectl get pods -n nginx-webserver
NAME READY STATUS RESTARTS AGE
nginx-deployment-6cfb64b7c5-bbpqq 1/1 Running 0 28m
nginx-deployment-6cfb64b7c5-c5nxx 1/1 Running 0 28m
nginx-deployment-6cfb64b7c5-l5kdb 1/1 Running 0 28m
nginx-deployment-6cfb64b7c5-qjqj6 1/1 Running 0 1m
nginx-deployment-6cfb64b7c5-t2ztb 1/1 Running 0 1mValidation: Isovalent Enterprise for Cilium
Run cilium connectivity test (an automated test that checks that Cilium has been deployed correctly and tests intra-node connectivity, inter-node connectivity, and network policies) to verify that everything is working as expected.
Output truncated:
[=] Kind [client-egress-l7-named-port]
........
[=] Skipping Kind [client-egress-l7-tls-deny-without-headers]
[=] Skipping Kind [client-egress-l7-tls-headers]
[=] Skipping Kind [client-egress-l7-set-header]
[=] Skipping Kind [echo-ingress-auth-always-fail]
[=] Skipping Kind [echo-ingress-mutual-auth-spiffe]
[=] Skipping Kind [pod-to-ingress-service]
[=] Skipping Kind [pod-to-ingress-service-deny-all]
[=] Skipping Kind [pod-to-ingress-service-allow-ingress-identity]
[=] Kind [dns-only]
........
[=] Kind [to-fqdns]
........
✅ All 42 tests (182 actions) successful, 13 tests skipped, 1 scenarios skipped.Conclusion
Hopefully, this post gave you a good overview of integrating an existing Kubernetes cluster running Isovalent Enterprise for Cilium with Azure Arc. If you have any feedback on the solution, please share it with us. You’ll find us on the Cilium Slack channel.
Further Reading
- Isovalent Enterprise for Cilium
- Azure and Isovalent main partner page
- Enabling Enterprise features in AKS with Isovalent Enterprise for Cilium
- Try Cilium using our interactive labs
Amit Gupta, Senior Technical Marketing Engineer