Tutorial: Tips and Tricks to install Cilium

Jan 16, 2023Cilium

Over 14,000 GitHub stars! Cilium’s popularity keeps on growing. If you’re new to the world of Cilium – like I was myself a year ago – the first thing you’ll want to do is install it. What I’ve discovered over the past 12 months is that there are many different ways to deploy Cilium! Whether you’re a newcomer to Cilium or an experienced user, I hope you’ll learn in this post some valuable tips and tricks to help you on your Cilium journey.

A privilege of my job is that I get to work closely with the engineers that are building and improving Cilium on a day-to-day basis. And when they release a new feature, well, I often want to be amongst the first ones to test it out.

Which means I often have to install the very latest Cilium version: not necessarily a mainline version (i.e 1.12.0), or even a release candidate (i.e 1.13.0-RC1). Release candidates are released on a regular basis in the run-up to the main release, to let users test some of the new features. Sometimes I have to test and deploy a Cilium version that has only just been pushed, either because it includes a bug fix or because it has a feature I really want to play with.

I learned a few things along the way and wanted to share some of my learnings on this post. Let’s go through the options at our disposal.

The “Easy” Way

Of course, you should always start with the official documentation when installing Cilium and this post doesn’t intend to replace the official docs: it’s rather a collection of tips and tricks.

The docs focus primarily on the two most commonly used installation tools for Cilium: cilium-cli and helm. We will cover both tools in this post.

Helm is the popular Kubernetes package manager, while cilium-cli is a purpose-built tool to install and manage Cilium. We’ll start with cilium-cli first. You will see how we can use either tool, or even both of them together.

I would recommend most first-time users to install Cilium with the cilium-cli tool and its command cilium install. It would install the latest stable release by default:

nicovibert:~$ cilium install
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.12.0"
ℹ️  using Cilium version "v1.11.3"
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected IPAM mode: kubernetes
🔮 Auto-detected datapath mode: tunnel
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=kind-kind,encryption.nodeEncryption=false,...
ℹ️  Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.11.3...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health

Some configuration options can be specified with cilium install. For example, cilium install --encryption wireguard can be used to enable transparent encryption with WireGuard, as described in this tutorial and this lab.

But if you want to fully customize your installation, you should use cilium-cli with helm-set (to specify configuration values on the command line based on Helm values) or helm-values to pull these values out of a YAML file. For example, this is what we use to install Cilium with BGP enabled in our BGP lab:

cilium install \
    --helm-set ipam.mode=kubernetes \
    --helm-set tunnel=disabled \
    --helm-set ipv4NativeRoutingCIDR="10.0.0.0/8" \
    --helm-set bgpControlPlane.enabled=true \
    --helm-set k8s.requireIPv4PodCIDR=true

You can see in the code snippet above that we are enabling features that are disabled by default (like BGP Control Plane).

The “Specific Release” Way

The examples shown so far install the latest stable release but the cilium-cli also lets you install a specific Cilium release. You can even check which Cilium version the cilium-cli supports with this command:

nicovibert:~$ cilium install --list-versions                                                                     
v1.13.0-rc4
v1.13.0-rc3
v1.13.0-rc2
v1.13.0-rc1
v1.13.0-rc0
v1.12.5 (default)
v1.12.4
v1.12.3
v1.12.2
[edited for brevity - there were about 120 versions!]
v1.6.10
v1.6.9
v1.6.8
v1.6.7
v1.6.6
v1.6.5
v1.9-dev
v1.8-dev
v1.7-dev
v1.6-dev

By default, Cilium will install the stable release (at time of writing, v1.12.5). I can specify another version from that list, for example v1.11.1:

root@server:~# cilium install --version=v1.11.1
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.17.0"
ℹ️  Using Cilium version 1.11.1
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected datapath mode: tunnel
🔮 Auto-detected kube-proxy has been installed
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.1 --set cluster.id=0,cluster.name=kind-kind,encryption.nodeEncryption=false,ipam.mode=kubernetes,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️  Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.11.1...
🚀 Creating Agent DaemonSet...
level=warning msg="spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use \"kubernetes.io/os\" instead" subsys=klog
level=warning msg="spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the \"priorityClassName\" field instead" subsys=klog
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health

You will notice that cilium-cli auto-detected the type of environment (kind in the example above) but it also works with AKS, EKS, etc… Cilium is indeed deployed differently depending on the type of underlying platform it’s built upon.

Detecting the environment is certainly a great benefit of cilium-cli and is useful even if you would rather use Helm: as you can see above, cilium-cli uses helm template in its backend to generate the manifests. If you look at the terminal snippets above and below, cilium-cli even gives the user the helm template command to use if they just want to use cilium-cli to auto-detect the right flags for their clusters.

In other words – you can use cilium-cli for its ease, helm because it’s commonly used for app deployment, or finally you can use both, by using cilium-cli to auto-detect the environment values and to help you set the adequate Helm values.

In addition, cilium-cli also includes a way to auto-generate Helm values in a file, with --helm-auto-gen-values. This flag effectively gives you a dry-run option: it generates the values, but doesn’t actually install it (you can refer to these values when deploying Cilium with Helm later on):

root@server:~# cilium install --version=v1.11.1 --encryption wireguard --helm-auto-gen-values helm-values.yaml
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.17.0"
ℹ️  Using Cilium version 1.11.1
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected datapath mode: tunnel
🔮 Auto-detected kube-proxy has been installed
ℹ️  L7 proxy disabled due to Wireguard encryption
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.1 --set cluster.id=0,cluster.name=kind-kind,encryption.enabled=true,encryption.nodeEncryption=false,encryption.type=wireguard,ipam.mode=kubernetes,kubeProxyReplacement=disabled,l7Proxy=false,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️  Generated helm values file "helm-values.yaml" successfully written


root@server:~# cat helm-values.yaml 
cluster:
  id: 0
  name: kind-kind
encryption:
  enabled: true
  nodeEncryption: false
  type: wireguard
ipam:
  mode: kubernetes
kubeProxyReplacement: disabled
l7Proxy: false
operator:
  replicas: 1
serviceAccounts:
  cilium:
    name: cilium
  operator:
    name: cilium-operator
tunnel: vxlan

I can then install Cilium with Helm by referring to the Helm value file created by cilium-cli:

root@server:~# cilium install --version=v1.11.1 --encryption wireguard --helm-values helm-values.yaml 
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.17.0"
ℹ️  Using Cilium version 1.11.1
🔮 Auto-detected cluster name: kind-clab-bgp-cplane-devel
🔮 Auto-detected datapath mode: tunnel
🔮 Auto-detected kube-proxy has been installed
ℹ️  L7 proxy disabled due to Wireguard encryption
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.1 --set cluster.id=0,cluster.name=kind-clab-bgp-cplane-devel,encryption.enabled=true,encryption.nodeEncryption=false,encryption.type=wireguard,ipam.mode=kubernetes,kubeProxyReplacement=disabled,l7Proxy=false,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️  Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.11.1...
🚀 Creating Agent DaemonSet...
level=warning msg="spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use \"kubernetes.io/os\" instead" subsys=klog
level=warning msg="spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the \"priorityClassName\" field instead" subsys=klog
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health

Here is another example where I install Cilium by referring to them in the command-line instead of in a separate YAML file. This is what I used to install one of the recent release candidates version to test SCTP support on Cilium.

helm repo add cilium https://helm.cilium.io/
helm upgrade --install cilium cilium/cilium --version 1.13.0-rc3 \
    --namespace kube-system \
    --set sctp.enabled=true \
    --set hubble.enabled=true \
    --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}" \
    --set hubble.relay.enabled=true \
    --set hubble.ui.enabled=true \
    --set hubble.ui.service.type=NodePort \
    --set hubble.relay.service.type=NodePort

As you can see, I use helm instead of cilium-cli to install the specific version. Note I specify the version 1.13.0-rc3 (SCTP support was introduced in RC1) and I also enabled SCTP and other additional features that are disabled by default, such as Hubble.

The “CI Build Image” Way

Where it gets more complex is working out how to install a version of Cilium that includes a particular fix or a new feature, but that is not yet available as an official release.

For example, I wanted to test a Gateway API feature that lets us load-balance traffic across backends. During my tests, I identified a bug which was quickly resolved by Tam, one of our rock-star developers working on Cilium. Once Tam got his PR merged, I was able to use the images generated during the CI process and published on the container image repository quay.io (with the Cilium images published on quay.io/cilium).

First, I had to get the details of the images generated during the build process:

What I copied at the end of this short video was this image location and its digest: quay.io/cilium/cilium-ci:db5b10436fa730de83275655d8f00e2e64db5e45.

To install this version, all I need is to reference these specific images for both the Cilium agent and the Cilium operator. Note that in my case, I also had to make sure the Helm chart was correct, which is why I had to clone the repo locally to use the latest Helm chart:

git clone https://github.com/cilium/cilium.git
cd cilium/
echo "Installing Cilium ....."
helm upgrade -i cilium ./install/kubernetes/cilium \
    --wait \
    --namespace kube-system \
    --set kubeProxyReplacement=strict \
    --set nodeinit.enabled=true \
    --set ipam.mode=kubernetes \
    --set image.repository=quay.io/cilium/cilium-ci \
    --set image.tag=db5b10436fa730de83275655d8f00e2e64db5e45 \
    --set image.pullPolicy=IfNotPresent \
    --set image.useDigest=false \
    --set operator.image.repository=quay.io/cilium/operator \
    --set operator.image.suffix=-ci \
    --set operator.image.tag=db5b10436fa730de83275655d8f00e2e64db5e45 \
    --set operator.image.pullPolicy=IfNotPresent \
    --set operator.image.useDigest=false \
    --set securityContext.privileged=true \
    --set gatewayAPI.enabled=true

The end result was a demo I posted on LinkedIn – you can also see it below:

Using cilium-cli and the helm-set flags, we can apply the same configuration with cilium install instead:

root@server:~# cilium install --helm-set=image.repository=quay.io/cilium/cilium-ci --helm-set=image.useDigest=false --helm-set=image.tag=db5b10436fa730de83275655d8f00e2e64db5e45 --helm-set=operator.image.useDigest=false --helm-set=operator.image.repository=quay.io/cilium/operator --helm-set=operator.image.suffix=-ci --helm-set=operator.image.tag=db5b10436fa730de83275655d8f00e2e64db5e45
🔮 Auto-detected Kubernetes kind: kind
✨ Running "kind" validation checks
✅ Detected kind version "0.17.0"
ℹ️  Using Cilium version 1.12.2
🔮 Auto-detected cluster name: kind-kind
🔮 Auto-detected datapath mode: tunnel
🔮 Auto-detected kube-proxy has been installed
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.12.2 --set cluster.id=0,cluster.name=kind-kind,encryption.nodeEncryption=false,image.repository=quay.io/cilium/cilium-ci,image.tag=db5b10436fa730de83275655d8f00e2e64db5e45,image.useDigest=false,ipam.mode=kubernetes,kubeProxyReplacement=disabled,operator.image.repository=quay.io/cilium/operator,operator.image.suffix=-ci,operator.image.tag=db5b10436fa730de83275655d8f00e2e64db5e45,operator.image.useDigest=false,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
ℹ️  Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.12.2...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
✅ Cilium was successfully installed! Run 'cilium status' to view installation health
root@server:~# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:       cilium             Running: 3
                  cilium-operator    Running: 1
Cluster Pods:     4/4 managed by Cilium
Image versions    cilium-operator    quay.io/cilium/operator-generic-ci:db5b10436fa730de83275655d8f00e2e64db5e45: 1
                  cilium             quay.io/cilium/cilium-ci:db5b10436fa730de83275655d8f00e2e64db5e45: 3

You don’t always have to use the Helm flag options with the cilium-cli: you can instead use some of the built-in options. For example, instead of using the --helm-set=operator.image, we can use the simpler --operator-image flag instead. Check out the full list of options with cilium install -h.

nicovibert:~$ cilium install -h
Install Cilium in a Kubernetes cluster

[edited for brevity]
Flags:
      --agent-image string                   Image path to use for Cilium agent
      --api-versions strings                 Kubernetes API versions to use for helm's Capabilities.APIVersions in case discovery fails
      --azure-client-id string               Client (application) ID of Azure Service Principal to use for installing Cilium (will create one if none provided)
      --azure-client-secret string           Client secret of Azure Service Principal to use for installing Cilium (will create one if none provided)
      --azure-resource-group string          Azure resource group name the cluster is in (required)
      --azure-subscription az account show   Azure subscription name the cluster is in (default az account show)
      --azure-tenant-id string               Tenant ID of Azure Service Principal to use for installing Cilium (will create one if none provided)
      --chart-directory string               Helm chart directory
      --cilium-ready-timeout duration        Timeout for Cilium to become ready before restarting unmanaged pods (default 5m0s)
      --config strings                       Set ConfigMap entries { key=value[,key=value,..] }
      --datapath-mode string                 Datapath mode to use { tunnel | aws-eni | gke | azure | aks-byocni } (default: autodetected).
      --disable-check strings                Disable a particular validation check
      --encryption string                    Enable encryption of all workloads traffic { disabled | ipsec | wireguard } (default "disabled")
      --helm-auto-gen-values string          Write an auto-generated helm values into this file
      --helm-set stringArray                 Set helm values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2)
      --helm-set-file stringArray            Set helm values from respective files specified via the command line (can specify multiple or separate values with commas: key1=path1,key2=path2)
      --helm-set-string stringArray          Set helm STRING values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2)
      --helm-values strings                  Specify helm values in a YAML file or a URL (can specify multiple)
      --helm-values-secret-name string       Secret name to store the auto-generated helm values file. The namespace is the same as where Cilium will be installed (default "cilium-cli-helm-values")
  -h, --help                                 help for install
      --image-suffix string                  Set all generated images with this suffix
      --image-tag string                     Set all images with this tag
      --inherit-ca string                    Inherit/import CA from another cluster
      --k8s-version string                   Kubernetes server version in case auto-detection fails
      --list-versions                        List all the available versions without actually installing
      --node-encryption                      Enable encryption of all node to node traffic
      --nodes-without-cilium strings         List of node names on which Cilium will not be installed
      --operator-image string                Image path to use for Cilium operator
      --restart-unmanaged-pods               Restart pods which are not being managed by Cilium (default true)
      --rollback                             Roll back installed resources on failure (default true)
      --version string                       Cilium version to install (default "v1.12.2")
      --wait                                 Wait for status to report success (no errors) (default true)
      --wait-duration duration               Maximum time to wait for status (default 5m0s)

[edited for brevity]

The “Latest” Way

If you don’t want a specific image build but rather the very latest version, you can simply use the -version=latest. Here is a quick demo:

That’s a really useful command and one I wish I had discovered earlier!

There is another obvious method to run the latest Cilium version: build it from the source code.

The “Compiled from Source” Way

Finally, let’s cover what might look like the most intimidating method on paper: compiling and installing Cilium from source. It turned out to be easier than expected, with a little help.

As soon as the most recent BGP feature was released (covered at length in this eCHO episode), I wanted to try it out for myself. I leveraged some scripts from the Cilium repo to simplify it. Once I had installed Go and Make on my machine, I was able to successfully deploy the latest version of Cilium on a Kind cluster, using this very handy script:

git clone https://github.com/cilium/cilium.git
source ./kind-shell-helpers.sh && cilium-build-and-deploy latest

It only took 5 minutes from start to finish, as you can see in the video below:

Summary

In this tutorial, I shared some tips and tricks I learned over the course of the past year deploying many different Cilium versions across different environments for very different use cases. I shared a few different options on how Cilium can be installed and why you might want to use a method over another or why you might use them together!

If you’d like to learn more, we let use different installation methods in many of the Isovalent labs. Check out the Getting Started with Cilium lab, the Cluster Mesh lab, the Transparent Encryption lab or the Egress Gateway lab.

I hope you found this blog post helpful. Thanks for reading.

Learn More

Nico Vibert
AuthorNico VibertSenior Technical Marketing Engineer