Blog

Tutorial: Deploying Red Hat OpenShift with Cilium

Dean Lewis

Published: November 16, 2023 | Updated: March 2, 2026 | Networking for Kubernetes

Cilium is available and supported across a number of Kubernetes platforms and offerings. One of the most popular is that of Red Hat OpenShift. We've previously covered why Cilium and Red Hat OpenShift are a match made in heaven, enhancing our clusters capabilities out of the box. If you want to learn how to deploy Red Hat OpenShift with Isovalent Networking for Kubernetes, powered by Cilium, follow this tutorial!

Why deploy Red Hat OpenShift with Cilium?

By introducing Cilium's eBPF powered networking, security and observability features to our Red Hat OpenShift clusters, application developers and Site Reliability Engineers gain access to granular application metrics and insights into the behavior of their applications, and SecOps teams can transparently encrypt traffic, ensuring a secure communication layer across multiple clusters and cloud environments. These are just a few examples of how bringing these two technologies together provide a robust infrastructure for managing applications within the OpenShift environment.

Isovalent Networking for Kubernetes has been available in the Red Hat Ecosystem Catalog since 2021, as well as being certified as a Certified OpenShift CNI Plug-in. The CNI badge is a specialization within Red Hat OpenShift certification for networking products that integrate with OpenShift using a CNI plugin. Users running OpenShift can feel confident that running Cilium will not negatively impact their Red Hat support experience.

On OpenShift, installation and upgrades are managed using the Cilium Lifecycle Operator, CLife, with Red Hat certified operator images, starting with Isovalent Networking for Kubernetes 1.17. The OpenShift certified version of Cilium is based on Red Hat Universal Base Images, and it passes the Operator certification requirements as well as the Kubernetes e2e tests.

Isovalent Networking for Kubernetes, powered by Cilium, is also supported and certified for deployment with OpenShift Virtualization and in Hosted Control Plane environments.

Who supports me when I use Cilium on OpenShift?

As Cilium has earned the Red Hat OpenShift Container Network Interface (CNI) certification by completing the operator certification and passing end-to-end testing, Red Hat will collaborate as needed with the ecosystem partner to troubleshoot the issue, as per their third-party software support statements.

For customers who have chosen Isovalent Networking for Kubernetes, the enterprise-grade Cilium, this ensures a complete support experience with Cilium by both vendors, working towards a common resolution.

How do I install Cilium on OpenShift?

Red Hat OpenShift Container Platform (OCP) offers three main flexible installation methods:

  • Installer provisioned infrastructure (IPI) – Deploy a cluster on provisioned infrastructure and the cluster it maintains.
  • User provisioned infrastructure (UPI) – Deploy a cluster on infrastructure that we prepare and maintain.
  • There is a third method, which is Agent-based, providing the flexbility of UPI, driven by the Assisted Installer (AI) tool.

In this tutorial, we will be performing an IPI installation deployed to a VMware vSphere platform, however the steps today to consume the Cilium Manifests during a cluster deployment also apply to a UPI installation.

I recommend becoming familar with the Red Hat OpenShift deployment documentation for our provided platform, as well as the Cilium Documentation.

We can check the support matrix for Red Hat OpenShift and Isovalent Networking for Kubernetes (Cilium) versions here.

You can watch the technical walkthrough of this blog post in action in the below recording.

Deploying Red Hat OpenShift with Cilium

Prerequisites

Before we get started with the installation itself, we will need to get the following pre-requsites in place.

  • Jump host to run the installation software from
    • We can download the openshift-install tool and oc tool from this Red Hat repository
    • Alternatively we can go to Red Hat Hybrid Cloud Console > Click OpenShift > Clusters > Create a cluster > Select our platform. This will give us download links to the software and our pull secret
  • A pull secret file/key from the Red Hat Cloud Console website
    • We can get one of these by just signing up for an account, any cluster created using this key will get a trial activation for a cluster for 60 days
  • Access to the DNS server which supports the infrastructure platform that we are deploying to
  • A SSH Key used for access to the deployed OpenShift nodes

Extract the software tools and place them in our user location. 

tar -zxvf openshift-client-linux-{version}.tar.gz
tar -zxvf openshift-install-linux-{version}.tar.gz

sudo cp openshift-install /usr/bin/local/openshift-install
sudo cp oc /usr/bin/local/oc
sudo cp kubectl /usr/bin/local/kubectl

Next, we need to download the VMware vCenter trusted root certificates and import them to our Jump Host.

curl -O https://{vCenter_FQDN}/certs/download.zip

Now unzip the file (we may need to install a software package for this sudo apt install unzip), and import them to the trusted store (Ubuntu uses the .crt files, hence importing the win folder).

unzip download.zip
cp certs/win/* /usr/local/share/ca-certificates
update-ca-certificates

We will need a user account to connect to vCenter with the correct permissions for the OpenShift-Install tool to deploy the vSphere cluster. If we do not want to use an existing account and permissions, we can use this PowerCLI script as an example to create the roles with the correct privileges based on the Red Hat documentation (at the time of writing).

Finally we will need a copy of the Cilium manifest files, which install the Clife Operator;

  • Isovalent Networking for Kubernetes release manifests
    • Go to the docs page (accessible for customers), navigate to the “Installing Isovalent Networking for Kubernetes on OpenShift” pages for the appropriate links.

DNS

For an OpenShift 4.20 IPI vSphere installation, we will need DNS and DHCP available for the cluster.

  • For the OpenShift 4.20 IPI installation, we need to configure two static DNS addresses. One for the cluster api access api.{clustername}.{basedomain} and one for cluster ingress access *.apps.{clustername}.{basedomain}.
    • In this tutorial I will be using the following:
      • Base Domain – veducate.lab
      • Cluster Name – ocp
      • Full example – api.ocp.veducate.lab
  • We will also need to create reverse lookup records for these addresses as well
  • These two addresses need to be part of the same subnet as our DHCP scope but excluded from DHCP.

Create the OpenShift Installation manifests

Now that we have our pre-reqs in place, we can start to deploy our cluster. When using the OpenShift-Install tool, we have three main command line options when creating a cluster:

openshift-install create cluster

  • This will run through a wizard to create the install-config.yaml file and then create the cluster automatically using terraform. Terraform is packaged as part of the installer software , meaning we don’t need terraform on our system as a pre-req.
  • If we run the below two commands listed, we can then still run this command to provision our cluster in an IPI method.
  • If we only use this command to provision a cluster, we will skip the necessary steps to bootstrap the cluster with the Cilium CNI.

openshift-install create install-config

  • This will run through a wizard to create the install-config.yaml file, and leave it in the root directory, or directory we’ve specify with the --dir={location} argument.
  • Modifications can be made to the install-config.yaml file, before running the above create cluster command.

openshift-install create manifests

  • This will create the manifests folder which controls the provisioning of the cluster. Most of the time this command is only used with UPI installations. When deploying Cilium, we are required to create this manifest folder and add the additional Cilium YAML files to this folder. This provides OpenShift the additional configuration files to bootstrap the cluster with Cilium.

First we will create the install-config.yaml, the easiest way to do this is by running the below command and answering the questions as part of the wizard :

$ ./openshift-install create install-config

? SSH Public Key /home/dean/.ssh/id_rsa.pub
? Platform vsphere
? vCenter vcsa.veducate.lab
? Username administrator@vsphere.local
? Password [? for help] **********
INFO Connecting to vCenter vcsa.veducate.lab      
INFO Defaulting to only available datacenter: Datacenter 
INFO Defaulting to only available cluster: /Datacenter/host/Cluster 
? Default Datastore /Datacenter/datastore/esxi-01-ds
? Network VLAN 2
? Virtual IP Address for API 192.168.200.220
? Virtual IP Address for Ingress 192.168.200.221
? Base Domain veducate.lab
? Cluster Name ocp
? Pull Secret [? for help] *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

INFO Install-Config created in: .

This will output a file called install-config.yaml, which contains all the infrastructure information that the tool will provision, in addition to the associated cloud provider, in this example, VMware vSphere.

The file contents will be similar to the below. (It is recommended to backup a copy of this file before continuing.)

Whichever deployment method used to create the file, we will need to edit this file to ensure that networkType is set to Cilium, and the Network Address CIDRs are configured as necessary for our environment.

additionalTrustBundlePolicy: Proxyonly
apiVersion: v1
baseDomain: veducate.lab
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform: {}
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: ocp
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.0.0.0/16
  networkType: Cilium
  serviceNetwork:
  - 172.30.0.0/16
platform:
  vsphere:
    apiVIPs:
    - 192.168.200.220
    failureDomains:
    - name: generated-failure-domain
      region: generated-region
      server: vcsa.veducate.lab
      topology:
        computeCluster: /Datacenter/host/Cluster
        datacenter: Datacenter
        datastore: /Datacenter/datastore/esxi-01-ds
        networks:
        - VLAN 2
        resourcePool: /Datacenter/host/Cluster/Resources
      zone: generated-zone
    ingressVIPs:
    - 192.168.200.221
    vcenters:
    - datacenters:
      - Datacenter
      password: vEducate-OCP!
      port: 443
      server: vcsa.veducate.lab
      user: administrator@vsphere.local
publish: External
pullSecret: '{"auths":{"cloud.openshift.com":.......}
sshKey: |
  ssh-rsa AAAAB3*******BAel0= dean@veducate

Now run the following command to create the OpenShift manifests file:

$ ./openshift-install create manifests

INFO Consuming Install Config from target directory 
I0302 12:47:18.358563   54166 session.go:227] "Created and cached vSphere client session" server="vcsa.veducate.lab" datacenter="" username="administrator@vsphere.local"
INFO Successfully populated MCS CA cert information: root-ca 2036-02-28T12:47:18Z 2026-03-02T12:47:18Z 
INFO Successfully populated MCS TLS cert information: root-ca 2036-02-28T12:47:18Z 2026-03-02T12:47:18Z 
INFO Adding clusters...                           
INFO Manifests created in: cluster-api, manifests and openshift

Next we need to copy across the Cilium manifests into the newly created manifests folder.

OpenShift will by default create a cluster-network-02-operator.yml file in the manifests folder. Within this file, it is expected that the fields networkType should be set to Cilium and all relevant clusterNetwork and serviceNetwork CIDRs should be defined, as per the configuration from the install-config.yaml file.

Because this example is using Kube-Proxy replacement, we need need to also configure OpenShift to not deploy Kube-Proxy as part of the bootstrap process by modifying the file cluster-network-02-config-local.yml to include spec.deployKubeProxy=false, as per the below example.

$ sed -i '/^spec:/a\  deployKubeProxy: false' manifests/cluster-network-02-config.yml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: null
  name: cluster
spec:
  deployKubeProxy: false
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkDiagnostics:
    mode: ""
    sourcePlacement:
      nodeSelector: null
      tolerations: null
    targetPlacement:
      nodeSelector: null
      tolerations: null
  networkType: Cilium
  serviceNetwork:
  - 172.30.0.0/16
status: {}

To configure the installation settings for Cilium, we can modify the ciliumconfig.yaml file. Here is an example of a simple CiliumConfig with Kube-Proxy replacement enabled (learn more about why replacing Kube-Proxy is neccessary in Kubernetes deployments) and we have defined the k8sServiceHost and k8sServicePort values, and finally the clusterPoolIPv4PodCIDRList address range matches that of the clusterNetwork address range configured in the OpenShift install:

apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  labels:
    app.kubernetes.io/name: clife
  name: ciliumconfig
spec:
  cluster:
    name: default
  securityContext:
    privileged: true
  ipam:
    mode: "cluster-pool"
    operator:
      clusterPoolIPv4PodCIDRList:
      - "10.128.0.0/14"
      clusterPoolIPv4MaskSize: 24
  cni:
    binPath: "/var/lib/cni/bin"
    confPath: "/var/run/multus/cni/net.d"
    exclusive: false
  prometheus:
    serviceMonitor:
      enabled: true
  hubble:
    enabled: true
  sessionAffinity: true
  # avoid conflicts
  clusterHealthPort: 9940
  # OVN Kubernetes default port, firewalls may already be configured for it
  tunnelPort: 4789
  kubeProxyReplacement: true
  k8sServiceHost: api.ocp.veducate.lab
  k8sServicePort: 6443

If you enable kube proxy replacement, as we are doing in this walkthrough, CLife and Cilium must be able to reach the Kubernetes API before the service layer is available. Set the API host and port in CLife manifests; apps_v1_deployment_clife-controller-manager.yaml and subscription.yaml.

$ KUBERNETES_SERVICE_HOST=api.ocp.veducate.lab
$ KUBERNETES_SERVICE_PORT=6443

$ yq -i '(.spec.template.spec.containers[] | select(has("name")) | select(.name == "manager")).env += [{"name": "KUBERNETES_SERVICE_HOST", "value": "'${KUBERNETES_SERVICE_HOST}'"}, {"name": "KUBERNETES_SERVICE_PORT", "value": "'${KUBERNETES_SERVICE_PORT}'"}]' apps_v1_deployment_clife-controller-manager.yaml

$ yq -i '.spec.config.env += [{"name": "KUBERNETES_SERVICE_HOST", "value": "'${KUBERNETES_SERVICE_HOST}'"}, {"name": "KUBERNETES_SERVICE_PORT", "value": "'${KUBERNETES_SERVICE_PORT}'"}]' subscription.yaml

How do I create the OpenShift Cluster with Cilium?

We are now ready to create the OpenShift Cluster and can proceed by running the command:

$ openshift-install create cluster

For more granular output we can use the following command argument for output to the terminal.

--log-level string   log level (e.g. "debug | info | warn | error") (default "info")

Full debug information is also contained in the hidden file .openshift_install.log in the location where the command is run from.

Below is an example output from creating the cluster with the informational log level set. 

INFO Consuming Openshift Manifests from target directory 
INFO Consuming Worker Machines from target directory 
INFO Consuming Common Manifests from target directory 
INFO Consuming OpenShift Install (Manifests) from target directory 
INFO Consuming Master Machines from target directory 
INFO Creating infrastructure resources...         
....
INFO Creating kubeconfig entry for capi cluster ocp-vx8gq-0 
INFO Waiting up to 15m0s (until 3:03PM GMT) for network infrastructure to become ready... 
INFO Network infrastructure is ready              
...
INFO Waiting up to 15m0s (until 3:03PM GMT) for machines [ocp-vx8gq-bootstrap ocp-vx8gq-master-0 ocp-vx8gq-master-1 ocp-vx8gq-master-2] to provision... 
INFO Control-plane machines are ready             
INFO Cluster API resources have been created. Waiting for cluster to become ready... 
INFO Consuming Cluster API Machine Manifests from target directory 
INFO Consuming Cluster API Manifests from target directory 
INFO Waiting up to 20m0s (until 3:15PM GMT) for the Kubernetes API at https://api.ocp.veducate.lab:6443... 
INFO API v1.33.5 up                               
INFO Waiting up to 1h0m0s (until 3:55PM GMT) for bootstrapping to complete...
INFO Destroying the bootstrap resources...        
INFO Waiting up to 40m0s (until 4:43PM) for the cluster at https://api.ocp.veducate.lab:6443 to initialize... 
INFO Checking to see if there is a route at openshift-console/console... 
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/dean/ocp/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp.veducate.lab 
INFO Login to the console with user: "kubeadmin", and password: "DyquU-FckQQ-CpN9g-7A57f"

How do I test Network Connectivity with Cilium?

For this guide the final piece is to run the standard Cilium tests on my cluster to confirm network connectivity.

We've run the provided export KUBECONFIG in the terminal output to connect to our cluster.

Before the Cilium network tests will run, due to the SecurityConstraintContext (SCC) that's implemented by OpenShift out of the box as a hardened security posture, we will need to apply a configuration to handle this. 

$ kubectl apply -f - <<EOF
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: cilium-test
allowHostPorts: true
allowHostNetwork: true
users:
  - system:serviceaccount:cilium-test:default
priority: null
readOnlyRootFilesystem: false
runAsUser:
  type: MustRunAsRange
seLinuxContext:
  type: MustRunAs
volumes: null
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostPID: false
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
allowedCapabilities: null
defaultAddCapabilities: null
requiredDropCapabilities: null
groups: null
EOF

Now we can proceed to configure the tests to run, first create a namespace for the test to run:

$ kubectl create ns cilium-test

Deploy the checks with the command:

$ kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/1.18.6/examples/kubernetes/connectivity-check/connectivity-check.yaml

This will configure a series of deployments that will use various connectivity paths to connect to each other. Connectivity paths include with and without service load-balancing and various network policy combinations. The pod name indicates the connectivity variant and the readiness and liveness gate indicates success or failure of the test:

$ kubectl get pods -n cilium-test
NAME                                                    READY   STATUS    RESTARTS   AGE
echo-a-568cb98744-tlvwv                                 1/1     Running   0          67s
echo-b-64db4dfd5d-q8kpj                                 1/1     Running   0          67s
echo-b-host-6b7bb88666-qnhz4                            1/1     Running   0          67s
host-to-b-multi-node-clusterip-6cfc94d779-5v2x7         1/1     Running   0          66s
host-to-b-multi-node-headless-5458c6bff-2v7m7           1/1     Running   0          66s
pod-to-a-allowed-cnp-55cb67b5c5-ltclc                   1/1     Running   0          66s
pod-to-a-c9b8bf6f7-z4k2h                                1/1     Running   0          66s
pod-to-a-denied-cnp-85fb9df657-ndg2n                    1/1     Running   0          66s
pod-to-b-intra-node-nodeport-55784cc5c9-t42kj           1/1     Running   0          66s
pod-to-b-multi-node-clusterip-5c46dd6677-jgzvf          1/1     Running   0          66s
pod-to-b-multi-node-headless-748dfc6fd7-2ggvq           1/1     Running   0          66s
pod-to-b-multi-node-nodeport-f6464499f-84t92            1/1     Running   0          66s
pod-to-external-1111-96c489555-srcvb                    1/1     Running   0          66s
pod-to-external-fqdn-allow-google-cnp-5f747dfc7-8jsxw   1/1     Running   0          66s

Note: If you deploy the connectivity check to a single node cluster, pods that check multi-node functionalities will remain in the Pending state. This is expected since these pods need at least 2 nodes to be scheduled successfully.

Can I migrate from OpenShift Networking to Cilium?

Yes, it is possible to move an existing OpenShift cluster from the built in networking stack, such as OpenShift SDN or OVN Kubernetes, to Cilium. In practice, this is not a simple in place toggle, it is an operational change that touches cluster networking, node configuration, and workload rollout behavior. It needs a plan, a maintenance window, and clear rollback criteria.

Isovalent supports customers who need to make this move. We can help you assess feasibility, design a migration runbook for your OpenShift version and platform, and rehearse the procedure in a non production environment before you touch production. If this is a scenario you are facing, contact us and we will bring the right engineers into the conversation.

Do you have any questions?

Summary

This tutorial takes you as far as the initial deployment of Red Hat OpenShift and Cilium including network connectivity testing. To continue to run this cluster further for development or production use-cases, I recommend continuing to follow the official Red Hat OpenShift documentation covering post installation cluster tasks.

As you can see, it's quick and easy to install Cilium into our Red Hat OpenShift environment, with minimal additional configuration from our existing workflows.

With Cilium installed in our cluster, we can now take advantage of features and use-cases such as:

Why not jump into our hands-on-Labs which take you through the various features of Cilium, Hubble and Tetragon. We recommend get started by diving into our three popular Isovalent Enterprise for Cilium labs covering Zero Trust Networking, Security Observability and Connectivity Visibility.

Isovalent Enterprise for Cilium: Zero Trust Visibility

Creating the right Network Policies can be difficult. In this lab, you will use Hubble metrics to build a Network Policy Verdict dashboard in Grafana showing which flows need to be allowed in your policy approach.

Isovalent Enterprise for Cilium: Security Visibility

Learn how to imulate the exploitation of a nodejs application, with a reverse shell inside of a container and monitor the lateral movement within the Kubernetes environment.

Isovalent Enterprise for Cilium: Connectivity Visibility with Hubble

Learn how Hubble Flow events provide metadata-aware, DNS-aware, and API-aware visibility for network connectivity within a Kubernetes environment using Hubble CLI, Hubble UI and Hubble Timescape, which provides historical data for troubleshooting.

/authors/dean-lewis

Dean Lewis, Senior Technical Marketing Engineer