Kubernetes doesn’t provide a network interface system by default. Instead, network plugins offer this functionality, with Cilium leading the way. Red Hat OpenShift suggests several alternate CNI plugins. Red Hat OpenShift has an extensive certification program to ensure CNI’s work correctly with their enterprise platform, allowing for the deployment of 3rd Party Kubernetes CNI's such as Cilium. Isovalent Enterprise for Cilium meets all these requirements and has become the de facto standard for customers looking to run Kubernetes in production. A highly requested feature from our customers using Red Hat OpenShift Service on AWS (ROSA) is the ability to choose a CNI that covers the three pillars of Networking, Security, and Observability. Today, we are happy to announce that Isovalent Enterprise for Cilium is fully supported as a CNI for ROSA. This blog shows how to deploy a Red Hat OpenShift Service on an AWS (ROSA) cluster without a preinstalled CNI plugin and then add Isovalent Enterprise for Cilium as the CNI plugin. We also add the Red Hat OpenShift Virtualisation Operator and deploy VM's on the cluster. Note- The blog has been validated on ROSA/HCP 4.17 and Isovalent Enterprise for Cilium 1.15. What is ROSA? ROSA is a fully-managed Red Hat OpenShift Service running natively on Amazon Web Services (AWS), providing a Kubernetes-based turnkey application platform. ROSA allows organizations to increase operational efficiency, refocus on innovation, and quickly build, deploy, and scale applications. With ROSA, everything you need to deploy and manage applications is bundled, including container management, automation (Operators), monitoring, and more, all backed by expert Red Hat site reliability engineers (SREs). ROSA provides you with benefits, including: Accelerate time to value– Focus on building and scaling applications that add value to the business Focus on Innovation– Simplify operations so your teams can refocus on innovation, not managing infrastructure Optimize Investment– Take advantage of current cloud investments and entitlements with AWS Hybrid cloud flexibility– Get consistent Red Hat OpenShift experience across any environment: public cloud, private cloud, edge What are the types of cluster topologies in ROSA? ROSA has the following cluster topologies: Hosted control plane (HCP) – The control plane is hosted within Red Hat’s AWS account and managed by Red Hat. Worker nodes are deployed in the customer’s AWS account. Classic – The control plane and worker nodes are deployed in the customer’s AWS account. ROSA with HCP offers a more efficient control plane architecture that helps reduce the AWS infrastructure fees incurred when running ROSA and allows for faster cluster creation times. ROSA with HCP and ROSA Classic can be enabled in the ROSA console. You can select which architecture you want to use when provisioning ROSA clusters using the ROSA CLI. In this blog, we will be talking about the Hosted Control Plane. What is Isovalent Enterprise for Cilium? Isovalent Enterprise for Cilium is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns. Why Isovalent Enterprise for Cilium for ROSA? Red Hat OpenShift Service on AWS (ROSA) offers a seamless hybrid cloud experience by combining Red Hat OpenShift's enterprise Kubernetes capabilities with AWS's infrastructure. Isovalent Enterprise for Cilium elevates this integration by addressing critical network, security, and observability challenges, ensuring ROSA users maximize the value of their clusters. Enhanced Networking for ROSA Workloads– ROSA provides the foundational compute stack necessary for application deployment. Still, Isovalent Enterprise for Cilium takes it further by leveraging eBPF-powered Cilium to optimize network performance. This ensures low-latency communication and efficient routing, even in high-throughput or latency-sensitive applications. Simplified Multi-Cluster Networking with ROSA– Unified communication across clusters becomes essential as organizations scale their ROSA environments. Cilium’s Cluster Mesh simplifies inter-cluster communication, enabling seamless pod IP routing and service discovery across multiple ROSA and non-ROSA Kubernetes clusters. This capability directly addresses the complexities of hybrid and multi-cloud deployments. Granular Network Security and Policy Enforcement– ROSA’s native capabilities are augmented by Cilium’s fine-grained, application-aware network policies. Security teams can define policies based on namespaces, labels, and application-based rules, such as HTTP API paths. Cilium’s DNS-aware and L7 policies enhance compliance for ROSA clusters deployed in regulated environments by securing communication with only approved external services. Comprehensive Observability Tailored for ROSA– Isovalent Enterprise provides ROSA users with unparalleled observability through tools like Hubble. Users gain detailed flow visibility into pod-to-pod and pod-to-service traffic, helping identify bottlenecks or unauthorized access. This observability extends to L7 traffic, ensuring ROSA applications meet performance and security benchmarks. Native AWS Integration with ROSA– Cilium’s capabilities, such as static egress gateways and load balancer support, enhance ROSA’s close integration with AWS services. These facilitate the connection of ROSA workloads to legacy environments or complex AWS network topologies, ensuring compatibility and operational efficiency in hybrid setups. Sidecar-Less Service Mesh for ROSA Applications– Traditional service meshes add operational complexity and performance overhead. Cilium’s sidecar-less service mesh approach allows ROSA users to leverage features like traffic management, observability, and security without additional costs or resource requirements. Certified for Red Hat OpenShift and supported by Red Hat and AWS– Since Isovalent Enterprise for Cilium is a certified Red Hat OpenShift CNI plugin, ROSA users can confidently adopt it without concerns about supportability or compatibility. It integrates seamlessly with ROSA’s managed services, ensuring reliability backed by joint Red Hat and AWS support. By adopting Isovalent Enterprise for Cilium, ROSA users can enhance their clusters, address critical networking, security, and observability challenges, and maintain operational simplicity. What support do you get when you use ROSA & Isovalent Enterprise for Cilium? Cilium is recognized as a Certified OpenShift CNI Plug-in by completing the operator certification and passing end-to-end testing. As such, Red Hat will collaborate with Isovalent to troubleshoot the issue, per their third-party software support statements. Customers who have chosen Isovalent Enterprise for Cilium can benefit from both vendors' complete support experience with Cilium, working towards a common resolution. Creating a ROSA with an HCP cluster without a CNI plugin Prerequisites Ensure that you have completed the AWS prerequisites. Access to AWS. Create a new account for free. Enabled the ROSA service in the AWS Console. If this is not enabled, the --no-cni flag will not be available when creating the ROSA cluster. Log in to your Red Hat account by using the ROSA CLI. Available AWS service quotas. Configured the latest ROSA CLI (rosa) on your installation host. Ensure that you have a configured virtual private cloud (VPC). We will use an OpenShift Terraform repo to configure a VPC for the ROSA HCP cluster. Install kubectl Install Helm Install Cilium CLI: Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other operating systems on your local machine(s) or server(s). Basic checks Log into your Red Hat account by using the ROSA CLI. Verify your user details. Verify the AWS quotas for creating a ROSA cluster. Verify the ROSA Openshift Client Creating an OpenID Connect configuration When using a ROSA cluster, you can create the OpenID Connect (OIDC) configuration before creating your cluster. This configuration is registered to be used with Red Hat OpenShift Cluster Manager. Create your OIDC configuration alongside the AWS resources. When creating your cluster, you must supply the OIDC config ID. The CLI output provides this value for --mode auto; otherwise, you must determine these values based on the AWS CLI output for --mode manual. You can list the possible OIDC configurations available for the clusters associated with your user organization. Save the OIDC configuration ID as a variable to use later. Creating the account-wide STS roles and policies Before using the ROSA CLI to create ROSA clusters with HCP, the required account-wide roles and policies, including the Operator policies, must be created. If the required account-wide STS roles are not in your AWS account, create them and attach the policies. Set your prefix as an environmental variable. Creating Operator roles and policies When using a ROSA with an HCP cluster, you must create the Operator IAM roles required for ROSA with HCP deployments. The cluster Operators use the Operator roles to obtain the temporary permissions required to carry out cluster operations, such as managing back-end storage, cloud provider credentials, and external access to a cluster. The Operator roles are now created and ready to use to build your ROSA cluster with HCP. Creating the AWS VPC For this blog, we will use Terraform to create the VPC in which the ROSA cluster will be created. However, this is optional, and you can also create your VPC in multiple ways (Using Ansible, AWS CloudFormation, AWS portal, AWS CLI, etc). Clone the openshift-cs Terraform repository. Set your region & cluster name as environmental variables. Create a terraform.tfvars file that contains all the variables to create a VPC for the ROSA cluster. Create the VPC using Terraform using the following commands. Once the VPC has been created, Cilium firewall prerequisites must be added to the default worker security group as Inbound rules. These are needed for Cilium-specific services (Hubble Relay, Cilium operator, Cilium agent, Spire agent, Mutual Authentication, Hubble Relay, Hubble Server, Cluster health checks) to work seamlessly. Creating the cluster When using the ROSA command line interface (CLI), rosa, to create a cluster, you can add an optional flag --no-cni to create a cluster without a CNI plugin. Set the following as environmental variables. Brief details about the values being passed while creating the ROSA cluster. Create the ROSA cluster. Check the status of your cluster by running the following command. Login to the ROSA cluster You can log in to the ROSA cluster using ROSA CLI. Although ROSA with HCP cluster installation is complete, the cluster cannot operate without a CNI plugin. Because the nodes are not ready, the workloads cannot be deployed. For example, the ROSA cluster web console is unavailable, so you must use the Red Hat OpenShift CLI (oc) to log in to the cluster. Other Red Hat OpenShift components, such as the HAProxy-based Ingress Controller, image registry, and Prometheus-based monitoring stack, are not running. This is expected behavior until you install a CNI provider. Install Isovalent as the CNI Install Isovalent as the CNI. Create a namespace for installing Cilium. Create a yaml file with the desired Helm values for Cilium deployment. Portmap It is used to support nodeports without KPR. Support for KPR is coming soon, and the blog post will be updated. With the ROSA cluster up and running, you can install Isovalent Enterprise for Cilium via Helm values, and access to Enterprise Helm charts can be obtained by contacting sales or support@isovalent.com Once Isovalent Enterprise for Cilium is installed, check the status of the nodes and pods. Isovalent status checks Check Cilium status & validate Cilium version Validate health check. cilium-health It is a tool available in Cilium that provides visibility into the overall health of the cluster’s networking and connectivity. You can check node-to-node health with cilium-health status. Cilium Connectivity Test The Cilium connectivity test deploys a series of services and deployments, and CiliumNetworkPolicy will use various connectivity paths to connect. Connectivity paths include those with and without service load-balancing and various network policy combinations. Red Hat OpenShift Virtualization on ROSA Red Hat OpenShift Virtualization is a feature of Red Hat OpenShift that allows you to run Virtual Machines (VMs) alongside your containers. This comes in very handy: while onboarding legacy applications that cannot be run on containers or require specific hardware/software. while doing a phased migration from VM to container-based workloads, sharing the same network, storage, and management tooling. In this section, we will deploy the Red Hat OpenShift Virtualisation Operator and create a VM for a Red Hat OpenShift Service AWS (ROSA) cluster. Installing the OpenShift Virtualization Operator Follow the procedure described to add bare-metal worker nodes (that support nested virtualization). Browse to the OpenShift Cluster Manager and select your cluster Click > “Machine Pools” tab. Click on the “Add Machine Pool” button, give it a name, and select a Bare Metal instance type such as m5d.metal. Set replicas to 1 and click Add machine pool. Click > Open Console button to open the OpenShift Console Click > Operators > Operator Hub Search for OpenShift Virtualization and Click > OpenShift Virtualization Accept all the defaults and click > Install Once installed, it should prompt you to create a hyperconverged instance. Click > Create HyperConverged. Accept all the defaults. Once the operator is deployed, refresh your browser. Optional- From the CLI you can see that a new namespace openshift-cnv is created with the respective pods. This can also be viewed from the console. Select Projects > openshift-cnv > Workloads > Pods A new Virtualization menu item will be available in the left-hand navigation. Click > Catalog > Template catalog Validating the Red Hat OpenShift Virtualization Deployment Let's create a VM using the newly deployed OpenShift Virtualization Operator. For this validation, we have chosen CirrOS to create the Virtual Machine. Create a boot source for CirrOS. Download the latest version of CirrOS on your machine/laptop. Once you have the image, you can create a data volume. Data Volumes(DV) are simply an abstraction on top of a Persistent Volume Claim (PVC). The DV will monitor and orchestrate the import/upload/clone of the data into the PVC. Use the virtctl CLI to upload the image and create a volume. Create a bootable volume from the data volume (PVC). Click > Virtualization > Bootable volumes > Add volume Select ‘Add Volume’ and ‘Use existing volume’ for the source type. Select the PVC that we just uploaded (cirros), and the existing StorageClass. The CirrOS image is only 20 MB, so set the disk size to 200 MiB just to be sure it will provision. You’ll find an existing ‘Volume metadata’ preference for CirrOS, and can then specify a default instance type. Choose the Red Hat Provided -> U series -> u1.nano type. Select ‘Save’ and the bootable volume will be created. Now that we have a bootable volume, we can start creating a CirrOS Virtual Machine. Click > Virtual Machines > Create Virtual Machine and select From template. Select CirrOS as the Volume. The instance details will be auto populated. Click > Customize VirtualMachine Click > Scripts tab > Click > Edit > Public SSH Click > Add new and copy your public SSH key. Check Automatically apply this key to any new Virtual machine you create in this project. Click > Save Click > Create Virtual Machine. The Virtual Machine will go through the provisioning, starting and running phases. You can also test that SSH key injection has worked using the virtctl CLI. Navigate back to the Virtual Machine overview, select the ‘Details’ tab, and scroll down to see the virtctl command to log in to the workstation: Optional - You can also see that the Virtual Machine is running under a pod that has been created by the virt-launcher as soon as the Virtual Machine is created. Conclusion Hopefully, this post gave you a good overview of deploying a Red Hat OpenShift Service on an AWS (ROSA) cluster without a preinstalled CNI plugin and then adding Isovalent Enterprise for Cilium as the CNI plugin. We also added the Red Hat OpenShift Virtualisation Operator and deployed VM's on the cluster. You can schedule a demo with our experts if you'd like to learn more. Try it out Choose your way to explore Cilium with our Cilium learning tracks focusing on features critical to engineers using Cilium in cloud environments. Cilium comes in different flavors, whether using GKE, AKS, Red Hat OpenShift, or EKS; not all of these features will apply to each managed Kubernetes Service. However, it should give you a good idea of features relevant to operating Cilium in cloud environments. Suggested reading What is ROSA ROSA with HCP without a CNI plugin Redhat support for 3rd Party Components 3rd Party CNI in ROSA Setting up your environment for ROSA with AWS STS ROSA AWS STS pre-requisites Supercharging OpenShift with Cilium and eBPF Isovalent in the Redhat Catalogue ROSA FAQ