Kubernetes VPC on OpenShift with Isovalent Networking for Virtualization

The Kubernetes flat networking model works when every workload is a Pod, but what happens when you introduce Virtual Machines into the mix?

Platform teams are not just running containers anymore. OpenShift Virtualization has now made Virtual Machines a first class citizen on Kubernetes, and these VM based workloads are landing on the same cluster as container-based microservices.

The moment VMs get onboarded next to containers, a challenge shows up that namespaces and standard Kubernetes network policy do not solve on their own: two tenants want to use the same IP address.

In this post we explore how Isovalent Networking for Virtualization (INV) solves that with Isovalent Private Networks, and why they are best understood as the Kubernetes VPC. In this post, we’ll focus on two demo builds using isolated private networks with overlapping CIDRs on an OpenShift Virtualization cluster, running a mix of VMs and containers inside them, and then look at how to apply micro-segmentation between the tiers. All of it using identity-aware eBPF enforcement.

The problem with a flat network

When you run a VM on Kubernetes, that VM lives inside a Pod. The network sees it as just another Pod, so it inherits the standard Kubernetes flat network design. Every workload can reach every other workload, and every address has to be unique across the whole cluster.

For a lot of setups that is completely fine. But once you start trying to isolate workloads, or run an internal private cloud where different teams or different customers share the same cluster, the flat network starts to feel not as robust anymore:

• Open Communication. There is no real network boundary between tenants, only whatever policy you remember to write.
• No overlapping addresses. Two tenants cannot both use 10.100.0.0/24. In a flat network an IP has to be unique, if your VM uses a static IP address currently, when it runs in Kubernetes, it'll need reconfiguring. Anyone who has tried to renumber a VM with hardcoded IPs in its app config knows how much friction that can cause.
• VMs have opinions. They expect static IPs, DHCP, and to keep their address across a live migration. Pods were never designed around any of that.

These are not new problems. Over the past couple of decades, cloud providers have provided network solutions capable of dealing with overlapping networks which is now known as a cloud VPC. Isovalent Private Networks is what fills this need for Kubernetes, a VPC for both VMs and containers.

Isovalent Private Networks: the Kubernetes VPC

Isovalent Private Networks (IPNs) are the foundation of INV, and they bring a VPC-style networking construct natively into Kubernetes. Anyone who has worked with a cloud VPC will find the operating model familiar. Each IPN is an isolated network domain with its own subnets and address space, and its own security perimeter. Workloads are placed into an IPN by intent. So an administrator declares which network a VM or Pod belongs to, and the control plane wires up the connectivity.

A VPC for Kubernetes is ideal for multitenancy:

• Isolation and multitenancy. Different IPNs are hard-isolated from each other. A workload in one private network cannot reach a workload in another unless you explicitly connect them.
• Overlapping CIDRs. Because each IPN is its own address domain, two tenants can both use 10.100.0.0/24 (for example) at the same time without collision.
• VMs and containers together. The same private network can host VMs running on OpenShift Virtualization and standard Pods, and they talk to each other over Layer 3.
• Native Kubernetes objects. An IPN is implemented as a ClusterwidePrivateNetwork custom resource, so it fits straight into GitOps and the tooling you already use.

And because INV is built on Cilium, everything inside an IPN still gets the full treatment of identity-based CiliumNetworkPolicy and IsovalentNetworkPolicy enforcement in the kernel, and Hubble observability across both VMs and containers.

OpenShift powered by Isovalent Workshop

Step into the role of an OpenShift platform owner preparing a customer-facing application for launch. In this hands-on workshop, learn how Isovalent brings together high-performance networking, policy enforcement, traffic visibility, BGP-based service advertisement, and runtime security to help you deliver applications with confidence.

• Register Now

The environment

The walkthrough runs on an OpenShift cluster with Isovalent Networking for Kubernetes as the CNI, plus OpenShift Virtualization and Isovalent Networking for Virtualization enabled.

The scenario is a small managed private cloud with two customers, Tenant A and Tenant B, each with their own Namespace. Both of them, independently, use 10.100.0.0/24 internally. The goal is to give each one its own network, let each run a VM and a container, and make sure they can never see each other, even though they share the exact same addresses.

Here is a breakdown:

Note that web-a and web-b both want 10.100.0.10, and both app Pods want 10.100.0.20. For traditional Kubernetes flat network this is a non-starter. With IPNs it is just two VPCs.

Prerequisites

Before starting, the Namespaces for Tenant A and Tenant B will be created for the VMs.

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    mutatevirtualmachines.kubemacpool.io: ignore

The same label goes on tenant-b.

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-b
  labels:
    mutatevirtualmachines.kubemacpool.io: ignore

Now, the VPCs can be built.

Building two VPCs with overlapping CIDRs

An IPN is defined with a ClusterwidePrivateNetwork. Tenant A's looks like this:

apiVersion: isovalent.com/v1alpha1
kind: ClusterwidePrivateNetwork
metadata:
  name: tenant-a-vpc
spec:
  subnets:
  - name: default
    cidrv4: 10.100.0.0/24
  networkAttachmentDefinitions:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: tenant-a

Two things to note. The subnets list defines the address space for this VPC. The networkAttachmentDefinitions block tells INV to automatically create a Network Attachment Definition in any namespace matching the selector, so Tenant A gets a ready-to-use attachment in its own namespace.

Tenant B's IPN is identical except for the name and the namespace selector. Note that it uses the same 10.100.0.0/24:

apiVersion: isovalent.com/v1alpha1
kind: ClusterwidePrivateNetwork
metadata:
  name: tenant-b-vpc
spec:
  subnets:
  - name: default
    cidrv4: 10.100.0.0/24
  networkAttachmentDefinitions:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: tenant-b

After applying both, the two private networks and the auto-created NAD show up in each tenant namespace (the lan network already existed on this cluster; the two tenant-*-vpc networks are new):

$ oc get clusterwideprivatenetworks.isovalent.com
NAME           VNI
lan
tenant-a-vpc
tenant-b-vpc

$ oc -n tenant-a get network-attachment-definitions
NAME                   AGE
tenant-a-vpc-default   4s

Two VPCs, same address space, fully isolated from each other by default. No policy required for that isolation, it is the nature of a private network.

Placing VMs and containers into a VPC

Next, workloads go into Tenant A's VPC. A workload joins a private network through the privnet.isovalent.com/network-attachment annotation, which declares the target network and the desired IP address. INV figures out the subnet from the address.

First the VM. This is a small Alpine VM that acts as Tenant A's web tier. This virtual machine has a static IP address:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: web-a
  namespace: tenant-a
spec:
  runStrategy: Always
  template:
    metadata:
      annotations:
        privnet.isovalent.com/network-attachment: |-
          {
            "network": "tenant-a-vpc",
            "ipv4": "10.100.0.10"
          }
      labels:
        app: web
        tier: frontend
    spec:
      domain:
        devices:
          disks:
          - name: containerdisk
            disk: { bus: virtio }
          - name: cloudinitdisk
            disk: { bus: virtio }
          interfaces:
          - name: primary
            bridge: {}
        resources:
          requests:
            memory: 512M
      networks:
      - name: primary
        pod: {}
      volumes:
      - name: containerdisk
        containerDisk:
          image: quay.io/kubevirt/alpine-with-test-tooling-container-disk:v1.8.2
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |
            #cloud-config
            password: alpine
            chpasswd: { expire: False }
          networkData: |
            version: 2
            ethernets:
              eth0:
                addresses: [ 10.100.0.10/32 ]
                gateway4: 169.254.0.1

The address is a /32 and the gateway is a link-local address on purpose. INV gives workloads Layer 3 connectivity to each other, so there is no shared broadcast domain, just routed reachability inside the VPC.

Then the container tiers, app-a and db-a, as plain Pods using the same annotation. The app tier:

apiVersion: v1
kind: Pod
metadata:
  name: app-a
  namespace: tenant-a
  annotations:
    privnet.isovalent.com/network-attachment: |-
      {
        "network": "tenant-a-vpc",
        "ipv4": "10.100.0.20"
      }
  labels:
    app: app
    tier: backend
spec:
  containers:
  - name: json-mock
    image: quay.io/cilium/json-mock:v1.4.0
    env:
    - name: PORT
      value: "8080"
  securityContext:
    seccompProfile:
      type: RuntimeDefault

db-a is the same idea on 10.100.0.30, listening on 5432:

apiVersion: v1
kind: Pod
metadata:
  name: db-a
  namespace: tenant-a
  annotations:
    privnet.isovalent.com/network-attachment: |-
      {
        "network": "tenant-a-vpc",
        "ipv4": "10.100.0.30"
      }
  labels:
    app: db
    tier: data
spec:
  containers:
  - name: json-mock
    image: quay.io/cilium/json-mock:v1.4.0
    env:
    - name: PORT
      value: "5432"
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Tenant B is deployed as a mirror image, reusing the exact same addresses as Tenant A but attached to tenant-b-vpc. For the web-b VM, only the network field in the attachment annotation differs from web-a, while the IP address stays the same:

      annotations:
        privnet.isovalent.com/network-attachment: |-
          {
            "network": "tenant-b-vpc",
            "ipv4": "10.100.0.10"
          }

And app-b, the same IP Address 10.100.0.20 as app-a, isolated inside tenant-b-vpc:

apiVersion: v1
kind: Pod
metadata:
  name: app-b
  namespace: tenant-b
  annotations:
    privnet.isovalent.com/network-attachment: |-
      {
        "network": "tenant-b-vpc",
        "ipv4": "10.100.0.20"
      }
  labels:
    app: app
    tier: backend
spec:
  containers:
  - name: json-mock
    image: quay.io/cilium/json-mock:v1.4.0
    env:
    - name: PORT
      value: "8080"
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Once everything is up, both VMs report their private network address, and both tenants are running the same IP address 10.100.0.10 at the same time:

$ oc -n tenant-a get vmi web-a -o wide
NAME    PHASE     IP            NODENAME             READY
web-a   Running   10.100.0.10   api.csno.cloud.chx   True

$ oc -n tenant-b get vmi web-b -o wide
NAME    PHASE     IP            NODENAME             READY
web-b   Running   10.100.0.10   api.csno.cloud.chx   True

The container tiers carry their VPC address on eth0. Note that it is a /32 network, since INV provides routed Layer 3 reachability rather than a shared subnet:

$ oc -n tenant-a exec app-a -- ip -4 -brief addr show dev eth0
eth0@if1903      UP             10.100.0.20/32

$ oc -n tenant-a exec db-a -- ip -4 -brief addr show dev eth0
eth0@if1905      UP             10.100.0.30/32

The cilium privnet status command gives a control-plane summary of every private network, its subnets, and how many endpoints are attached. Both VPCs report the same 10.100.0.0/24, with Tenant A holding three endpoints (the VM plus two Pods) and Tenant B holding two:

$ cilium privnet status -n cilium
Cluster                    default                    Status  OK

  Network lan                                                 OK
    Subnets   192.168.1.0/24
    Routes    0.0.0.0/0 via 192.168.1.1
    Endpoints           default (local)               2 Endpoints
    No INBs configured

  Network tenant-a-vpc                                        OK
    Subnets   10.100.0.0/24
    Routes    No routes defined for network
    Endpoints           default (local)               3 Endpoints
    No INBs configured

  Network tenant-b-vpc                                        OK
    Subnets   10.100.0.0/24
    Routes    No routes defined for network
    Endpoints           default (local)               2 Endpoints
    No INBs configured

Both tenants are running the same addresses at the same time. Our next step is checking that they behave like real VPCs.

Verifying isolation across overlapping VPCs

First, verify connectivity inside a VPC. From the web-a VM console, the app-a container is reachable even though one is a VM and the other is a Pod. To INV they are just two endpoints in the same private network:

$ virtctl -n tenant-a console web-a
web-a:~$ ip -4 -brief addr show dev eth0
eth0             UP             10.100.0.10/32
web-a:~$ curl --silent --write-out '\n' 10.100.0.20:8080/hostname
{
  "hostname": "app-a"
}

That is a VM and a Pod communicating directly over a single Layer 3 private network.

Now, from web-a, a request goes to 10.100.0.20. In a flat network that address would be ambiguous, there are two of them. Inside tenant-a-vpc, it resolves to exactly one place, app-a, and never to app-b in the other VPC:

web-a:~$ curl --silent --write-out '\n' 10.100.0.20:8080/hostname
{
  "hostname": "app-a"
}

Running the same request from web-b in Tenant B, against the identical 10.100.0.20, resolves to app-b, its own workload inside tenant-b-vpc:

$ virtctl -n tenant-b console web-b
web-b:~$ ip -4 -brief addr show dev eth0
eth0             UP             10.100.0.10/32
web-b:~$ curl --silent --write-out '\n' 10.100.0.20:8080/hostname
{
  "hostname": "app-b"
}

There is no route from Tenant A's VPC into Tenant B's VPC at all. The two 10.100.0.10 VMs and the two 10.100.0.20 Pods run side by side, each invisible to the other: isolated address domains that overlap without colliding.

Traffic can be visualized in Hubble:

Micro-segmentation inside a VPC

Isolation between tenants is the first half. The other half is control within a tenant. By default, every workload inside tenant-a-vpc can talk to every other one, so right now web-a can reach both the app tier and the db tier directly. In a tiered application the front end should not talk straight to the database. Only the app tier should.

An IsovalentNetworkPolicy can select private network endpoints by identity, since this is built on Cilium. The key detail to note is the cni:com.isovalent.private-network.name label. Adding it to the subject selector scopes the policy to a specific private network, and the peer selectors inherit that constraint automatically.

apiVersion: isovalent.com/v1alpha1
kind: IsovalentNetworkPolicy
metadata:
  name: allow-app-to-db
  namespace: tenant-a
spec:
  endpointSelector:
    matchLabels:
      app: db
      cni:com.isovalent.private-network.name: tenant-a-vpc
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: app
    toPorts:
    - ports:
      - port: "5432"
        protocol: TCP

A matching allow-web-to-app policy pairs with it so the front end can still reach the app tier on 8080:

apiVersion: isovalent.com/v1alpha1
kind: IsovalentNetworkPolicy
metadata:
  name: allow-web-to-app
  namespace: tenant-a
spec:
  endpointSelector:
    matchLabels:
      app: app
      cni:com.isovalent.private-network.name: tenant-a-vpc
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: web
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP

With the policies applied, the front end can still reach the app tier, but its direct path to the database is now gone:

web-a:~$ curl --silent --max-time 5 --write-out '\n' 10.100.0.20:8080/hostname
{
  "hostname": "app-a"
}

web-a:~$ curl --max-time 5 10.100.0.30:5432/hostname
curl: (28) Connection timed out after 5001 milliseconds

Meanwhile the app tier reaches the database exactly as intended:

$ oc -n tenant-a exec app-a -- curl --silent --max-time 5 --write-out 'HTTP %{http_code}\n' 10.100.0.30:5432/hostname
{
  "hostname": "db-a"
}HTTP 200

And since this is based on Cilium, the dropped flow is visible in Hubble. The source endpoint is the virt-launcher Pod that hosts the web-a VM:

$ hubble observe --namespace tenant-a --verdict DROPPED
Aug 20 23:40:25.449: tenant-a/virt-launcher-web-a-dfvrs:41194 (ID:16350) <> tenant-a/db-a:5432 (ID:28193) Policy denied DROPPED (TCP Flags: SYN)

Same identity-based policy model already used for containers in the multitenancy series, now applied uniformly across VMs and Pods in a private network.

Summary

Standard Kubernetes networking is flat by design, and that design starts to become a challenge the moment a multitenant platform runs both VMs and container workloads. Namespaces provide an organizational boundary, and network policy provides rules, but neither provides isolated, overlapping address domains.

Isovalent Private Networks treat each tenant as its own Kubernetes VPC. Two tenants can share the same IP address space on one OpenShift cluster, run a mix of VMs and containers, stay completely isolated from each other, and still have the tiers micro-segmented within a tenant. All with identity-based policy enforced in the kernel and full Hubble visibility across VMs and Pods.

IPN is ideal for a platform team trying to consolidate virtual machines and containers onto OpenShift without giving up the network boundaries the business depends on. The VPC model learned in the cloud finally has a native home in Kubernetes.

Schedule an OpenShift powered by Isovalent demo

• Schedule a demo

/authors/christian-hernandez

Christian Hernandez, Technical Marketing Engineer