The Kubernetes flat networking model works when every workload is a Pod, but what happens when you introduce Virtual Machines into the mix? Platform teams are not just running containers anymore. OpenShift Virtualization has now made Virtual Machines a first class citizen on Kubernetes, and these VM based workloads are landing on the same cluster as container-based microservices. The moment VMs get onboarded next to containers, a challenge shows up that namespaces and standard Kubernetes network policy do not solve on their own: two tenants want to use the same IP address. In this post we explore how Isovalent Networking for Virtualization (INV) solves that with Isovalent Private Networks, and why they are best understood as the Kubernetes VPC. In this post, we’ll focus on two demo builds using isolated private networks with overlapping CIDRs on an OpenShift Virtualization cluster, running a mix of VMs and containers inside them, and then look at how to apply micro-segmentation between the tiers. All of it using identity-aware eBPF enforcement. The problem with a flat network When you run a VM on Kubernetes, that VM lives inside a Pod. The network sees it as just another Pod, so it inherits the standard Kubernetes flat network design. Every workload can reach every other workload, and every address has to be unique across the whole cluster. For a lot of setups that is completely fine. But once you start trying to isolate workloads, or run an internal private cloud where different teams or different customers share the same cluster, the flat network starts to feel not as robust anymore: Open Communication. There is no real network boundary between tenants, only whatever policy you remember to write. No overlapping addresses. Two tenants cannot both use 10.100.0.0/24. In a flat network an IP has to be unique, if your VM uses a static IP address currently, when it runs in Kubernetes, it'll need reconfiguring. Anyone who has tried to renumber a VM with hardcoded IPs in its app config knows how much friction that can cause. VMs have opinions. They expect static IPs, DHCP, and to keep their address across a live migration. Pods were never designed around any of that. These are not new problems. Over the past couple of decades, cloud providers have provided network solutions capable of dealing with overlapping networks which is now known as a cloud VPC. Isovalent Private Networks is what fills this need for Kubernetes, a VPC for both VMs and containers. Isovalent Private Networks: the Kubernetes VPC Isovalent Private Networks (IPNs) are the foundation of INV, and they bring a VPC-style networking construct natively into Kubernetes. Anyone who has worked with a cloud VPC will find the operating model familiar. Each IPN is an isolated network domain with its own subnets and address space, and its own security perimeter. Workloads are placed into an IPN by intent. So an administrator declares which network a VM or Pod belongs to, and the control plane wires up the connectivity. A VPC for Kubernetes is ideal for multitenancy: Isolation and multitenancy. Different IPNs are hard-isolated from each other. A workload in one private network cannot reach a workload in another unless you explicitly connect them. Overlapping CIDRs. Because each IPN is its own address domain, two tenants can both use 10.100.0.0/24 (for example) at the same time without collision. VMs and containers together. The same private network can host VMs running on OpenShift Virtualization and standard Pods, and they talk to each other over Layer 3. Native Kubernetes objects. An IPN is implemented as a ClusterwidePrivateNetwork custom resource, so it fits straight into GitOps and the tooling you already use. And because INV is built on Cilium, everything inside an IPN still gets the full treatment of identity-based CiliumNetworkPolicy and IsovalentNetworkPolicy enforcement in the kernel, and Hubble observability across both VMs and containers. The environment The walkthrough runs on an OpenShift cluster with Isovalent Networking for Kubernetes as the CNI, plus OpenShift Virtualization and Isovalent Networking for Virtualization enabled. The scenario is a small managed private cloud with two customers, Tenant A and Tenant B, each with their own Namespace. Both of them, independently, use 10.100.0.0/24 internally. The goal is to give each one its own network, let each run a VM and a container, and make sure they can never see each other, even though they share the exact same addresses. Here is a breakdown: Note that web-a and web-b both want 10.100.0.10, and both app Pods want 10.100.0.20. For traditional Kubernetes flat network this is a non-starter. With IPNs it is just two VPCs. Prerequisites Before starting, the Namespaces for Tenant A and Tenant B will be created for the VMs. The same label goes on tenant-b. Now, the VPCs can be built. Building two VPCs with overlapping CIDRs An IPN is defined with a ClusterwidePrivateNetwork. Tenant A's looks like this: Two things to note. The subnets list defines the address space for this VPC. The networkAttachmentDefinitions block tells INV to automatically create a Network Attachment Definition in any namespace matching the selector, so Tenant A gets a ready-to-use attachment in its own namespace. Tenant B's IPN is identical except for the name and the namespace selector. Note that it uses the same 10.100.0.0/24: After applying both, the two private networks and the auto-created NAD show up in each tenant namespace (the lan network already existed on this cluster; the two tenant-*-vpc networks are new): Two VPCs, same address space, fully isolated from each other by default. No policy required for that isolation, it is the nature of a private network. Placing VMs and containers into a VPC Next, workloads go into Tenant A's VPC. A workload joins a private network through the privnet.isovalent.com/network-attachment annotation, which declares the target network and the desired IP address. INV figures out the subnet from the address. First the VM. This is a small Alpine VM that acts as Tenant A's web tier. This virtual machine has a static IP address: The address is a /32 and the gateway is a link-local address on purpose. INV gives workloads Layer 3 connectivity to each other, so there is no shared broadcast domain, just routed reachability inside the VPC. Then the container tiers, app-a and db-a, as plain Pods using the same annotation. The app tier: db-a is the same idea on 10.100.0.30, listening on 5432: Tenant B is deployed as a mirror image, reusing the exact same addresses as Tenant A but attached to tenant-b-vpc. For the web-b VM, only the network field in the attachment annotation differs from web-a, while the IP address stays the same: And app-b, the same IP Address 10.100.0.20 as app-a, isolated inside tenant-b-vpc: Once everything is up, both VMs report their private network address, and both tenants are running the same IP address 10.100.0.10 at the same time: The container tiers carry their VPC address on eth0. Note that it is a /32 network, since INV provides routed Layer 3 reachability rather than a shared subnet: The cilium privnet status command gives a control-plane summary of every private network, its subnets, and how many endpoints are attached. Both VPCs report the same 10.100.0.0/24, with Tenant A holding three endpoints (the VM plus two Pods) and Tenant B holding two: Both tenants are running the same addresses at the same time. Our next step is checking that they behave like real VPCs. Verifying isolation across overlapping VPCs First, verify connectivity inside a VPC. From the web-a VM console, the app-a container is reachable even though one is a VM and the other is a Pod. To INV they are just two endpoints in the same private network: That is a VM and a Pod communicating directly over a single Layer 3 private network. Now, from web-a, a request goes to 10.100.0.20. In a flat network that address would be ambiguous, there are two of them. Inside tenant-a-vpc, it resolves to exactly one place, app-a, and never to app-b in the other VPC: Running the same request from web-b in Tenant B, against the identical 10.100.0.20, resolves to app-b, its own workload inside tenant-b-vpc: There is no route from Tenant A's VPC into Tenant B's VPC at all. The two 10.100.0.10 VMs and the two 10.100.0.20 Pods run side by side, each invisible to the other: isolated address domains that overlap without colliding. Traffic can be visualized in Hubble: Micro-segmentation inside a VPC Isolation between tenants is the first half. The other half is control within a tenant. By default, every workload inside tenant-a-vpc can talk to every other one, so right now web-a can reach both the app tier and the db tier directly. In a tiered application the front end should not talk straight to the database. Only the app tier should. An IsovalentNetworkPolicy can select private network endpoints by identity, since this is built on Cilium. The key detail to note is the cni:com.isovalent.private-network.name label. Adding it to the subject selector scopes the policy to a specific private network, and the peer selectors inherit that constraint automatically. A matching allow-web-to-app policy pairs with it so the front end can still reach the app tier on 8080: With the policies applied, the front end can still reach the app tier, but its direct path to the database is now gone: Meanwhile the app tier reaches the database exactly as intended: And since this is based on Cilium, the dropped flow is visible in Hubble. The source endpoint is the virt-launcher Pod that hosts the web-a VM: Same identity-based policy model already used for containers in the multitenancy series, now applied uniformly across VMs and Pods in a private network. Summary Standard Kubernetes networking is flat by design, and that design starts to become a challenge the moment a multitenant platform runs both VMs and container workloads. Namespaces provide an organizational boundary, and network policy provides rules, but neither provides isolated, overlapping address domains. Isovalent Private Networks treat each tenant as its own Kubernetes VPC. Two tenants can share the same IP address space on one OpenShift cluster, run a mix of VMs and containers, stay completely isolated from each other, and still have the tiers micro-segmented within a tenant. All with identity-based policy enforced in the kernel and full Hubble visibility across VMs and Pods. IPN is ideal for a platform team trying to consolidate virtual machines and containers onto OpenShift without giving up the network boundaries the business depends on. The VPC model learned in the cloud finally has a native home in Kubernetes.