OpenShift offers amazing metrics and observability out of the box. But what if you want to go back in time and see exactly what your network was doing when something went wrong? Red Hat OpenShift has always been strong when it comes to observability. Platform teams get built-in monitoring, alerting, and dashboards for their cluster and workloads without bolting on a separate stack. For a Platform Engineer, that builds a lot of confidence in the day-to-day. But metrics tell you that something happened. They don’t always tell you what happened on the network, when it happened, and why a connection was allowed or denied. That’s the gap this blog explores. It walks through how to use Hubble Timescape on OpenShift to investigate a network issue after the fact. The good news is that OpenShift already has great metrics and observability. Hubble Timescape just makes that already good experience even better by adding historical, identity-aware visibility into the network itself. OpenShift with Isovalent Networking for Kubernetes For this blog, the environment is Red Hat OpenShift with Isovalent Networking for Kubernetes installed as the CNI. Isovalent Networking for Kubernetes is built on Cilium, and it’s an official, Red Hat certified third party CNI for OpenShift. This is great, because it’s a supported networking platform that plugs directly into the OpenShift experience platform teams already know. It’s installed on OpenShift during the initial cluster installation, using the Operator from OperatorHub. Once the Operator is running, the CNI and its configuration are managed through a CiliumConfig custom resource, which keeps everything consistent with how OpenShift manages the rest of the platform. The CNI is one of the most critical components of the Cloud Native stack, and that makes a supported configuration on OpenShift essential for any platform team. On top of that, because it’s based on Cilium, it brings eBPF powered networking, identity-based security, and Hubble observability. What Hubble Timescape Is Before going any further, it’s worth quickly going over Hubble and Hubble Timescape. Hubble is the observability layer that comes with Cilium. It’s powered directly by the eBPF datapath, so it gives deep, real-time visibility into network flows, DNS queries, HTTP requests, and policy decisions. The key word there is real-time. Hubble shows what is happening right now. Hubble Timescape takes that same rich flow data and stores it so it can be queried historically. Instead of only seeing live traffic, you can go back and analyze what the network was doing minutes or hours ago. It ships with its own UI for visualizing and filtering flows, and it even implements Hubble’s GetFlow API, so the same hubble CLI already in use can query historical data. This solves a very real problem for Platform Engineers. Network issues are often intermittent, or they get reported after the fact. By the time someone opens a ticket saying “something isn’t right,” the live traffic is long gone. Metrics might tell you a spike happened, but they won’t reconstruct the exact flows. Hubble Timescape makes it possible to rewind and ask “what actually happened on the network?” rather than trying to reproduce the problem live. This environment uses integrated Hubble Timescape. This is the simplest option, tuned for a small, single cluster setup, and it’s enabled directly through the Operator. There’s no separate stack to deploy, it becomes part of the existing Isovalent Networking for Kubernetes installation. Enabling a feature gate and turning Timescape on in the CiliumConfig is all it takes: By default, integrated Timescape runs in an ephemeral mode that only retains data for a short window and does not survive restarts. For anything beyond a quick look, backing Timescape with a Persistent Volume is recommended so the data sticks around. That’s done by adding the persistence block under the timescape section, as shown above. With that in place, historical network visibility runs right alongside OpenShift’s existing observability. Now to put it to work. The Problem: A Network Policy That Isn’t Doing Its Job Here’s a scenario. A team is running a workload in a namespace called test. There is a client workload that talks to a webapp service. The webapp exposes two endpoints, a /public endpoint for information anyone can see, and a /private endpoint that returns sensitive information. The team did the right thing. They applied a CiliumNetworkPolicy to control access to the webapp. Their intent was clear: the client should only be able to reach the /public endpoint, and nothing else. The problem is that even with the policy applied, the client can still reach /private. Private information is leaking, and the team isn’t sure why. This is exactly the kind of “it looks like it’s working, but it isn’t” situation where historical visibility pays off. Hubble Timescape is the right tool to figure out what’s going on and fix it. Verifying the Issue First, confirm the behavior the team is seeing. From the client pod, the /public endpoint is reachable as expected: That part is fine. But the whole point of the policy was to stop access to /private. Testing that endpoint also returns data, which is not what the team wanted: So the policy is applied, traffic is flowing, and sensitive data is exposed. The connection is clearly being allowed, but from the command line alone it’s hard to see why the existing policy isn’t tight enough. This is where Hubble Timescape comes in. Using Hubble Timescape The Hubble Timescape UI opens on the overview page. The overview shows the default cluster, which is the OpenShift cluster in use. Any additional clusters would show up here as well. Clicking on the default box, or selecting it from the Cluster filter dropdown, opens it up. This gives an overview of all the connections in the OpenShift cluster. That’s a lot to take in, so selecting test from the namespace filter narrows it down. Now the client service shows up making a call to the webapp service, which is exactly the traffic from the test. Calls out to the OpenShift DNS service also appear, which is normal. Clicking on the webapp box shows that there was an incoming connection from the client pod and that it was allowed, denoted by a green checkmark. So far this matches what the command line showed: the connection is getting through. Next is the flows per minute section. It shows a chart of the connections over time, with an orange dot standing out. That dot indicates a CiliumNetworkPolicy was applied during that window. Hovering over the dot shows information about the policy that was applied. When clicking View policy, the Policy Editor page opens, with the YAML on the left and a diagram of the flow on the right. Inspecting the YAML and looking at the diagram makes the issue easy to spot right away. The CiliumNetworkPolicy does allow access to the webapp, but it allows all connections to port 80. It never restricts which HTTP paths are allowed, which is why /private is slipping through. The intent was /public only, but the policy is only doing Layer 3 and Layer 4 filtering. The fix is to tighten this up with a Layer 7 rule so that only GET requests to /public are allowed. That can be built directly in the editor. In the diagram, the In Namespace section contains the pod selector for app=client. Clicking on app=client brings up the editor dialog. When clicking on Advanced, the option to add an HTTP rule appears. From there, a more robust Layer 7 HTTP rule can be added. Entering /public for the Path and GET for the Method does the job. Clicking Save returns to the overview page. Clicking Show diff shows exactly what was added to the policy. To apply this policy, the Download button in the upper right downloads the updated policy, ready to apply. Going from spotting the problem in historical flows to generating the corrected policy, all in the same UI, is a big part of why Hubble Timescape fits so well into the OpenShift workflow. There’s no need to reproduce the issue live or guess at the fix. Verifying the Solution With the policy downloaded, applying it to the test namespace is a single command: Running the same tests as before, the /public endpoint still works, which is exactly what the team wanted: But this time, the /private endpoint is denied. The Layer 7 rule is doing its job: Back in the Hubble Timescape UI, this is easy to confirm visually. The connection to /private now shows up as dropped, closing the loop on the investigation. You can also see this using the hubble CLI, filtering out by namespace and protocol. The sensitive endpoint is protected, the intended access still works, and both the problem and the fix were proven using the historical record instead of chasing live traffic. Summary OpenShift gives Platform Engineers a fantastic foundation of metrics and observability, and what Hubble Timescape does is extend that foundation into the network itself. Metrics tell you something changed. Hubble Timescape lets you rewind and see the actual flows, the identities involved, and the policy decisions behind them. This walkthrough used integrated Hubble Timescape on OpenShift to investigate a network policy that looked correct but was still leaking private information. It covered verifying the issue, using the historical flow data to find the gap in the existing CiliumNetworkPolicy, building a tighter Layer 7 rule right in the UI, applying it, and confirming the fix. All of that was possible because Isovalent Networking for Kubernetes brings Cilium and Hubble into OpenShift as a certified CNI, and integrated Timescape made the historical piece easy to turn on. For Platform Engineers already invested in OpenShift’s observability story, Hubble Timescape is a natural next step. It makes an already good experience with OpenShift even better.