Introducing the Hubble Policy Change Tracker, a new feature which records Network Policy Changes along side network flows, allowing you to easily troubleshoot changes in your Kubernetes networking Introduction Using Cilium and Hubble together is a no-brainer because Hubble turns all of that eBPF data into something you can actually use without making you think at the kernel level first. Today I'm looking at a new Hubble Enterprise feature called the network policy tracker, and the goal is to troubleshoot Kubernetes network policies directly from the UI. In the flows view I'm looking at a demo application running in the tenant-jobs namespace. You can already see the application topology, the live flows between components, and some layer 7 visibility. Using Cilium and Hubble together is an no-brainer because it gives you rich data at that eBPF kernel level and not that you need to know eBPF to on San all that data because Hubble actually makes. Use of it for you so you can dive straight into the network flows and that's where we're going to concentrate today as part of a new Hubble Enterprise. Feature which is the network policy tracker and that allows us to troubleshoot Network policies using the Hubble Enterprise UI that I have in front of us so first foremost if I go to the flows section we're actually just. Going to back off this for the time being and we're going to go to a live view of My Demo application that's running inside of my environment in a nam bace called tenant job so we can see here there's a couple of different components to it that are running we've got a crawler app it's making some calls. Externally to a well-known website we've got known common Services inside of our application such as kaer elter. Deploy network policy troubleshooting I'm also using Hubble Timescape here, which means I can move from the live view into historical flow data. I've already set the time range to the point where these policies were deployed, and now I can see both the captured flows and the policy objects that belong to this environment. Search zooke keeper there as well we even got those identified little icons as well and here you can even see we getting layer seven information we got a network policy that has been deployed as part of that so that's all of and running inside of our environment so what I'm going to do now is I'm going to enable a Hubble Timescape which allows us to go back into historical datas and flows inside of our environment it's been an ontable live view and gives me. My time range here to select from that PS that data from our time State time scape setup already left it set here to the time that I deployed the network polic my environment and first you'll see on the flows table no can we see the flows that we are now We've now captured. For this application we can actually see the details of the network policies that have been created in my environment and when they were created and we're going. To dive into that into a little bit more detail in a moment using the pulse's view of the Hubble Enterprise you are first and foremost I'm going to go into. network policy troubleshooting observability To create a real troubleshooting scenario, I go to the terminal, delete one network policy, and introduce a bad port into one of the layer 7 policies. After that I have to wait a little for those events to be ingested into Timescape. I can watch the ingestion progress in the Timescape Grafana dashboards as well. Once the data has caught up, I move the UI to the later time range and now the dropped flows are visible. I also see two orange markers showing that one policy was deleted and another one was updated. My terminal window I'm just going to make a little bit of AIT change here so the first thing we're going to do we're. Going to delete one of those Network policies already that's going to ca some instability inside of my environment and we're going to start to see cloes denied. For example and we also are going to make a erroneous change to one of our layer seven PES with inside of this environment as well so here we're going to replace the parts to an invalid part and again we're going to use the network policy view out of the H Ender. Prise to troubleshoot going forward so if I now go back into my time scape environment so one of the things that we need to is wait for these events. Just to be ingested into the time State database so what we can also do to see where we're up to with that using time skapes we have some graan. Dashboards that allow us to pull information what's going on at the ingestion ratee so these are available out the box when you deploy the Timescape as well just go through the I veent Enterprise documentation of how to set these up see that my ingestion is running and we can see what the latest. Flow is that's being captured inside of my environment as well so we're just going to wait a couple of minutes for that to Capt up and then we'll see this visualize on the Hubble Enterprise UI Okay. So, we can now see that we've caught up with our flows we can even see that this K8 events field has also caught up. Policy configuration If I open the deleted policy, the policy view automatically selects the historical version, and the change tracker shows me when that policy was created and when it was deleted. From there I can download that older version and reapply it to the cluster so the affected flows can start working again. I can do the same thing for the updated policy, except in that case I also get a diff view that shows me exactly which fields changed. In this example, the diff makes it obvious that the port values were changed, so I download the known-good version and restore it with kubectl apply. What I like about this workflow is that I can correlate the policy change directly with the flow verdicts in one place instead of jumping between the cluster, source control, and a CI/CD system. If I meant to block traffic, I can verify that quickly. If I broke something by mistake, I can also see it quickly and restore the previous policy. As well cuz there was a captured change at the Kubernetes level for that policy change tracker component so let's go back into our Hubble Enterprise Ur I'm. Now going to update my time frame to my later time in the future and now we can see as I mentioned. We've got those flows that are dropping here so we can see that's because we've deleted one of those policies and we can also see as well we have these two. Orange dots listed here and that shows us one that a policy has been deleted and if we hover up the second one that another policy has been updated. As well so let's see that into a little bit more detail so I'm going to hover over that first policy again into deleted and click view policy this takes into the policy's view within the H. Enterprise UI and we see it's already selected the historical policy as well so what you'll notice in this latest version is that we now have this component here which is our change tracker and it shows us when the created version and also the deleted version because there was no changes to this particular policy there's kind of not too much to see there but we still have the ability now we can download this policy and then we can reapply to our cluster as well so that we can. Ensure that those flows work as expected C forward if I go back to my flows environment. we also know that we had an issue with our layer seven Ingress visibility so let's take to view that policy as well so here again we've got a very similar. View but we can see that this has changed to updated we've also got a little bit of a different button here and that allows us to show the differences between the previous policy. And the current policy that's in place now if I highlight that it looks a little bit like a git diff output which is very handy and then we can see. What actually Chang so here we can see that my q c command changed the port numbers that were available so what we can now do is if I go back to the created so now we can download that file I'm going to click here to download there see I've already got two copies of that already and then we're going to reapply that back into my environment so. If I go into my terminal window New Cube CTL apply in the new space of T jobs h. And f file and then if I just drag all those files across this is the one that I had earlier I can reapply that back to my environment and that will update what's happened there and then again what's going to happen shortly is we'll get a new version of this here and we can go. Back and see that part has been reverted but and that's just a quick overview of the new feature which allows us to trollo network policies inside of our environment really easy in a differential typee way as well and actually correlate that back to the verdicts of flows that we see with inside of our environment as well so really great to know one if a policy that has either been applied to an environment or has been altered ins environment is actually having a positive and negative effect that you expected to have so for example if you meant to stop traffic inside of your environment or maybe you wanted to allow. Configure network policy troubleshooting That's the quick overview of the network policy tracker. Thank you very much. Traffic environment a quick and easy way to see that and then also even double check the configuration of those policies without having to go back to your cluster or into maybe a cicd system. And Source control to see what Chang we made there first and foremost you can do it all from this one easy interface thank you very much.