In this walkthrough, Dean Lewis, Senior Technical Marketing Engineer at Isovalent, covers Hubble Enterprise Timescape for historical network flow observability and monitoring, further enhancing the capabilities of the Cilium platform. Introduction Hello everyone. In this video, we're going to dive into the Hubble Enterprise feature Timescape, which sits alongside some of our other Enterprise features such as role-based access control and our policy editor. First, let's dive into the architectural overview of how we bring this together and make it possible. Configure Hubble Timescape We first start by setting up the Hubble Enterprise export inside of Hubble Enterprise, and I'll get to show you the config later on in this recording as well. We're going to export that data into some type of storage, whether that be S3-compatible storage, Google storage, or even Azure Blob Storage. Then in Timescape itself, we deploy a number of components. The first point is an ingester, so we read that data from the object storage and push it into a ClickHouse database. There are a couple of options for deploying ClickHouse databases, and you've got a really easy out-of-the-box setup that we manage for you as part of the Timescape installation, which is probably the best and easiest. You can also use the SaaS-based ClickHouse service, or deploy ClickHouse yourself and point Timescape at an existing database. Of course, we've got the Timescape server itself, which interfaces with Hubble UI Enterprise and the CLI as well. We've also got optional components, for example the trimmer, which manages how to ensure that the data size doesn't grow unexpectedly regardless of the time-to-live settings that you've got for aging out that data. That's enough of a quick overview there, so let's just dive straight into the feature itself. I'm going to go into my Hubble UI Enterprise, and I'm already looking at one of my demo apps in my tenant-jobs namespace. We can already see flows being brought into the system. With Hubble Timescape, we're addressing the problem that Hubble Relay has a ring buffer, so there's only a maximum amount of data that you can keep in there. Although it's really great for seeing live information, if you need to go back and troubleshoot things, that data has probably left the ring buffer. Timescape fixes that by storing that historical data ready for you to troubleshoot in the future. To activate, I'm simply going to disable the live view, and as we can see it's going to start to load and pull data from my Timescape server. We can now see all the flows that are available that I can dive back into. We've got a time range here as well, and I can manually change it for whatever information I'm looking for. Yesterday I had some reports from my users that the application wasn't working correctly for whatever reason, so I want to have a quick look in here at that time period. Deploy Hubble Timescape Let's do half an hour or so of when it was reported, and now we've dived into that time and it shows me all those available flows. If there's any change to that service map during that time, it will rebuild that service map as well, and if there's a particular area that I want to pinpoint into, I can simply drag and drop over that timeframe. Another area that Timescape makes available, which I think is really cool, is when we think about the Tetragon events. We can click on the processes view. In that same timeframe, or in any timeframe I'm interested in, I can choose the pod that I'm interested in. Let's look at that crawler pod. It will show me a process ancestry tree of all of the different events that have been captured based on my Tetragon tracing policy that I've put in place. This is interesting because it's actually dragging that data in for me. You do have the ability to visualize this data without Hubble Timescape enabled in your environment, however that comes at the cost of needing to manually export those events from Tetragon and then manually upload them into the Hubble UI. With the Timescape server, it gives you the ability to store them automatically and bring them in as well. In this process tree I can actually see some really interesting information, for example that this crawler pod made outgoing connections that looked like maybe a reverse shell, and I can see those in these events and view that as well. Policy configuration One of the other features that I like about Timescape is what happens when you've deleted a namespace or deleted part of your application that's no longer running. You can't see it in live view any more, but you actually still get access to that data because it was stored in the ClickHouse database. If I scroll to the bottom, we can see this hotel-demo namespace I deployed yesterday. It was a demo application that took up too many resources inside of my cluster, so I removed it shortly after. It doesn't have that little green box because it's no longer receiving live data, because it doesn't exist inside of my cluster any more. But if I click on there, and I've already got a time range that I know I was pulling flows through, I can still search through those flows. I can still go through that service map and dive into that data. For example, I might want to look into the recommendation service and see the flows that are available there. As we can see, it pulls through all that data, and even for that recommendation service I can see all the labels that are available there, and if there was any process information captured as part of that namespace using Tetragon, I can also go back and view that. Let's have a quick look at my storage. What do these events look like? I've got my Timescape bucket already deployed here, and you'll see this being pointed towards my MinIO storage environment as well. We can see where the node data comes from and the data that's available, and we can see all the updates that have been pushed through. Hubble Timescape observability Next, what I want to look at is what the settings are behind the scenes to make this possible, so let's look at my Helm values. First we're going to look at my Hubble Enterprise values. Of course, we need to make sure that data is exported so we can bring that into Hubble Timescape itself. We do that by ensuring that we've got an export mode set up. I'm using Fluentd for this, and I've got a couple of settings set for Fluentd with TLS and output configuration as well. You can see here I'm simply pointing into my MinIO environment, which runs inside of the same cluster today because again that's my demo environment. We can see the bucket that I'm setting it to, and the path of where we want to pick up that data in MinIO. Next, with our Hubble Timescape Helm values, we can see that I've got my ClickHouse cluster enabled, which means it's going to deploy that ClickHouse cluster for me as well. As I mentioned earlier, we can also use an existing ClickHouse cluster, whether it's the SaaS-based service or one that you've deployed yourself. I've also told it that I want to monitor in Prometheus, so I'm going to look at that shortly. We've also deployed something called the trimmer. This is an optional component. We can provide a maximum age and age out data, but we can also look at the overall size of the environment and cap it at a particular size regardless of the time-based retention configuration. Finally, if I click into my Hubble UI values, it's really simple. We're enabling Timescape in the Hubble UI as well, which gives us that button that lets us untick live view, select a time range, and search. Okay, so moving on, how do we actually monitor this? We've spent some time building metrics that can be pushed into Prometheus and visualized through Grafana as well. The first one I wanted to look into is the Hubble Timescape server itself. This helps us understand what queries are going on inside the environment. If I go down to the last five minutes, we can see that I've been running some queries inside as part of this recording, and we can see if they failed or not. That gives us a really good way to actually see what the health of that environment is if we're having problems pulling through that data. In my environment, because I'm running this with quite limited resources and I have this running for about 24 hours now, we've got quite a lot of data being put in there, so we now start to struggle a little bit with that ClickHouse database and the Hubble Enterprise environment, which is to be expected because I haven't designed this to be of any particular performance scale. Conclusion If we go back and look at the ingester dashboard, this is probably one of the most important dashboards. We can see whether or not ingestion has been running. We can see what the newest timestamp is that is available inside our Hubble UI when we search as well, so obviously if this is starting to lag behind by an hour or two, we know that potentially there's an issue with ingestion. If we go back and look at the last six hours, we can actually see where it keeps changing from yes to no. When I was playing around with a couple of the demo apps this morning, I was exhausting those resources, and we could actually see then that the ingester wasn't being able to pull that data fully into my environment. I thought that made it really nice in the demo to say, okay, well you always show us things that look great, but what happens when it doesn't always look fantastic? If I go back into my dashboards view and look at the general folder, I also have some ClickHouse monitoring set up there as well, so we can bring in details about the health of the ClickHouse environment and look at what happens when those queries are run. That's probably what you're going to expect from a more database point of view, to see if the performance there is up to speed, and these types of metrics can be very important if you are running large-scale Kubernetes clusters and therefore have to size your Timescape deployment alongside that. So that kind of brings today's demo to an end, and what I wanted to call out for anybody that wanted to get hands-on with the Hubble Timescape feature is that you can actually do that with Isovalent hands-on labs. If you go to our website, click on the hands-on labs view, and hover over the labs tab, you can view all of our predefined learning paths that we make available to help steer you in different areas or job roles. You can also click on view all labs and look for the Isovalent Enterprise connectivity and visibility lab, which covers the Hubble Timescape components at the end. You've got those different ways to explore Cilium, and if you go through each of the labs in a particular path, you'll see the connectivity and visibility content there as well. Thank you so much for your time joining me today.