Networking in Kubernetes is widely perceived as complicated, hard to understand, and even harder to debug. The truth is - Developers and Network Engineers alike find Kubernetes networking intimidating. Developers can’t easily visualize multiple layers of networking, and traditional network engineering tools don’t work with a constantly changing sea of Pods. In this short talk, we will introduce Hubble Timescape, a forensics tool to visualize application communication in a Kubernetes environment and debug connection problems. The audience will walk away with the knowledge of how to troubleshoot Kubernetes networking with confidence. Introduction As I mentioned, I'm working here at Isovalent on observability, mostly in the context of our new-ish Enterprise product, which is called Hubble Timescape, and this is what I will introduce. Today but before we get into Hubble Timescape, let's talk about the plain old Hubble. What's. Hubble? Hubble is named after the Hubble Telescope, as you can guess. And it has this pretty cute logo that looks like a telescope but it's a visibility layer for Cilium it provides visibility to network flows, both L4 and L7 flows. Hubble Timescape overview And it also provides Kubernetes identity to these flows. We'll see how flows look like exactly in a moment in a demo but how Hubble works is. That Cilium itself is a DaemonSet deployed to each node in a cluster, and Hubble. Stores flows in a ring buffer per node we have a centralized component we call Hubble Relay that exposes a gRPC endpoint, and that allows users to query data from these ring buffers on each node. Normally, users will interact with Hubble via either UI or CLI, as we'll see in a moment in a demo. One big limitation here is the ring buffers, because their size is configurable. You can configure the size, but even the maximum size in a very big cluster, or maybe not very big. But other very active cluster because network activity is what matters here, even large ring. Buffer in a very active cluster will hold only a very limited number of flows if we want to store. L absolutely all network activity that's happening in the cluster. So that was our main motivation to look into something like Hubble but with persistent storage, and Hubble Timescape. Hubble Timescape observability Is exactly that it's like Hubble with persistent storage. Not exactly like Hubble? Because it does much more it has support also for example, FGS events. So, FGS is a component in the Hubble Enterprise offering, and it collects. Process-level events also gathered by eBPF. Some of those are network events like socket events. But there are also others like process execution or file operations. So it stores. Timestamp stats much more than just network flows, but we'll focus on network flows today. So, the storage layer of Timescape is ClickHouse database. ClickHouse is a columnar database. That allows us to efficiently perform analytical queries on the huge amount of flows that we have. The data is ingested to ClickHouse from block storage, typically if it's deployed to a cloud. Provider, it would be S3 or GCS we will see that in a moment in the architectural diagram. And Timescape also exposes gRPC endpoints that are queried by the same Hubble UI and CLI. Here we have the architecture diagram of Timescape in the middle, there is a blob storage, typically S3, GCS, or it can be a local directory the flows and other events are exported to this blob storage it can be done by Hubble Enterprise components. That take care of this export, but it can also be integrated into any other log forwarding pipeline. So events from each node in the cluster or in multiple clusters are exported to S3. At the bottom, this bottom box is the Timescape architecture. And it's pretty typical of what you would expect from a system like that. So there is an ingestor. And a server. Well, the ingestor is ingesting data from blob storage into ClickHouse, and the server. Is serving user queries. User queries are executed typically from either the Hubble UI or Hubble CLI. Okay, that's enough talking. Let's take a look at Hubble UI first. Deploy Hubble Timescape This is how Hubble UI looks like. Here I have a cluster with microservices demo application. Deployed to it, and what we see here is a service map view this is what we will focus on. We see a service map this is ideal for one namespace. Here we see a service map generated from flows collected and parsed by Hubble. So what we see here is the actual connections that are. Being made in the cluster, actual traffic from the cluster. Hopefully, that matches what the application set or expected, but this is like the real view we can click on one of those boxes, one. Of the services, and we can see only flows related going to or from that service if we zoom in we can also see Kubernetes metadata, including labels. Hubble is aware of Kubernetes identity. And Cilium gives us this visibility into all useful Kubernetes metadata for our services. We can also filter flows by labels for example, if we filter by app, we can see traffic related. To only the relevant applications. At the bottom, we have a table with live flows happening in the cluster at the moment. Now, let's talk about Timescape if we switch to the live view, we. Get something similar to Hubble, but actually what's happening under the hood is very different. We were querying previously gRPC endpoints that were getting data from ring buffers, now we are. Querying a Timescape server which gets data from the database we have a time selector that lets. Us see flows and service maps at different times the flows table shows L4 and L7 flows. Clicking on a flow shows details like TCP and HTTP connection details, including request. Headers like gRPC requests we can also see TCP flags, and an enterprise feature is. Flow aggregation, which aggregates flows into one per TCP connection. Moving on to Hubble CLI, we can use it with the Timescape server deployment by port forwarding. We can query flows from a specific time, for example, one hour starting on Tuesday. And limit the results to 20 flows to avoid a stream of millions of flows. From a database, we can query the first 20. Results we have filters to narrow down our search, such as filtering. By Kubernetes namespace we can see flows. From or to the monitoring namespace and filter only flows from the monitoring namespace. If we're interested in a specific flow at the top, we can filter by the source and destination IP. Addresses to see if there are similar flows. By using Kubernetes metadata, we can see. Patterns in the requests happening between these two IPs, which may occur every second. Other interesting filters include filtering by L4 and L7 details -. TCP flags or HTTP path we can also filter by HTTP status to check for L7 errors we see flows with HTTP status 200 and 500, indicating that there were. Some errors during that time that's all I had for this demo. Thank you. Q&A Are there any questions or are we doing questions at the end? I just had a quick question about the use cases for Timescape. What do you see? Is it mostly for troubleshooting or? Yes, one thing would be of course troubleshooting issues either with an application or network layer. I didn't show that, but I can share the App UI once again.". So here we see that all flows were forwarded. That's good, and we can filter by dropped flows. There is nothing, and this is great. But if there were any dropped flows, that means that we have incorrect Network policies likely. Of that often means that, and because we have also Kubernetes identity here, including labels. It's very easy to find these gaps, that we have in network policies or any traffic. That is happening that shouldn't be happening because network policies don't allow that. So yeah, like filtering by dropped flows and finding Kubernetes identity. Typically. Network policies would use labels for selecting pods, so this is very useful for troubleshooting.