“Hubble is the most helpful tool I’ve seen for network visibility in Kubernetes”
Hearing feedback like this from users such as Matthias is rewarding for the team behind Hubble, the eBPF observability tool for Kubernetes networking powered by Cilium. Since Hubble was released nearly four years ago, we’ve received many similar comments from users across the Kubernetes community.
But while it’s been deployed by many users across thousands of clusters, we still find that it remains in the shadow of its older sibling Cilium.
It’s therefore a good time to re-introduce Hubble.
In this multi-part series, we will recap what Hubble is, the use cases it addresses for platform engineers and application owners. We will follow by introducing some of the features customers will find in Isovalent Enterprise for Cilium before sharing our thoughts on the future of Hubble and observability.
To find out more about Hubble, we quizzed Robin Hahling (Staff Software Engineer at Isovalent and core maintainer of Hubble) on his knowledge of Hubble.
What is Hubble?
Hubble is a fully distributed networking and security observability platform. It is built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner.
By building on top of Cilium, Hubble can leverage eBPF for visibility. By relying on eBPF, all visibility is programmable and allows for a dynamic approach that minimizes overhead while providing deep and detailed visibility as required by users. Hubble has been created and specifically designed to make best use of these new eBPF powers.
Why did we need another tool?
There are lots of monitoring and observability systems available so did we really another one? We believe so. Hubble is actually doing something that was simply not possible before. Previous generation network analysis tools – like Wireshark – don’t have much context about the traffic, in particular in a cloud native environment.
Hubble adds the Kubernetes context – labels, pods, namespaces, network policy verdicts – to the network packets.
It also operates not just at the Layers 3/4 but also at Layer 7.
Which L7 protocol is supported by Hubble?
Hubble supports deep visibility into L7 protocols such as HTTP, DNS and Kafka. Given that gRPC uses HTTP for transport, Hubble can also provide granular visibility into this type of traffic.
For HTTP, Hubble can provide detailed information about request methods (GET, POST, etc…), request status, headers, URLs, protocol and derive metrics such as request duration.
What type of events and metrics can we observe with Hubble?
The dashboard shows some of the detailed network flows that can be aggregated by Hubble.
Is there any way to search for a specific network event?
With the volume of data that can be captured by Hubble, it can sometimes be a challenge to filter through the noise. Thankfully, you can use filters to only capture traffic based on criteria such as: traffic to or from a specific pod, port, service, namespace, label, IP address. You can also use some of the HTTP parameters we talked about earlier like HTTP methods or status code. Users can also combine all these filters to find the needle in the haystack!
How is Hubble built?
Hubble is made up of several components.
The Hubble server runs on each node and retrieves the eBPF-based visibility from Cilium. It is embedded into the Cilium agent in order to achieve high performance and low-overhead. It offers a gRPC service to retrieve flows and expose Prometheus metrics.
The Hubble Relay is there to provide full network visibility across the entire cluster – or across clusters. When the Hubble Relay is deployed, Hubble provides full network visibility by providing a Hubble API which scopes the entire cluster or even multiple clusters in a ClusterMesh scenario.
Consuming data from individual Hubble Servers and from the Hubble Relay can be done in two different ways.
How do we consume the data collected by Hubble?
As mentioned above Hubble provides two main methods to query and inspect network flows – CLI-based and UI-based:
- The Hubble CLI is a command-line binary able to connect to either the gRPC API of Hubble Relay or the local server to retrieve flow events.
- The graphical User Interface utilizes relay-based visibility to provide a graphical service dependency and connectivity map.
As shown earlier, we can also get the Hubble data scraped by Prometheus and visualized on Grafana dashboards.
Which use cases does Hubble support?
It usually starts when engineers want to have visibility of the cluster they didn’t previously had. The Service map gives them a quick way to see what is running on the cluster.
The other common use case is troubleshooting. When there is a problem with an app, you cannot SSH into each and every node and look into it; you need a tool to show you the overall health of your cluster. Hubble enables users to troubleshoot applications and environments in a meaningful way, navigating easily through what can often be a rather complex environment.
Who is using Hubble?
A brief look at the public Cilium user list will highlight the fact that the vast majority of Cilium users leverage Hubble but if you want some specific examples, you can read the following stories:
- Nexxiot – 0 network outage with 100,000+ devices in the field
- Form3 – Cilium connects clusters across multiple clouds for failover
- Hetzner – Massive increase in RPS and throughput while reducing CPU usage for ingress
- PostFinance – solved iptables challenges around scale, observability, and latency
- Publishing Company – Secures 100,000+ RPS in a Multi-Tenant Environment
- Retail – Connects 390+ Stores and 4.3 Billion Website Visitors with Cilium
- S&P Global – Cilium is their multi cloud super highway
- Utmost – Implementing zero trust networking at 4,000 flows per second
- VSHN – Reduced support burden with Isovalent Enterprise for Cilium
Is there any reason not to use Hubble?
There are a couple of caveats to consider when using Hubble. Any engine that collects deep observability data would typically incur some minor performance costs and Hubble is no exception, albeit Hubble and Cilium’s use of eBPF minimizes the overhead.
One of the current limitations with Hubble is that users cannot specify which data Hubble collects – it is a all-or-nothing behaviour. This will be addressed in an upcoming Cilium release where we will be adding some filters to refine the type of events Hubble processes (for example to globally filter out certain events that are not of interest to the user).
The other minor drawback is around the latency introduced when enabling Layer 7 visibility. This feature is disabled by default when Hubble is set up and has to be enabled using annotations or network policies. The reason latency is introduced is that traffic has to be redirected to an embedded user-space Envoy proxy for L7 parsing and this would add some latency, albeit a very modest one.
Where can I learn more about Hubble?
Alternatively, if you’d rather use a free dedicated cloud environment to try out Hubble, you can head out to the lab pages as many of them include Hubble.
Here are some of the labs we recommend:
Observability with Hubble and Cilium Service Mesh
A feature that users expect from their Service Mesh is observability. It is available, with Cilium and Hubble, without the overhead of sidecars. Start the lab to find out how.Start Lab
Intra-Cluster and Inter-Cluster Traffic
Hubble works natively with Cluster Mesh - Cilium's multi cluster feature for global load-balancing and service discovery.Start Lab
HTTP Golden Signals
With Hubble, you can monitor HTTP Golden Signals with minimal effort! Try it yourself in this Hubble & Grafana lab.Start Lab
Visualise IPv6 Traffic with Hubble
Hubble also supports IPv6! Find out more in the Cilium IPv6 Networking and Observability lab.Start Lab
In the part 2 of this series, we will look at some at the use cases in the Hubble Enteprise edition that comes with Isovalent Enterprise for Cilium.
- What is Hubble?
- Why did we need another tool?
- Which L7 protocol is supported by Hubble?
- What type of events and metrics can we observe with Hubble?
- Is there any way to search for a specific network event?
- How is Hubble built?
- How do we consume the data collected by Hubble?
- Which use cases does Hubble support?
- Who is using Hubble?
- Is there any reason not to use Hubble?
- Where can I learn more about Hubble?