Cilium Hubble Series (Part 1): Re-introducing Hubble

“Hubble is the most helpful tool I’ve seen for network visibility in Kubernetes”

Hearing feedback like this from users such as Matthias is rewarding for the team behind Hubble, the eBPF observability tool for Kubernetes networking powered by Cilium. Since Hubble was released nearly four years ago, we’ve received many similar comments from users across the Kubernetes community.

But while it’s been deployed by many users across thousands of clusters, we still find that it remains in the shadow of its older sibling Cilium.

It’s therefore a good time to re-introduce Hubble.

In this multi-part series, we will recap what Hubble is, the use cases it addresses for platform engineers and application owners. We will follow by introducing some of the features customers will find in Isovalent Enterprise for Cilium before sharing our thoughts on the future of Hubble and observability.

To find out more about Hubble, we quizzed Robin Hahling (Staff Software Engineer at Isovalent and core maintainer of Hubble) on his knowledge of Hubble.

What is Hubble?

The Cilium documentation and initial launch blog post answer this question succinctly :

Hubble is a fully distributed networking and security observability platform. It is built on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner.

By building on top of Cilium, Hubble can leverage eBPF for visibility. By relying on eBPF, all visibility is programmable and allows for a dynamic approach that minimizes overhead while providing deep and detailed visibility as required by users. Hubble has been created and specifically designed to make best use of these new eBPF powers.

The core of Hubble is open source and was donated, alongside Cilium, to the CNCF. Hubble is augmented with Enterprise components, which we will walk through in the next post.

Why did we need another tool?

There are lots of monitoring and observability systems available so did we really another one? We believe so. Hubble is actually doing something that was simply not possible before. Previous generation network analysis tools – like Wireshark – don’t have much context about the traffic, in particular in a cloud native environment.

Hubble adds the Kubernetes context – labels, pods, namespaces, network policy verdicts – to the network packets.

It also operates not just at the Layers 3/4 but also at Layer 7.

Which L7 protocol is supported by Hubble?

Hubble supports deep visibility into L7 protocols such as HTTP, DNS and Kafka. Given that gRPC uses HTTP for transport, Hubble can also provide granular visibility into this type of traffic.

For HTTP, Hubble can provide detailed information about request methods (GET, POST, etc…), request status, headers, URLs, protocol and derive metrics such as request duration.

What type of events and metrics can we observe with Hubble?

Observability is traditionally based on metrics, logs and traces. Hubble offers metrics collection, logging of network flows and integrates with OpenTelemetry for distributed tracing.

The dashboard shows some of the detailed network flows that can be aggregated by Hubble.

Is there any way to search for a specific network event?

With the volume of data that can be captured by Hubble, it can sometimes be a challenge to filter through the noise. Thankfully, you can use filters to only capture traffic based on criteria such as: traffic to or from a specific pod, port, service, namespace, label, IP address. You can also use some of the HTTP parameters we talked about earlier like HTTP methods or status code. Users can also combine all these filters to find the needle in the haystack!

How is Hubble built?

Hubble is made up of several components.

The Hubble server runs on each node and retrieves the eBPF-based visibility from Cilium. It is embedded into the Cilium agent in order to achieve high performance and low-overhead. It offers a gRPC service to retrieve flows and expose Prometheus metrics.

The Hubble Relay is there to provide full network visibility across the entire cluster – or across clusters. When the Hubble Relay is deployed, Hubble provides full network visibility by providing a Hubble API which scopes the entire cluster or even multiple clusters in a ClusterMesh scenario.

Consuming data from individual Hubble Servers and from the Hubble Relay can be done in two different ways.

How do we consume the data collected by Hubble?

As mentioned above Hubble provides two main methods to query and inspect network flows – CLI-based and UI-based:

The Hubble CLI is a command-line binary able to connect to either the gRPC API of Hubble Relay or the local server to retrieve flow events.
The graphical User Interface utilizes relay-based visibility to provide a graphical service dependency and connectivity map.

As shown earlier, we can also get the Hubble data scraped by Prometheus and visualized on Grafana dashboards.

Which use cases does Hubble support?

It usually starts when engineers want to have visibility of the cluster they didn’t previously have. The Service map gives them a quick way to see what is running on the cluster.

The other common use case is troubleshooting. When there is a problem with an app, you cannot SSH into each and every node and look into it; you need a tool to show you the overall health of your cluster. Hubble enables users to troubleshoot applications and environments in a meaningful way, navigating easily through what can often be a rather complex environment.

Who is using Hubble?

A brief look at the public Cilium user list will highlight the fact that the vast majority of Cilium users leverage Hubble but if you want some specific examples, you can read the following stories:

Nexxiot – 0 network outage with 100,000+ devices in the field
Form3 – Cilium connects clusters across multiple clouds for failover
Hetzner – Massive increase in RPS and throughput while reducing CPU usage for ingress
PostFinance – solved iptables challenges around scale, observability, and latency
Publishing Company – Secures 100,000+ RPS in a Multi-Tenant Environment
Retail – Connects 390+ Stores and 4.3 Billion Website Visitors with Cilium
S&P Global – Cilium is their multi cloud super highway
Utmost – Implementing zero trust networking at 4,000 flows per second
VSHN – Reduced support burden with Isovalent Enterprise for Cilium

Is there any reason not to use Hubble?

There are a couple of caveats to consider when using Hubble. Any engine that collects deep observability data would typically incur some minor performance costs and Hubble is no exception, albeit Hubble and Cilium’s use of eBPF minimizes the overhead.
One of the current limitations with Hubble is that users cannot specify which data Hubble collects – it is a all-or-nothing behaviour. This will be addressed in an upcoming Cilium release where we will be adding some filters to refine the type of events Hubble processes (for example to globally filter out certain events that are not of interest to the user).

The other minor drawback is around the latency introduced when enabling Layer 7 visibility. This feature is disabled by default when Hubble is set up and has to be enabled using annotations or network policies. The reason latency is introduced is that traffic has to be redirected to an embedded user-space Envoy proxy for L7 parsing and this would add some latency, albeit a very modest one.

Where can I learn more about Hubble?

The official introduction to Cilium and Hubble is worth a read. Installing Hubble using the official documentation only takes a few minutes once Cilium has been installed.

Alternatively, if you’d rather use a free dedicated cloud environment to try out Hubble, you can head out to the lab pages as many of them include Hubble.

Here are some of the labs we recommend:

Observability with Hubble and Cilium Service Mesh

A feature that users expect from their Service Mesh is observability. It is available, with Cilium and Hubble, without the overhead of sidecars. Start the lab to find out how.

Start Lab