Download the Cheat Sheet (No sign-up required)
What is Hubble?
Hubble, built on top of Cilium and eBPF, is a networking and security observability platform, enabling deep visibility into the communication and services of your cloud native workloads.
Relying on eBPF, all visibility is programmable, allowing for a dynamic approach that minimizes overhead while providing deep and detailed visibility as required by users. Hubble has been created and specifically designed to best use these new eBPF powers.
Hubble can answer questions such as:
Service dependencies & communication map
- What services are communicating with each other? How frequently? What does the service dependency graph look like?
- What HTTP calls are being made? What Kafka topics does a service consume from or produce to?
Network monitoring & alerting
- Is any network communication failing? Why is communication failing? Is it DNS? Is it an application or network problem? Is the communication broken on layer 4 (TCP) or layer 7 (HTTP)?
- Which services have experienced a DNS resolution problem in the last 5 minutes? Which services have recently experienced an interrupted TCP connection or have seen connections timing out? What is the rate of unanswered TCP SYN requests?
Application monitoring
- What is the rate of 5xx or 4xx HTTP response codes for a particular service or across all clusters?
- What is my cluster’s 95th and 99th percentile latency between HTTP requests and responses? Which services are performing the worst? What is the latency between the different services?
Security observability
- Which services had connections blocked due to network policy? What services have been accessed from outside the cluster? Which services have resolved a particular DNS name?
Introducing the Hubble Cheat Sheet
When learning about any new tooling, it's always useful to have an easy-to-use list or revision notes on how to use the tool. Therefore, we’ve produced the Hubble Cheat Sheet, which complements our Cilium Cheat Sheet.
Either click the image or the button below, which will open a new browser tab to the PDF file for you to save. No sign-up is required.
I remember when I worked in IT operational roles, I would often take various cheat sheets for the technologies I worked with and hang them on the walls of the office. It certainly helped me learn Microsoft PowerShell at the time.
Download the Cheat Sheet (No sign-up required)
Components of Hubble
Below is a high-level overview of the components that make up Hubble Observability with Cilium in a Kubernetes Cluster.
Cilium Agent – Runs the cilium-agent binary which acts as a CNI to manage connectivity, observability, and security for all CNI-managed Kubernetes pods.
Hubble Relay – Provides a cluster-wide API for querying Hubble flow data, which can be accessed directly or via the Hubble CLI and UI.
Hubble UI – Provides a graphical UI for visualizing network flow data, network policy, and security-related events.
Accessing Hubble
To access the Hubble CLI, we can configure this using the Cilium CLI, if you need to install both CLI tools, follow the links below:
$ cilium hubble port-forward&
Forwarding from 0.0.0.0:4245 -> 4245
Forwarding from [::]:4245 -> 4245To access the UI
$ cilium hubble ui
Forwarding from 0.0.0.0:12000 -> 8081
Forwarding from [::]:12000 -> 8081Checking Hubble Status
Seeing the current and max flows at 100% is expected, as the Hubble relay ring buffer fills, older events will automatically be dropped.
hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 11917/12288 (96.98%)
Flows/s: 11.74
Connected Nodes: 3/3
Selecting which traffic flows to observe from the Hubble Relay
$ hubble observe
--all Get all flows stored in Hubble's buffer
--first N Get first N flows stored in Hubble's buffer
-f, --follow Follow flows output
--last N Get last N flows stored in Hubble's buffer (default 20)Observing and filtering traffic examples
Observe by Resource
With hubble observe you can filter by resources, either looking at incoming/outgoing or all traffic for that resource, below is a list of the filters available.
--from-label filter Show only flows originating in an endpoint with the given labels (e.g. "key1=value1")
--from-namespace filter Show all flows originating in the given Kubernetes namespace
--from-pod filter Show all flows originating in the given pod name prefix([namespace/]<pod-name>). If namespace is not provided, 'default' is used
--from-port filter Show only flows with the given source port (e.g. 8080)
--from-service filter Show flows where the source IP address matches the ClusterIP address of the given service name prefix([namespace/]<svc-name>).
-l, --label filter Show only flows related to an endpoint with the given labels (e.g. "key1=value1")
-n, --namespace filter Show all flows related to the given Kubernetes namespace
--node-name filter Show all flows which match the given node names (e.g. "k8s*", "test-cluster/*.company.com")
--not filter[=true] Reverses the next filter to be blacklist i.e. --not --from-ip 2.2.2.2
--pod filter Show all flows related to the given pod name prefix ([namespace/]<pod-name>). If namespace is not provided, 'default' is used.
--service filter Show flows where either the source or destination IP address matches the ClusterIP address of the given service name prefix ([namespace/]<svc-name>)
--to-label filter Show only flows terminating in an endpoint with given labels (e.g. "key1=value1")
--to-namespace filter Show all flows terminating in the given Kubernetes namespace
--to-pod filter Show all flows terminating in the given pod name prefix([namespace/]<pod-name>). If namespace is not provided, 'default' is used
--to-port filter Show only flows with the given destination port (e.g. 8080)
--to-service filter Show flows where the destination IP address matches the ClusterIP address of the given service name prefix ([namespace/]<svc-name>)
The following examples will show a mix of these filters in use.
Observe by Protocol
--protocol filter Show only flows which match the given L4/L7 flow protocol (e.g. "udp", "http")
$ hubble observe --pod deathstar --protocol http
May 4 13:23:40.501: default/tiefighter:42690 -> default/deathstar-c74d84667-cx5kp:80 http-request FORWARDED (HTTP/1.1 POST http://deathstar.default.svc.cluster.local/v1/request-landing)
$ hubble observe --namespace tenant-jobs --protocol dns
Aug 3 15:13:18.943: tenant-jobs/coreapi-767cf69fb8-cvqxl:53740 (ID:44253) -> kube-system/coredns-787d4945fb-6vvfg:53 (ID:43153) dns-request proxy FORWARDED (DNS Query elasticsearch-master.tenant-jobs.svc.cluster.local.google.internal. AAAA)
Observe by Policy Verdict
--verdict filter Show only flows with this verdict [FORWARDED, DROPPED, AUDIT, REDIRECTED, ERROR, TRACED, TRANSLATED]
$ hubble observe --pod deathstar --verdict DROPPED
May 4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
Observe by FQDN
--fqdn filter Show all flows related to the given fully qualified domain name (e.g. "*.cilium.io").
--from-fqdn filter Show all flows originating at the given fully qualified domain name (e.g. "*.ebpf.io").
--to-fqdn filter Show all flows terminating at the given fully qualified domain name (e.g. "*.isovalent.com").
$ hubble observe --to-fqdn api.github.com
Aug 3 15:12:13.929: tenant-jobs/crawler-5c645d68f4-qchk8:47180 (ID:21067) -> api.github.com:80 (world) policy-verdict:all EGRESS ALLOWED (TCP Flags: SYN)
Observe by HTTP Method, Path and Status
--http-method filter Show only flows which match this HTTP method (e.g. "GET", "POST")
--http-path filter Show only flows which match this HTTP path regular expressions (e.g. "/page/\d+")
--http-status filter Show only flows which match this HTTP status code prefix (e.g. "404", "5+")
$ hubble observe --namespace tenant-jobs --http-path /applicants
Aug 3 15:16:18.351: tenant-jobs/resumes-86bbf46b88-n6mcn:51768 (ID:20808) -> tenant-jobs/coreapi-767cf69fb8-cvqxl:9080 (ID:44253) http-request FORWARDED (HTTP/1.1 POST http://coreapi:9080/applicants)
$ hubble observe --label app=resumes --http-method POST
Aug 3 15:16:28.591: tenant-jobs/resumes-86bbf46b88-n6mcn:51768 (ID:20808) <- tenant-jobs/coreapi-767cf69fb8-cvqxl:9080 (ID:44253) http-response FORWARDED (HTTP/1.1 200 33ms (POST http://coreapi:9080/applicants)
A more complex example
Filters can be combined, too, the below example filters for flows of HTTP requests any pod with the label "app=core-api", where the HTTP path is "/applicants" and the HTTP method is "PUT".
$ hubble observe --namespace tenant-jobs --from-label 'app=coreapi' --protocol http --http-path /applicants --http-method PUT
Aug 3 15:26:41.563: tenant-jobs/coreapi-767cf69fb8-cvqxl:49662 (ID:44253) -> tenant-jobs/elasticsearch-master-0:9200 (ID:16821) http-request FORWARDED (HTTP/1.1 PUT http://elasticsearch-master.tenant-jobs.svc.cluster.local:9200/applicants/_create/827)
You can use the following argument to exclude data from results:
--not filter[=true] Reverses the next filter to be blacklist i.e. --not --from-ip 2.2.2.2This example command ensures no flows from anything with a specific label are returned when viewing all flows from a namespace
$ hubble observe -n tenant-jobs --not --label app=coreapi
Formatting the Output
--color string Colorize the output when the output format is one of 'compact' or 'dict'. The value is one of 'auto' (default), 'always' or 'never' (default "auto")
-o, --output string Specify the output format, one of:
compact: Compact output
dict: Each flow is shown as KEY:VALUE pair
jsonpb: JSON encoded GetFlowResponse according to proto3's JSON Watch the Hubble CLI Walkthrough Video
Along with this new content, you can watch the below video to see the commands in action!
H5_WtRaVpDg - "Cilium Hubble Cheatsheet - Kubernetes Network Observability in a Nutshell" Video
Where can I learn more?
You can also get started with Hubble in our Isovalent Labs, I recommend the "Isovalent Enterprise for Cilium: Connectivity Visibility" lab, which also shows off the enterprise features for Hubble, such as Role Based Access Control and Hubble Timescape, the historical datastore for network flows and process events.
Lab: Isovalent Enterprise for Cilium: Connectivity Visibility
This lab provides an introduction to Isovalent Enterprise for Cilium capabilities related to connectivity observability.
Dean Lewis, Senior Technical Marketing Engineer
Dean Lewis is a Senior Technical Marketing Engineer at Isovalent - the company behind the open-source cloud native solution Cilium. Dean had a varied background working in the technology fields, from support to operations to architectural design and delivery at IT Solutions Providers based in the UK, before moving to VMware and focusing on cloud management and cloud native, which remains as his primary focus. You can find Dean in the past and present speaking at various Technology User Groups and Industry Conferences, as well as his personal blog.