Cilium Hubble Cheatsheet – Kubernetes Network Observability in a Nutshell
What is Hubble?
By relying on eBPF, all visibility is programmable and allows for a dynamic approach that minimizes overhead while providing deep and detailed visibility as required by users. Hubble has been created and specifically designed to make best use of these new eBPF powers.
Hubble can answer questions such as:
Service dependencies & communication map
- What services are communicating with each other? How frequently? What does the service dependency graph look like?
- What HTTP calls are being made? What Kafka topics does a service consume from or produce to?
Network monitoring & alerting
- Is any network communication failing? Why is communication failing? Is it DNS? Is it an application or network problem? Is the communication broken on layer 4 (TCP) or layer 7 (HTTP)?
- Which services have experienced a DNS resolution problem in the last 5 minutes? Which services have experienced an interrupted TCP connection recently or have seen connections timing out? What is the rate of unanswered TCP SYN requests?
- What is the rate of 5xx or 4xx HTTP response codes for a particular service or across all clusters?
- What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services?
- Which services had connections blocked due to network policy? What services have been accessed from outside the cluster? Which services have resolved a particular DNS name?
Introducing the Hubble Cheat Sheet
When getting to grips with any new tooling, it’s always useful having a easy consumable list or revision notes on how to use the tool. And with that, we’ve produced the Hubble Cheat Sheet, to help you get started, click to open the image in a new tab, right-click “Save Image As” to download locally:
I remember when I worked in IT operational roles, and I would take various cheat sheets for the technologies I worked with often, and have them hung up on the walls. It certainly helped with learning Microsoft PowerShell at the time.
Components of Hubble
Below covers a high level overview of the components that make up Hubble Observability with Cilium in a Kubernetes Cluster.
Cilium Agent – Runs the cilium-agent binary which acts as a CNI to manage connectivity, observability, and security for all CNI-managed Kubernetes pods.
Hubble Relay – Provides a cluster-wide API for querying Hubble flow data, which can be accessed directly or via the Hubble CLI and UI.
Hubble UI – Provides a graphical UI for visualizing network flow data, network policy, and security-related events.
To access the Hubble CLI, we can configure this using the Cilium CLI, if you need to install both CLI tools, follow the links below:
$ cilium hubble port-forward& Forwarding from 0.0.0.0:4245 -> 4245 Forwarding from [::]:4245 -> 4245
To access the UI
$ cilium hubble ui Forwarding from 0.0.0.0:12000 -> 8081 Forwarding from [::]:12000 -> 8081
Checking Hubble Status
Seeing the current and max flows at 100% is expected, as the Hubble relay ring buffer fills, older events will automatically be dropped.
hubble status Healthcheck (via localhost:4245): Ok Current/Max Flows: 11917/12288 (96.98%) Flows/s: 11.74 Connected Nodes: 3/3
Selecting which traffic flows to observe from the Hubble Relay
$ hubble observe --all Get all flows stored in Hubble's buffer --first N Get first N flows stored in Hubble's buffer -f, --follow Follow flows output --last N Get last N flows stored in Hubble's buffer (default 20)
Observing and filtering traffic examples
Observe by Resource
hubble observe you can filter by resources, either looking at incoming/outgoing or all traffic for that resource, below is a list of the filters available.
--from-label filter Show only flows originating in an endpoint with the given labels (e.g. "key1=value1") --from-namespace filter Show all flows originating in the given Kubernetes namespace --from-pod filter Show all flows originating in the given pod name prefix([namespace/]<pod-name>). If namespace is not provided, 'default' is used --from-port filter Show only flows with the given source port (e.g. 8080) --from-service filter Show flows where the source IP address matches the ClusterIP address of the given service name prefix([namespace/]<svc-name>). -l, --label filter Show only flows related to an endpoint with the given labels (e.g. "key1=value1") -n, --namespace filter Show all flows related to the given Kubernetes namespace --node-name filter Show all flows which match the given node names (e.g. "k8s*", "test-cluster/*.company.com") --not filter[=true] Reverses the next filter to be blacklist i.e. --not --from-ip 184.108.40.206 --pod filter Show all flows related to the given pod name prefix ([namespace/]<pod-name>). If namespace is not provided, 'default' is used. --service filter Show flows where either the source or destination IP address matches the ClusterIP address of the given service name prefix ([namespace/]<svc-name>) --to-label filter Show only flows terminating in an endpoint with given labels (e.g. "key1=value1") --to-namespace filter Show all flows terminating in the given Kubernetes namespace --to-pod filter Show all flows terminating in the given pod name prefix([namespace/]<pod-name>). If namespace is not provided, 'default' is used --to-port filter Show only flows with the given destination port (e.g. 8080) --to-service filter Show flows where the destination IP address matches the ClusterIP address of the given service name prefix ([namespace/]<svc-name>)
The following examples will show a mix of these filters in use.
Observe by Protocol
--protocol filter Show only flows which match the given L4/L7 flow protocol (e.g. "udp", "http") $ hubble observe --pod deathstar --protocol http May 4 13:23:40.501: default/tiefighter:42690 -> default/deathstar-c74d84667-cx5kp:80 http-request FORWARDED (HTTP/1.1 POST http://deathstar.default.svc.cluster.local/v1/request-landing) $ hubble observe --namespace tenant-jobs --protocol dns Aug 3 15:13:18.943: tenant-jobs/coreapi-767cf69fb8-cvqxl:53740 (ID:44253) -> kube-system/coredns-787d4945fb-6vvfg:53 (ID:43153) dns-request proxy FORWARDED (DNS Query elasticsearch-master.tenant-jobs.svc.cluster.local.google.internal. AAAA)
Observe by Policy Verdict
--verdict filter Show only flows with this verdict [FORWARDED, DROPPED, AUDIT, REDIRECTED, ERROR, TRACED, TRANSLATED] $ hubble observe --pod deathstar --verdict DROPPED May 4 13:23:47.852: default/xwing:42818 <> default/deathstar-c74d84667-cx5kp:80 Policy denied DROPPED (TCP Flags: SYN)
Observe by FQDN
--fqdn filter Show all flows related to the given fully qualified domain name (e.g. "*.cilium.io"). --from-fqdn filter Show all flows originating at the given fully qualified domain name (e.g. "*.ebpf.io"). --to-fqdn filter Show all flows terminating at the given fully qualified domain name (e.g. "*.isovalent.com"). $ hubble observe --to-fqdn api.github.com Aug 3 15:12:13.929: tenant-jobs/crawler-5c645d68f4-qchk8:47180 (ID:21067) -> api.github.com:80 (world) policy-verdict:all EGRESS ALLOWED (TCP Flags: SYN)
Observe by HTTP Method, Path and Status
--http-method filter Show only flows which match this HTTP method (e.g. "GET", "POST") --http-path filter Show only flows which match this HTTP path regular expressions (e.g. "/page/\\d+") --http-status filter Show only flows which match this HTTP status code prefix (e.g. "404", "5+") $ hubble observe --namespace tenant-jobs --http-path /applicants Aug 3 15:16:18.351: tenant-jobs/resumes-86bbf46b88-n6mcn:51768 (ID:20808) -> tenant-jobs/coreapi-767cf69fb8-cvqxl:9080 (ID:44253) http-request FORWARDED (HTTP/1.1 POST http://coreapi:9080/applicants) $ hubble observe --label app=resumes --http-method POST Aug 3 15:16:28.591: tenant-jobs/resumes-86bbf46b88-n6mcn:51768 (ID:20808) <- tenant-jobs/coreapi-767cf69fb8-cvqxl:9080 (ID:44253) http-response FORWARDED (HTTP/1.1 200 33ms (POST http://coreapi:9080/applicants)
A more complex example
Filters can be combined, too, the below example filters for flows of HTTP requests any pod with the label “app=core-api”, where the HTTP path is “/applicants” and the HTTP method is “PUT”.
$ hubble observe --namespace tenant-jobs --from-label 'app=coreapi' --protocol http --http-path /applicants --http-method PUT Aug 3 15:26:41.563: tenant-jobs/coreapi-767cf69fb8-cvqxl:49662 (ID:44253) -> tenant-jobs/elasticsearch-master-0:9200 (ID:16821) http-request FORWARDED (HTTP/1.1 PUT http://elasticsearch-master.tenant-jobs.svc.cluster.local:9200/applicants/_create/827)
You can use the following argument to exclude data from results:
--not filter[=true] Reverses the next filter to be blacklist i.e. --not --from-ip 220.127.116.11
This example command ensures no flows from anything with a specific label are returned when viewing all flows from a namespace
$ hubble observe -n tenant-jobs --not --label app=coreapi
Formatting the Output
--color string Colorize the output when the output format is one of 'compact' or 'dict'. The value is one of 'auto' (default), 'always' or 'never' (default "auto") -o, --output string Specify the output format, one of: compact: Compact output dict: Each flow is shown as KEY:VALUE pair jsonpb: JSON encoded GetFlowResponse according to proto3's JSON
Watch the Hubble CLI Walkthrough Video
Along with this new content, you can watch the below video to see the commands in action!
Where can I learn more?
You can also get started with Hubble in our Isovalent Labs, I recommend the “Isovalent Enterprise for Cilium: Connectivity Visibility” lab, which also shows off the enterprise features for Hubble, such as Role Based Access Control and Hubble Timescape, the historical datastore for network flows and process events.
Lab: Isovalent Enterprise for Cilium: Connectivity Visibility
This lab provides an introduction to Isovalent Enterprise for Cilium capabilities related to connectivity observability.Start Lab