“At any given moment, you have the power to say: this is not how the story is going to end.” —Christine Mason Miller. With that, we resume from where we left off while Deploying Isovalent Enterprise for Cilium. In this tutorial, you will learn how to enable Enterprise features (Layer-3, 4 & 7 policies, DNS-based policies, and observe the Network Flows using Hubble-CLI) in an Azure Kubernetes Service (AKS) cluster running Isovalent Enterprise for Cilium.
What is Isovalent Enterprise for Cilium?
Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.
Why Isovalent Enterprise for Cilium?
For enterprise customers requiring support and use of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended.
This offering brings complete flexibility in terms of access to Cilium features while retaining the advantageous ease of use and integration with Azure seamlessly.
Pre-Requisites
The following prerequisites need to be taken into account before you proceed with this tutorial:
You have installed Isovalent Enterprise for Cilium via:
Users can get in touch with their partner Sales/SE representative(s) at sales@isovalent.com for more detailed insights into the below-explained features and get access to the requisite documentation and hubble CLI software images.
What is in it for my Enterprise?
Isovalent Enterprise provides a range of advanced enterprise features that we will demonstrate in this tutorial.
Layer 3/ Layer4 Policy
When using Cilium, endpoint IP addresses are irrelevant when defining security policies. Instead, you can use the labels assigned to the pods to define security policies. The policies will be applied to the right pods based on the labels irrespective of where or when it is running within the cluster.
The layer 3 policy establishes the base connectivity rules regarding which endpoints can talk to each other.
Layer 4 policy can be specified in addition to layer 3 policies or independently. It restricts the ability of an endpoint to emit and/or receive packets on a particular port using a particular protocol.
You can take the example of a Star Wars-inspired example, in which there are three microservices applications: deathstar, tiefighter, and xwing. The deathstar runs an HTTP web service on port 80, which is exposed as a Kubernetes Service to load-balance requests to deathstar across two pod replicas. The deathstar service provides landing services to the empire’s spaceships so that they can request a landing port. The tiefighter pod represents a landing-request client service on a typical empire ship and xwing represents a similar service on an alliance ship. They exist so that we can test different security policies for access control to deathstar landing services.
Validate L3/ L4 Policies
Deploy three services deathstar, xwing and firefighter
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.13/examples/minikube/http-sw-app.yaml
service/deathstar created
deployment.apps/deathstar created
pod/tiefighter created
pod/xwing created
Kubernetes will deploy the pods and service in the background.
Running kubectl get pods,svc will inform you about the progress of the operation.
kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
pod/client 1/1 Running 0 23h
pod/deathstar-54bb8475cc-4gcv4 1/1 Running 0 3m9s
pod/deathstar-54bb8475cc-lq6sv 1/1 Running 0 3m9s
pod/server 1/1 Running 0 23h
pod/tiefighter 1/1 Running 0 3m9s
pod/xwing 1/1 Running 0 3m9s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/deathstar ClusterIP 10.0.114.36 <none>80/TCP 3m9s
service/kubernetes ClusterIP 10.0.0.1 <none>443/TCP 4d1h
Check basic access
From the perspective of the deathstar service, only the ships with label org=empire are allowed to connect and request landing. Since we have no rules enforced, both xwing and tiefighter will be able to request landing.
You can start with the basic policy restricting deathstar landing requests to only the ships that have a label org=empire. This will not allow any ships that don’t have the org=empire label to even connect with the deathstar service. This is a simple policy that filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), so it is often referred to as an L3/L4 network security policy.
The above policy whitelists traffic sent from any pods with label org=empire to deathstar pods with label org=empire, class=deathstar on TCP port 80.
You can now apply this L3/L4 policy:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.13/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
Now if we run the landing requests, only the tiefighter pods with the label org=empire will succeed. The xwing pods will be blocked.
This request will hang, so press Control-C to kill the curl request, or wait for it to time out.
HTTP-aware L7 Policy
Layer 7 policy rules are embedded into Layer 4 Examples rules and can be specified for ingress and egress. A layer 7 request is permitted if at least one of the rules matches. If no rules are specified, then all traffic is permitted. If a layer 4 rule is specified in the policy, and a similar layer 4 rule with layer 7 rules is also specified, then the layer 7 portions of the latter rule will have no effect.
For wireguard encryption, l7Proxy is set to False and hence it is recommended that users should enable the same by updating the ARM template or via Azure CLI.
This will be available in an upcoming release.
In order to provide the strongest security (i.e., enforce least-privilege isolation) between microservices, each service that calls deathstar’s API should be limited to making only the set of HTTP requests it requires for legitimate operation.
For example, consider that the deathstar service exposes some maintenance APIs which should not be called by random empire ships.
Cilium is capable of enforcing HTTP-layer (i.e., L7) policies to limit what URLs the firefighter pod is allowed to reach.
Validate L7 Policy
Apply L7 Policy- Here is an example policy file that extends our original policy by limiting tiefighter to making only a POST /v1/request-landing API call, but disallowing all other calls (including PUT /v1/exhaust-port).
Update the existing rule (from the L3/L4 section) to apply L7-aware policy to protect deathstar
As this rule builds on the identity-aware rule, traffic from pods without the label org=empire will continue to be dropped causing the connection to time out:
DNS-based policies are very useful for controlling access to services running outside the Kubernetes cluster. DNS acts as a persistent service identifier for both external services provided by Google, etc., and internal services such as database clusters running in private subnets outside Kubernetes. CIDR or IP-based policies are cumbersome and hard to maintain as the IPs associated with external services can change frequently. The Cilium DNS-based policies provide an easy mechanism to specify access control while Cilium manages the harder aspects of tracking DNS to IP mapping.
Validate DNS-Based Policies
In line with our Star Wars theme examples, you can use a simple scenario where the Empire’s mediabot pods need access to GitHub for managing the Empire’s git repositories. The pods shouldn’t have access to any other external service.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.13.4/examples/kubernetes-dns/dns-sw-app.yaml
pod/mediabot created
Apply DNS Egress Policy- The following Cilium network policy allows mediabot pods to only access api.github.com
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.13.4/examples/kubernetes-dns/dns-matchname.yaml
ciliumnetworkpolicy.cilium.io/fqdn created
Testing the policy, we see that mediabot has access to api.github.com but doesn’t have access to any other external service, e.g., support.github.com
kubectl exec mediabot -- curl -I -s https://api.github.com |head -1
HTTP/1.1 200 OK
kubectl exec mediabot -- curl -I -s https://support.github.com |head -1
curl: (28) Connection timed out after 5000 milliseconds
command terminated with exit code 28
This request will hang, so press Control-C to kill the curl request, or wait for it to time out.
Combining DNS, Port, and L7 Rules
The DNS-based policies can be combined with port (L4) and API (L7) rules to further restrict access. In our example, we will restrict mediabot pods to access GitHub services only on port 443. The toPorts section in the policy below achieves the port-based restrictions along with the DNS-based policies.
Validate the combination of DNS, Port and L7-based Rules
Cilium supports the transparent encryption of Cilium-managed host traffic and traffic between Cilium-managed endpoints either using WireGuard® or IPsec. In this tutorial, we will be talking about Wireguard.
For wireguard encryption, l7Proxy is set to False and hence it is recommended that users disable the same by updating the ARM template or via Azure CLI.
Wireguard
When WireGuard is enabled in Cilium, the agent running on each cluster node will establish a secure WireGuard tunnel between it and all other known nodes in the cluster.
Packets are not encrypted when they are destined to the same node from which they were sent. This behavior is intended. Encryption would provide no benefits in that case, given that the raw traffic can be observed on the node anyway.
Validate Wireguard Encryption
To demonstrate Wireguard encryption, users can create a client pod that is spun up on one node and a server pod that is spun up on another node in AKS.
The client is doing a “wget” towards the server every 2 seconds.
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client 1/1 Running 0 4s 192.168.1.30 aks-nodepool1-18458950-vmss000000 <none><none>server 1/1 Running 0 16h 192.168.0.38 aks-nodepool1-18458950-vmss000001 <none><none>
Run a bash shell in one of the Cilium pods with kubectl -n kube-system exec -ti ds/cilium -- bash and execute the following commands:
Check that WireGuard has been enabled (number of peers should correspond to a number of nodes subtracted by one):
Install tcpdump on the node where the server pod has been created.
apt-get update
apt-get -y install tcpdump
Check that traffic (HTTP requests and responses) is sent via the cilium_wg0 tunnel device on the node where the server pod has been created:
tcpdump -n -i cilium_wg0
07:04:23.294242 IP 192.168.1.30.40170 >192.168.0.38.80: Flags [P.], seq1:70, ack 1, win 507, options [nop,nop,TS val 1189809356 ecr 3600600803], length 69: HTTP: GET / HTTP/1.1
07:04:23.294301 IP 192.168.0.38.80 >192.168.1.30.40170: Flags [.], ack 70, win 502, options [nop,nop,TS val 3600600803 ecr 1189809356], length 007:04:23.294568 IP 192.168.0.38.80 >192.168.1.30.40170: Flags [P.], seq1:234, ack 70, win 502, options [nop,nop,TS val 3600600803 ecr 1189809356], length 233: HTTP: HTTP/1.1 200 OK
07:04:23.294747 IP 192.168.1.30.40170 >192.168.0.38.80: Flags [.], ack 234, win 506, options [nop,nop,TS val 1189809356 ecr 3600600803], length 0
Kube-Proxy Replacement
One of the additional benefits of using Cilium is its extremely efficient data plane. It’s particularly useful at scale, as the standard kube-proxy is based on a technology – iptables – that was never designed with the churn and the scale of large Kubernetes clusters.
Validate Kube-Proxy Replacement
Users can first validate that the Cilium agent is running in the desired mode with kube-proxy set to Strict:
Verify that the NodePort service has been created:
kubectl get svc my-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx NodePort 10.0.87.130 <none>80:31076/TCP 2s
With the help of the cilium service list command, validate that Cilium’s eBPF kube-proxy replacement created the new NodePort services under port 31076: (Truncated O/P)
At the same time, you can verify using iptables in the host namespace (on the node), that no iptables rules for the service are present:
iptables-save |grep KUBE-SVC
Last but not least, a simple curl test shows connectivity for the exposed NodePort port 31076 as well as for the ClusterIP:
curl127.0.0.1:31076 -I
HTTP/1.1 200 OK
Server: nginx/1.25.1
Date: Tue, 27 Jun 202313:31:00 GMT
Content-Type: text/html
Content-Length: 615Last-Modified: Tue, 13 Jun 202315:08:10 GMT
Connection: keep-alive
ETag: "6488865a-267"Accept-Ranges: bytes
curl10.0.87.130 -I
HTTP/1.1 200 OK
Server: nginx/1.25.1
Date: Tue, 27 Jun 202313:31:31 GMT
Content-Type: text/html
Content-Length: 615Last-Modified: Tue, 13 Jun 202315:08:10 GMT
Connection: keep-alive
ETag: "6488865a-267"Accept-Ranges: bytes
curl10.10.0.5:31076 -I
HTTP/1.1 200 OK
Server: nginx/1.25.1
Date: Tue, 27 Jun 202313:31:47 GMT
Content-Type: text/html
Content-Length: 615Last-Modified: Tue, 13 Jun 202315:08:10 GMT
Connection: keep-alive
ETag: "6488865a-267"Accept-Ranges: bytes
Upgrade AKS clusters running kube-proxy
Note- When I initially wrote this blog post at that time kube-proxy was enabled as a daemonset. You can now go ahead and create or upgrade an AKS cluster on Isovalent Enterprise for Cilium, and your AKS clusters will no longer have Kube-proxy-based iptables implementation.
In this example, you can see a service that was created running a Kube-Proxy-based implementation on an AKS cluster (not running Isovalent Enterprise for Cilium).
The AKS cluster is then upgraded to Isovalent Enterprise for Cilium. As you can see traffic works seamlessly.
There are no more iptables but Cilium endpoints that come into play.
Observing Network Flows with Hubble CLI
Hubble’s CLI extends the visibility that is provided by standard kubectl commands like kubectl get pods to give you more network-level details about a request, such as its status and the security identities associated with its source and destination.
The Hubble CLI can be leveraged for observing network flows from Cilium agents. Users can observe the flows from their local machine workstation for troubleshooting or monitoring. For this tutorial, users can see that all hubble outputs are related to the tests that are done above. Users can try other tests and see the same results with different varying values as expected.
Setup Hubble Relay Forwarding
Use kubectl port forward to hubble-relay, then edit the hubble config to point at the remote hubble server component.
Hubble status can check the overall health of Hubble within your cluster. If using Hubble Relay, a counter for the number of connected nodes will appear in the last line of the output.
hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 8,190/8,190 (100.00%)Flows/s: 30.89Connected Nodes: 2/2
View Last N Events
Hubble observe displays the most recent events based on the number filter. Hubble Relay will display events over all the connected nodes:
If you have a CiliumNetworkPolicy that enforces DNS or HTTP policy, we can use the –type l7 filtering options for hubble to check the HTTP methods and DNS resolution attempts of our applications.
Hubble provides a field called VERDICT that displays one of FORWARDED, ERROR, or DROPPED for each flow. DROPPED could indicate an unsupported protocol within the underlying platform or Network Policy enforcing pod communication. Hubble is able to introspect the reason for ERROR or DROPPED flows and display the reason within the TYPE field of each flow.
hubble observe --output table --verdict DROPPED
Jun 26 08:18:59.517 default/tiefighter:37562 default/deathstar-54bb8475cc-dp646:80 http-request DROPPED HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port
Jun 26 08:19:12.451 default/tiefighter:35394 default/deathstar-54bb8475cc-dp646:80 http-request DROPPED HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port
Filter by Pod or Namespace
To show all flows for a specific pod, filter with the --pod flag
To view filter events through the jq tool can swap the output to json mode. Visualize your metadata through jq which will help to see more metadata around the workload labels like pod name/namespace assigned to both source and destination. This information is accessible by Cilium because it is encoded in the packets based on pod identities.
Amit Gupta is a Senior Technical Marketing Engineer at Isovalent that is powering eBPF cloud-native networking and security. Amit has 20+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source and has worked in the past with companies like Motorola, Juniper, Avi Networks (acquired by VMware), and Prosimo. He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area in his time outside of work. Amit is an avid runner and cyclist and also spends a considerable amount of time helping kids in orphanages.
In this tutorial, users will learn how to deploy Isovalent Enterprise for Cilium on your AKS cluster from Azure Marketplace on a new cluster and also upgrade an existing cluster from an AKS cluster running Azure CNI powered by Cilium to Isovalent Enterprise for Cilium.
Amit Gupta
Industry insights you won’t delete. Delivered to your inbox weekly.
