Back to blog

Enabling Enterprise Features for Cilium in Elastic Kubernetes Service (EKS)

Amit Gupta
Amit Gupta
Published: Updated: Isovalent
Enabling Enterprise Features for Cilium in Elastic Kubernetes Service (EKS)

“At any given moment, you have the power to say, this is not how the story will end.”—Christine Mason Miller. Let’s resume where we left off while deploying Isovalent Enterprise for Cilium from the AWS marketplace. This tutorial teaches you how to enable Enterprise features in an Elastic Kubernetes Service (EKS) cluster running Isovalent Enterprise for Cilium from the AWS marketplace on an EKS cluster.

What is Isovalent Enterprise for Cilium?

Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.

Why Isovalent Enterprise for Cilium?

For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits:

  • Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security.
  • Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
  • Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments.
  • Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
  • Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making deploying and managing complex microservices architectures easy.
  • Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.

Why AWS marketplace?

AWS Marketplace is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. In AWS Marketplace, you can find, try, buy, and deploy the software and services needed to build new solutions and manage your cloud infrastructure. The catalog includes solutions for different industries and technical areas, free trials, and consulting services from AWS partners. Included among these solutions are Kubernetes application-based container offers. These offers contain applications that are meant to run on Kubernetes clusters such as Elastic Kubernetes Service (EKS).

Pre-Requisites

The following prerequisites need to be taken into account before you proceed with this tutorial:

  • Access to AWS marketplace. Create a new account for free.
  • The Cilium operator requires the following EC2 privileges to perform ENI creation and IP allocation.
  • An EKS cluster is running Isovalent Enterprise for Cilium from the AWS marketpace.
  • Install kubectl
  • Install Helm
  • Install eksctl
  • Install awscli
  • Cilium CLI: Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s).
  • Hubble CLI: To access the observability data collected by Hubble, you can install the Hubble CLI. You can install Hubble CLI for Linux, macOS, or other distributions on their local machine (s) or server (s).

Layer 3/ Layer 4 Policy

When using Cilium, endpoint IP addresses are irrelevant when defining security policies. Instead, you can use the labels assigned to the Pods to define security policies. The policies will be applied to the right Pods based on the labels, irrespective of where or when they run within the cluster.

The layer 3 policy establishes the base connectivity rules regarding which endpoints can talk to each other. 

The layer 4 policy can be specified independently or in addition to the layer 3 policies. It restricts an endpoint’s ability to emit and/or receive packets on a particular port using a protocol.

You can take a Star Wars-inspired example in which there are three microservices applications: deathstar, tiefighter, and xwing. The deathstar runs an HTTP web service on port 80, which is exposed as a Kubernetes Service to load-balance requests to deathstar across two pod replicas. The deathstar service provides landing services to the empire’s spaceships so that they can request a landing port. The tiefighter pod represents a landing-request client service on a typical empire ship, and xwing represents a similar service on an alliance ship. They exist so that you can test different security policies for access control to deathstar landing services.

Validate L3/ L4 Policies

  • Deploy three services: deathstar, xwing, and firefighter
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15/examples/minikube/http-sw-app.yaml
service/deathstar created
deployment.apps/deathstar created
pod/tiefighter created
pod/xwing created
  • Kubernetes will deploy the Pods and service in the background.
  • Running kubectl get Pods,svc will inform you about the progress of the operation.
kubectl get pods,svc
NAME                             READY   STATUS    RESTARTS   AGE
pod/client                       1/1     Running   0          23h
pod/deathstar-54bb8475cc-4gcv4   1/1     Running   0          3m9s
pod/deathstar-54bb8475cc-lq6sv   1/1     Running   0          3m9s
pod/server                       1/1     Running   0          23h
pod/tiefighter                   1/1     Running   0          3m9s
pod/xwing                        1/1     Running   0          3m9s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/deathstar    ClusterIP   10.0.114.36    <none>        80/TCP    3m9s
service/kubernetes   ClusterIP   10.0.0.1       <none>        443/TCP   4d1h
  • Check basic access
    • From the perspective of the deathstar service, only the ships with the label org=empire are allowed to connect and request landing. Since you have no rules enforced, both xwing and tiefighter can request landing.
kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
  • You can start with the basic policy restricting deathstar landing requests to only the ships that have a label org=empire. This will not allow any ships that don’t have the org=empire label to even connect with the deathstar service. This simple policy filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), often called an L3/L4 network security policy.
  • The above policy whitelists traffic sent from any Pods with the label org=empire to deathstar Pods with the label org=empire, class=deathstar on TCP port 80.
  • You can now apply this L3/L4 policy:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
  • If you run the landing requests, only the tiefighter Pods with the label org=empire will succeed. The xwing Pods will be blocked.
kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
  • Now, the same request run from an xwing pod will fail:
kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

This request will hang, so press Control-C to kill the curl request or wait for it to time out.

HTTP-aware L7 Policy

Layer 7 policy rules are embedded into Layer 4 Example rules and can be specified for ingress and egress. A layer 7 request is permitted if at least one of the rules matches. If no rules are specified, then all traffic is permitted. If a layer 4 rule is specified in the policy, and a similar layer 4 rule with layer 7 rules is also specified, then the layer 7 portions of the latter rule will have no effect.

Note-

To provide the strongest security (i.e., enforce least-privilege isolation) between microservices, each service that calls deathstar’s API should be limited to making only the set of HTTP requests it requires for legitimate operation.

For example, consider that the deathstar service exposes some maintenance APIs that random empire ships should not call.

kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Panic: deathstar exploded

goroutine 1 [running]:
main.HandleGarbage(0x2080c3f50, 0x2, 0x4, 0x425c0, 0x5, 0xa)
        /code/src/github.com/empire/deathstar/
        temp/main.go:9 +0x64
main.main()
        /code/src/github.com/empire/deathstar/
        temp/main.go:5 +0x85

Cilium can enforce HTTP-layer (i.e., L7) policies to limit the firefighter pod’s ability to reach URLs. 

Validate L7 Policy

  • Apply L7 Policy—An example policy file extends our original policy by limiting tiefighter to making only a POST /v1/request-landing API call and disallowing all other calls (including PUT /v1/exhaust port).
  • Update the existing rule (from the L3/L4 section) to apply L7-aware policy to protect deathstar
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.13/examples/minikube/sw_l3_l4_l7_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 configured
  • re-run a curl towards deathstar & exhaust port
kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed

kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
Access denied
  • As this rule builds on the identity-aware rule, traffic from Pods without the label org=empire will continue to be dropped, causing the connection to time out:
kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

DNS-Based Policies

DNS-based policies are useful for controlling access to services outside the Kubernetes cluster. DNS acts as a persistent service identifier for external services, for example, by Google, and internal services, such as database clusters running in private subnets outside Kubernetes. CIDR or IP-based policies are cumbersome and hard to maintain as the IPs associated with external services can change frequently. The Cilium DNS-based policies provide an easy mechanism to specify access control, while Cilium manages the harder aspects of tracking DNS to IP mapping.

Validate DNS-Based Policies

  • In line with our Star Wars theme examples, you can use a simple scenario where the Empire’s mediabot Pods need access to GitHub to manage the Empire’s git repositories. The Pods shouldn’t have access to any other external service.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-sw-app.yaml
pod/mediabot created
  • Apply DNS Egress Policy. The following Cilium network policy allows mediabot Pods to only access api.github.com
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-matchname.yaml
ciliumnetworkpolicy.cilium.io/fqdn created
  • Testing the policy, you can see that mediabot has access to api.github.com but doesn’t have access to any other external service, e.g., support.github.com
kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/1.1 200 OK

kubectl exec mediabot -- curl -I -s https://support.github.com | head -1
curl: (28) Connection timed out after 5000 milliseconds
command terminated with exit code 28

This request will hang, so press Control-C to kill the curl request or wait for it to time out.

Combining DNS, Port, and L7 Rules

The DNS-based policies can be combined with port (L4) and API (L7) rules to restrict access further. In our example, you can restrict mediabot Pods to access GitHub services only on port 443. The toPorts section of the policy below achieves the port-based restrictions and DNS-based policies.

Validate the combination of DNS, Port, and L7-based Rules

  • Applying the policy
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-port.yaml
ciliumnetworkpolicy.cilium.io/fqdn configured
  • Testing, the access to https://support.github.com on port 443 will succeed, but the access to http://support.github.com on port 80 will be denied.
kubectl exec mediabot -- curl -I -s https://support.github.com | head -1
HTTP/1.1 200 OK

kubectl exec mediabot -- curl -I -s --max-time 5 http://support.github.com | head -1
command terminated with exit code 28

Encryption

Cilium supports the transparent encryption of Cilium-managed host traffic and traffic between Cilium-managed endpoints using WireGuard® or IPsec. In this tutorial, you will learn about Wireguard.

Note-

Wireguard (Pod-to-Pod)

When WireGuard is enabled in Cilium, the agent running on each cluster node will establish a secure WireGuard tunnel between it and all other known Nodes.

Packets are not encrypted when destined to the same node from which they were sent. This behavior is intended. Encryption would provide no benefits, given that the raw traffic can be observed on the node anyway.

Validate Wireguard Encryption

  • To demonstrate Wireguard encryption, users can create a client pod spun up on one node and a server pod spun up on another in EKS.
    • The client does a “wget” towards the server every 2 seconds.
kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP              NODE                                               NOMINATED NODE   READINESS GATES
client                           1/1     Running   0          20s     172.31.46.22    ip-172-31-46-109.ap-southeast-2.compute.internal   <none>           <none>
server                           1/1     Running   0          61s     172.31.27.246   ip-172-31-20-222.ap-southeast-2.compute.internal   <none>           <none>
  • Run a bash shell in one of the Cilium Pods with kubectl -n kube-system exec -ti ds/cilium -- bash and execute the following commands:
    • Check that WireGuard has been enabled (the number of peers should correspond to the number of Nodes subtracted by one):
kubectl -n cilium-system exec ds/cilium -- cilium status | grep Encryption

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
Encryption:              Wireguard       [NodeEncryption: Enabled, cilium_wg0 (Pubkey: ############################################=, Port: 51871, Peers: 1)]
  • Install tcpdump on the node where the server pod has been created.
yum update
yum install -y tcpdump
  • Check that traffic (HTTP requests and responses) is sent via the cilium_wg0 tunnel device on the node where the server pod has been created:
tcpdump -n -i cilium_wg0

09:36:33.907499 IP 172.31.46.109.49918 > 172.31.20.222.8472: OTV, flags [I] (0x08), overlay 0, instance 3623
IP 172.31.46.22.46816 > 172.31.27.246.80: Flags [P.], seq 1:70, ack 1, win 494, options [nop,nop,TS val 2530839475 ecr 1953183113], length 69: HTTP: GET / HTTP/1.1
09:36:33.907540 IP 172.31.20.222.39534 > 172.31.46.109.8472: OTV, flags [I] (0x08), overlay 0, instance 14484
IP 172.31.27.246.80 > 172.31.46.22.46816: Flags [.], ack 70, win 483, options [nop,nop,TS val 1953183114 ecr 2530839475], length 0
09:36:33.907734 IP 172.31.20.222.39534 > 172.31.46.109.8472: OTV, flags [I] (0x08), overlay 0, instance 14484
IP 172.31.27.246.80 > 172.31.46.22.46816: Flags [P.], seq 1:234, ack 70, win 483, options [nop,nop,TS val 1953183115 ecr 2530839475], length 233: HTTP: HTTP/1.1 200 OK
09:36:33.907862 IP 172.31.20.222.39534 > 172.31.46.109.8472: OTV, flags [I] (0x08), overlay 0, instance 14484
IP 172.31.27.246.80 > 172.31.46.22.46816: Flags [FP.], seq 234:849, ack 70, win 483, options [nop,nop,TS val 1953183115 ecr 2530839475], length 615: HTTP
09:36:33.908430 IP 172.31.46.109.49918 > 172.31.20.222.8472: OTV, flags [I] (0x08), overlay 0, instance 3623
IP 172.31.46.22.46816 > 172.31.27.246.80: Flags [.], ack 234, win 493, options [nop,nop,TS val 2530839476 ecr 1953183115], length 0

Wireguard (Node-to-Node)

By default, WireGuard-based encryption only encrypts traffic between Cilium-managed Pods. You can also enable node-to-node encryption, which encrypts node-to-node, pod-to-node, and node-to-pod traffic.

Validate Wireguard Encryption

Run a bash shell in one of the Cilium Pods with kubectl -n kube-system exec -ti ds/cilium -- bash and execute the following commands:

  • Check that WireGuard has been enabled (NodeEncryption: Enabled)
kubectl -n cilium-system exec ds/cilium -- cilium status | grep Encryption

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
Encryption:              Wireguard       [NodeEncryption: Enabled, cilium_wg0 (Pubkey: /############################################=, Port: 51871, Peers: 1)]
  • Install tcpdump on the node(s).
yum update
yum install -y tcpdump
  • Check that node-to-node traffic is sent via the cilium_wg0 tunnel device.
kubectl get nodes -o wide

NAME                                             STATUS   ROLES    AGE    VERSION               INTERNAL-IP       EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-192-168-104-238.ap-south-1.compute.internal   Ready    <none>   131m   v1.29.3-eks-ae9a62a   192.168.104.238   <none>        Amazon Linux 2   5.10.219-208.866.amzn2.x86_64   containerd://1.7.11
ip-192-168-141-217.ap-south-1.compute.internal   Ready    <none>   131m   v1.29.3-eks-ae9a62a   192.168.141.217   <none>        Amazon Linux 2   5.10.219-208.866.amzn2.x86_64   containerd://1.7.11
  • Traffic is encrypted between the two Nodes through the WireGuard interface.
    You can observe that Cilium Health requires port TCP 4240 for all the Nodes for Cilium Health monitoring, and this traffic is also encrypted between the Nodes.
192.168.104.238.53264: Flags [.], ack 4081575545, win 509, options [nop,nop,TS val 3501711868 ecr 2677193498], length 0
13:38:40.999783 IP 192.168.104.238.53264 > 192.168.141.217.4240: Flags [.], ack 1, win 502, options [nop,nop,TS val 2677208603 ecr 3501697020], length 0
13:38:41.074585 IP 192.168.141.217.4240 > 192.168.104.238.38798: Flags [.], ack 3421044590, win 509, options [nop,nop,TS val 4095391564 ecr 136823413], length 0
13:38:41.074616 IP 192.168.104.238.38798 > 192.168.141.217.4240: Flags [.], ack 1, win 502, options [nop,nop,TS val 136838515 ecr 4095376640], length 0

Ingress

Cilium uses the standard Kubernetes Ingress resource definition with an ingressClassName of cilium.

Cilium allows you to specify the load balancer mode for the Ingress resource:

  • dedicated: The Ingress controller will create a dedicated load balancer for the Ingress.
  • shared: The Ingress controller will use a shared load balancer for all Ingress resources.

Each Ingress controller mode has its benefits and drawbacks. The shared mode saves resources by sharing a single LoadBalancer Service across all Ingress resources in the cluster. In contrast, the dedicated mode can help to avoid potential conflicts (e.g., path prefix) between resources.

Note: Ingress helm values are available when you create an EKS cluster from the AWS marketplace, as listed in the prerequisites above.

Validate Ingress in EKS

  • Create a demo app in which the ingress configuration routes traffic to backend services from the bookinfo demo microservices app from the Istio project.
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl get pods -o wide

NAME                             READY   STATUS    RESTARTS   AGE   IP                NODE
              NOMINATED NODE   READINESS GATES
details-v1-65599dcf88-rsn8c      1/1     Running   0          25s   192.168.153.19    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
productpage-v1-9487c9c5b-47bkt   1/1     Running   0          24s   192.168.153.160   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
ratings-v1-59b99c644-5sqbh       1/1     Running   0          25s   192.168.140.54    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
reviews-v1-5985998544-9jtdx      1/1     Running   0          25s   192.168.104.1     ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
reviews-v2-86d6cc668-svdvt       1/1     Running   0          25s   192.168.156.198   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
reviews-v3-dbb5fb5dd-kwdrb       1/1     Running   0          25s   192.168.104.79    ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
  • Deploy an ingress with Cilium Ingress Controller type as dedicated.
    • This example routes requests for the path /details to the details service, and / to the productpage service.
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.7/examples/kubernetes/servicemesh/basic-ingress.yaml
  • You’ll see a LoadBalancer service automatically created for this ingress when you get the list of services.
    • For EKS, you will observe an FQDN for the external IP rather than an actual IP address.
kubectl get svc

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                      AGE
cilium-ingress-basic-ingress   LoadBalancer   10.100.42.179    a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com   80:30493/TCP,443:31109/TCP   9s
details                        ClusterIP      10.100.160.211   <none>                                                                     9080/TCP                     8m7s
kubernetes                     ClusterIP      10.100.0.1       <none>                                                                     443/TCP                      3h3m
productpage                    ClusterIP      10.100.71.44     <none>                                                                     9080/TCP                     8m7s
ratings                        ClusterIP      10.100.106.195   <none>                                                                     9080/TCP                     8m7s
reviews                        ClusterIP      10.100.142.166   <none>                                                                     9080/TCP                     8m7s
  • The external IP address is populated into the Ingress
kubectl get ingress

NAME            CLASS    HOSTS   ADDRESS                                                                    PORTS   AGE
basic-ingress   cilium   *       a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com   80      100s
  • You can also check the respective port mappings created in AWS for the Load Balancer.
    • aws elb describe-load-balancers

O/P Truncated

aws elb describe-load-balancers

  "LoadBalancerName": "a7381f04d19b14629a443ad3aeacee80",
            "DNSName": "a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com",
            "CanonicalHostedZoneName": "a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com",
            "CanonicalHostedZoneNameID": "ZP97RAFLXTNZK",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 443,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31109
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 80,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30493
                    },
                    "PolicyNames": []
                }
            ],
  • Send HTTP Requests- Check (with curl or in your browser) that you can make HTTP requests to that external address. The / path takes you to the home page for the bookinfo application.
curl --fail -s http://a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com/details/1 -v

* Trying 3.108.107.12:80...
* Connected to a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com (3.108.107.12) port 80 (#0)
> GET /details/1 HTTP/1.1
> Host: a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< server: envoy
< date: Tue, 16 Jul 2024 09:29:36 GMT
< content-length: 178
< x-envoy-upstream-service-time: 3
<
* Connection #0 to host a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com left intact
{"id":1,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}
  • As you keep adding Ingress resources, the Cilium Ingress controller will create dedicated LoadBalancer Services for each.
kubectl get ingress -A

NAMESPACE   NAME             CLASS    HOSTS   ADDRESS
PORTS   AGE
default     basic-ingress    cilium   *       ae433be6727054783896c9895876fb6f-1617631252.ap-south-1.elb.amazonaws.com   80      43s
default     basic-ingress1   cilium   *       a4b53b55d28ed45ca82285dc23fae422-1118518200.ap-south-1.elb.amazonaws.com   80      6s
default     basic-ingress2   cilium   *       af797a4519fc84884a2bf09a64a3a157-2068600342.ap-south-1.elb.amazonaws.com   80      29s
  • Optional: Deploy an ingress with LoadBalancer type as Shared. Notice how the Ingress controller will use a shared load balancer for all Ingress resources when you add ingress.
kubectl get svc -n cilium-system

NAME             TYPE           CLUSTER-IP       EXTERNAL-IP
   PORT(S)                      AGE
cilium-agent     ClusterIP      None             <none>
   9964/TCP                     21m
cilium-ingress   LoadBalancer   10.100.109.91    a36e5511e81ee4a6d8e7561b12f37d48-1961023961.ap-south-1.elb.amazonaws.com   80:32170/TCP,443:30474/TCP   21m
hubble-metrics   ClusterIP      None             <none>
   9965/TCP                     21m
hubble-peer      ClusterIP      10.100.204.210   <none>
   443/TCP                      21m
hubble-relay     ClusterIP      10.100.247.140   <none>
   80/TCP                       21m
hubble-ui        ClusterIP      10.100.194.203   <none>
   80/TCP                       21m
kubectl get ingress -A

NAMESPACE   NAME             CLASS    HOSTS   ADDRESS
PORTS   AGE
default     basic-ingress    cilium   *       a36e5511e81ee4a6d8e7561b12f37d48-1961023961.ap-south-1.elb.amazonaws.com   80      2m6s
default     basic-ingress1   cilium   *       a36e5511e81ee4a6d8e7561b12f37d48-1961023961.ap-south-1.elb.amazonaws.com   80      116s
default     basic-ingress2   cilium   *       a36e5511e81ee4a6d8e7561b12f37d48-1961023961.ap-south-1.elb.amazonaws.com   80      4s

Securing Ingress with a Cilium Network Policy

  • By default, all the external traffic is allowed. Let’s apply a network policy to lock down the external traffic.
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.7/examples/kubernetes/servicemesh/policy/external-lockdown.yaml
  • With this policy applied, any request originating from outside the cluster will be rejected with a 403 Forbidden status code:
curl --fail -s http://a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com/details/1 -v

*   Trying 15.207.92.38:80...
* Connected to a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com (15.207.92.38) port 80 (#0)
> GET /details/1 HTTP/1.1
> Host: a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-length: 15
< content-type: text/plain
< date: Tue, 16 Jul 2024 09:38:30 GMT
< server: envoy
* The requested URL returned error: 403
* Closing connection 0
  • You can also see the request being denied using hubble from another terminal.
 kubectl --namespace=cilium-system exec -i -t cilium-44r2d -- hubble observe -f --identity ingress

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
Jul 16 09:40:34.100: 192.168.59.131:60732 (ingress) -> default/cilium-ingress-basic-ingress:30493 (world) http-request DROPPED (HTTP/1.1 GET http://a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com/details/1)
Jul 16 09:40:34.100: 192.168.59.131:60732 (ingress) <- default/cilium-ingress-basic-ingress:30493 (world) http-response FORWARDED (HTTP/1.1 403 0ms (GET http://a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com/details/1))

Kube-Proxy Replacement

One of the additional benefits of using Cilium is its extremely efficient data plane. It’s particularly useful at scale, as the standard kube-proxy is based on a technology – iptables – that was never designed with the churn and the scale of large Kubernetes clusters.

Note-

  • Delete the kube-proxy Daemonset and configmap. For existing installations running kube-proxy as a DaemonSet, remove it using the commands below.
    • kubectl -n kube-system delete ds kube-proxy
    • kubectl -n kube-system delete cm kube-proxy
  • Remove kube-proxy iptable entries from each node
    • With root permissions, ensure that iptable entries pertinent to kube-proxy are removed.
    • This is done to clear out AWS-VPC-CNI-related rules.
      • iptables-save | grep -v KUBE | iptables-restore
      • iptables-save | grep -E -v 'AWS-SNAT-CHAIN|AWS-CONNMARK-CHAIN' | iptables-restore
    • If you provision an EKS cluster without the AWS-VPC-CNI plugin, you don’t need to do the above step.
  • The user needs to configure Cilium to point to the Kube API endpoint; otherwise, the cluster will be in an unhealthy state. Make sure the cluster is configured with these helm values when Cilium is installed.
kubeProxyReplacement=true
k8sServiceHost=${API_SERVER_IP}
k8sServicePort=${API_SERVER_PORT}

Validate Kube-Proxy Replacement

  • Users can first validate that the Cilium agent is running in the desired mode with kube-proxy set to True:
kubectl -n cilium-system exec ds/cilium -- cilium status --verbose

Defaulted container "cilium-agent" out of: cilium-agent, config (init), mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init), install-cni-binaries (init)
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.29+ (v1.29.4-eks-036c24b) [linux/amd64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideEnvoyConfig", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumEnvoyConfig", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Secrets", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   True   [eth0   172.31.20.222 fe80::92:37ff:fe07:92f5 (Direct Routing), eth1   172.31.24.191 fe80::79:72ff:fe27:e28d, eth2   172.31.16.207 fe80::c:12ff:fea6:50d9]
  • You can also check that kube-proxy is not running as a Daemonset on the EKS cluster.
kubectl get ds -A

NAMESPACE       NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
cilium-system   cilium                   2         2         2       2            2           kubernetes.io/os=linux            18s
cilium-system   cilium-license-manager   2         2         0       2            0           <none>                            18s
cilium-system   cilium-node-init         2         2         2       2            2           kubernetes.io/os=linux            18s
kube-system     aws-node                 0         0         0       0            0           io.cilium/aws-node-enabled=true   5h19m
  • You can deploy nginx Pods, create a new NodePort service, and validate that Cilium installed the service correctly.
  • The following yaml is used for the backend Pods:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 50
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
  • Verify that the NGINX Pods are up and running: 
kubectl get pods -o wide

NAME                        READY   STATUS    RESTARTS   AGE   IP                NODE                                             NOMINATED NODE   READINESS GATES
my-nginx-684dd4dcd4-2xczh   1/1     Running   0          59s   192.168.103.127   ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-4zwxp   1/1     Running   0          60s   192.168.124.44    ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-6vdn9   1/1     Running   0          61s   192.168.125.195   ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-9kfd5   1/1     Running   0          59s   192.168.149.197   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-bhph7   1/1     Running   0          60s   192.168.115.146   ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-blwhn   1/1     Running   0          61s   192.168.109.242   ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-clfkq   1/1     Running   0          60s   192.168.117.57    ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-d4795   1/1     Running   0          60s   192.168.139.219   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-dpsbd   1/1     Running   0          59s   192.168.111.137   ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-gmtdl   1/1     Running   0          58s   192.168.138.53    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-lktg9   1/1     Running   0          60s   192.168.154.30    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-np297   1/1     Running   0          58s   192.168.140.54    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-p27qr   1/1     Running   0          61s   192.168.106.212   ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-ptlvp   1/1     Running   0          61s   192.168.107.77    ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-qjvx9   1/1     Running   0          59s   192.168.132.236   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-qpgsb   1/1     Running   0          61s   192.168.145.53    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-sqp7q   1/1     Running   0          59s   192.168.153.19    ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-xdqhd   1/1     Running   0          58s   192.168.139.221   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-xz5vj   1/1     Running   0          61s   192.168.147.200   ip-192-168-141-217.ap-south-1.compute.internal   <none>           <none>
my-nginx-684dd4dcd4-ztpnr   1/1     Running   0          61s   192.168.109.34    ip-192-168-104-238.ap-south-1.compute.internal   <none>           <none>
  • Create a NodePort service for the instances:
kubectl expose deployment my-nginx --type=NodePort --port=80
  • Verify that the NodePort service has been created:
kubectl get svc my-nginx

NAME       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
my-nginx   NodePort   10.100.170.109   <none>        80:31170/TCP   7s
  • With the help of the cilium service list command, validate that Cilium’s eBPF kube-proxy replacement created the new NodePort services under port 31170: (Truncated O/P)
101   192.168.136.122:31170   NodePort       1 => 192.168.140.54:80 (active)
                                             2 => 192.168.119.204:80 (active)
                                             3 => 192.168.152.88:80 (active)
                                             4 => 192.168.153.160:80 (active)
                                             5 => 192.168.115.146:80 (active)
                                             6 => 192.168.120.176:80 (active)
                                             7 => 192.168.121.132:80 (active)
                                             8 => 192.168.99.58:80 (active)
                                             9 => 192.168.141.24:80 (active)
                                             10 => 192.168.105.30:80 (active)
                                             11 => 192.168.124.44:80 (active)
                                             12 => 192.168.147.200:80 (active)
                                             13 => 192.168.153.19:80 (active)
                                             14 => 192.168.111.137:80 (active)
                                             15 => 192.168.132.236:80 (active)
                                             16 => 192.168.138.53:80 (active)
                                             17 => 192.168.107.77:80 (active)
                                             18 => 192.168.132.120:80 (active)
                                             19 => 192.168.104.79:80 (active)
                                             20 => 192.168.103.127:80 (active)
                                             21 => 192.168.119.76:80 (active)
                                             22 => 192.168.156.198:80 (active)
                                             23 => 192.168.154.30:80 (active)
                                             24 => 192.168.139.221:80 (active)
                                             25 => 192.168.139.219:80 (active)
                                             26 => 192.168.149.197:80 (active)
                                             27 => 192.168.104.1:80 (active)
                                             28 => 192.168.144.122:80 (active)
                                             29 => 192.168.109.242:80 (active)
                                             30 => 192.168.110.119:80 (active)
                                             31 => 192.168.117.57:80 (active)
                                             32 => 192.168.144.64:80 (active)
                                             33 => 192.168.152.12:80 (active)
                                             34 => 192.168.145.140:80 (active)
                                             35 => 192.168.124.197:80 (active)
                                             36 => 192.168.105.136:80 (active)
                                             37 => 192.168.117.59:80 (active)
                                             38 => 192.168.125.195:80 (active)
102   192.168.141.217:31170   NodePort       1 => 192.168.140.54:80 (active)
                                             2 => 192.168.119.204:80 (active)
                                             3 => 192.168.152.88:80 (active)
                                             4 => 192.168.153.160:80 (active)
                                             5 => 192.168.115.146:80 (active)
                                             6 => 192.168.120.176:80 (active)
                                             7 => 192.168.121.132:80 (active)
                                             8 => 192.168.99.58:80 (active)
                                             9 => 192.168.141.24:80 (active)
                                             10 => 192.168.105.30:80 (active)
                                             11 => 192.168.124.44:80 (active)
                                             12 => 192.168.147.200:80 (active)
                                             13 => 192.168.153.19:80 (active)
                                             14 => 192.168.111.137:80 (active)
                                             15 => 192.168.132.236:80 (active)
                                             16 => 192.168.138.53:80 (active)
                                             17 => 192.168.107.77:80 (active)
                                             18 => 192.168.132.120:80 (active)
                                             19 => 192.168.104.79:80 (active)
                                             20 => 192.168.103.127:80 (active)
                                             21 => 192.168.119.76:80 (active)
                                             22 => 192.168.156.198:80 (active)
                                             23 => 192.168.154.30:80 (active)
                                             24 => 192.168.139.221:80 (active)
                                             25 => 192.168.139.219:80 (active)
                                             26 => 192.168.149.197:80 (active)
                                             27 => 192.168.104.1:80 (active)
                                             28 => 192.168.144.122:80 (active)
                                             29 => 192.168.109.242:80 (active)
                                             30 => 192.168.110.119:80 (active)
                                             31 => 192.168.117.57:80 (active)
                                             32 => 192.168.144.64:80 (active)
                                             33 => 192.168.152.12:80 (active)
                                             34 => 192.168.145.140:80 (active)
                                             35 => 192.168.124.197:80 (active)
                                             36 => 192.168.105.136:80 (active)
                                             37 => 192.168.117.59:80 (active)
                                             38 => 192.168.125.195:80 (active)
103   192.168.157.251:31170   NodePort       1 => 192.168.140.54:80 (active)
                                             2 => 192.168.119.204:80 (active)
                                             3 => 192.168.152.88:80 (active)
                                             4 => 192.168.153.160:80 (active)
                                             5 => 192.168.115.146:80 (active)
                                             6 => 192.168.120.176:80 (active)
                                             7 => 192.168.121.132:80 (active)
                                             8 => 192.168.99.58:80 (active)
                                             9 => 192.168.141.24:80 (active)
                                             10 => 192.168.105.30:80 (active)
                                             11 => 192.168.124.44:80 (active)
                                             12 => 192.168.147.200:80 (active)
                                             13 => 192.168.153.19:80 (active)
                                             14 => 192.168.111.137:80 (active)
                                             15 => 192.168.132.236:80 (active)
                                             16 => 192.168.138.53:80 (active)
                                             17 => 192.168.107.77:80 (active)
                                             18 => 192.168.132.120:80 (active)
                                             19 => 192.168.104.79:80 (active)
                                             20 => 192.168.103.127:80 (active)
                                             21 => 192.168.119.76:80 (active)
                                             22 => 192.168.156.198:80 (active)
                                             23 => 192.168.154.30:80 (active)
                                             24 => 192.168.139.221:80 (active)
                                             25 => 192.168.139.219:80 (active)
                                             26 => 192.168.149.197:80 (active)
                                             27 => 192.168.104.1:80 (active)
                                             28 => 192.168.144.122:80 (active)
                                             29 => 192.168.109.242:80 (active)
                                             30 => 192.168.110.119:80 (active)
                                             31 => 192.168.117.57:80 (active)
                                             32 => 192.168.144.64:80 (active)
                                             33 => 192.168.152.12:80 (active)
                                             34 => 192.168.145.140:80 (active)
                                             35 => 192.168.124.197:80 (active)
                                             36 => 192.168.105.136:80 (active)
                                             37 => 192.168.117.59:80 (active)
                                             38 => 192.168.125.195:80 (active)
  • At the same time, you can verify using iptables in the host namespace (on the node), there are no iptables rules for the service are present: 
iptables-save | grep KUBE-SVC
  • Last but not least, a simple curl test shows connectivity for the exposed NodePort port 31170 as well as for the ClusterIP: 
curl 127.0.0.1:31170 -I
HTTP/1.1 200 OK
Server: nginx/1.27.0
Date: Tue, 16 Jul 2024 11:56:57 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 May 2024 13:22:30 GMT
Connection: keep-alive
ETag: "6655da96-267"
Accept-Ranges: bytes

 curl 10.100.170.109 -I
HTTP/1.1 200 OK
Server: nginx/1.27.0
Date: Tue, 16 Jul 2024 11:58:08 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 May 2024 13:22:30 GMT
Connection: keep-alive
ETag: "6655da96-267"
Accept-Ranges: bytes

curl 192.168.136.122:31170 -I
HTTP/1.1 200 OK
Server: nginx/1.27.0
Date: Tue, 16 Jul 2024 11:59:32 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 May 2024 13:22:30 GMT
Connection: keep-alive
ETag: "6655da96-267"
Accept-Ranges: bytes

curl  192.168.141.217:31170 -I
HTTP/1.1 200 OK
Server: nginx/1.27.0
Date: Tue, 16 Jul 2024 11:59:45 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 May 2024 13:22:30 GMT
Connection: keep-alive
ETag: "6655da96-267"
Accept-Ranges: bytes

curl 192.168.157.251:31170 -I
HTTP/1.1 200 OK
Server: nginx/1.27.0
Date: Tue, 16 Jul 2024 11:59:57 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 28 May 2024 13:22:30 GMT
Connection: keep-alive
ETag: "6655da96-267"
Accept-Ranges: bytes

Hubble CLI

Hubble’s CLI extends the Visibility that is provided by standard kubectl commands like kubectl get pods to give you more network-level details about a request, such as its status and the security identities associated with its source and destination.

The Hubble CLI can be leveraged to observe network flows from Cilium agents. Users can observe the flows from their local machine workstation for troubleshooting or monitoring. For this tutorial, users can see that all hubble outputs are related to the tests that are done above. Users can try other tests and see the same results with different values as expected.

Setup Hubble Relay Forwarding

Use the kubectl port forward to hubble-relay, then edit the hubble config to point at the remote hubble server component.

kubectl port-forward -n kube-system svc/hubble-relay --address 0.0.0.0 4245:80

Hubble Status

Hubble status can check the overall health of Hubble within your cluster. If using Hubble Relay, a counter for the number of connected Nodes will appear in the last line of the output.

hubble status

Healthcheck (via localhost:4245): Ok
Current/Max Flows: 8,190/8,190 (100.00%)
Flows/s: 31.18
Connected Nodes: 2/2

View Last N Events

hubble observe displays the most recent events based on the number filter. Hubble Relay will display events over all the connected Nodes:

hubble observe --last 200

Jul 15 09:50:22.422: 172.31.46.109:46138 (host) -> 3.104.69.134:443 (world) to-network FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:22.641: default/client:37505 (ID:3623) -> kube-system/coredns-57d946db4c-sf42g:53 (ID:334) to-overlay FORWARDED (UDP)
Jul 15 09:50:22.641: 172.31.46.109:51871 (host) -> 172.31.20.222:51871 (remote-node) to-network FORWARDED (UDP)
Jul 15 09:50:22.642: default/client:37505 (ID:3623) <- kube-system/coredns-57d946db4c-sf42g:53 (ID:334) to-endpoint FORWARDED (UDP)
Jul 15 09:50:22.643: default/client:58052 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: SYN)
Jul 15 09:50:22.643: default/client:58052 (ID:3623) <- default/server:80 (ID:14484) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Jul 15 09:50:22.643: default/client:58052 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK)
Jul 15 09:50:22.643: default/client:58052 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:22.644: default/client:58052 (ID:3623) <- default/server:80 (ID:14484) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:22.645: default/client:58052 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK, FIN)
Jul 15 09:50:22.645: default/client:58052 (ID:3623) <- default/server:80 (ID:14484) to-endpoint FORWARDED (TCP Flags: ACK, FIN, PSH)
Jul 15 09:50:22.645: default/client:58052 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: RST)
Jul 15 09:50:22.645: default/client:58052 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: RST)
Jul 15 09:50:22.804: tenant-jobs/kafka-0:52640 (ID:19797) -> tenant-jobs/zookeeper-774bffb67f-x58cg:2181 (ID:55778) to-stack FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:22.805: tenant-jobs/kafka-0:52640 (ID:19797) -> tenant-jobs/zookeeper-774bffb67f-x58cg:2181 (ID:55778) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:22.805: tenant-jobs/kafka-0:52640 (ID:19797) <- tenant-jobs/zookeeper-774bffb67f-x58cg:2181 (ID:55778) to-stack FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:22.805: tenant-jobs/kafka-0:52640 (ID:19797) <- tenant-jobs/zookeeper-774bffb67f-x58cg:2181 (ID:55778) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Jul 15 09:50:23.852: cilium-system/hubble-relay-7cc6cd8bfc-rws9l:34394 (ID:52364) <- 172.31.46.109:4244 (host) to-endpoint FORWARDED (TCP Flags: ACK)
Jul 15 09:50:23.852: cilium-system/hubble-relay-7cc6cd8bfc-rws9l:34394 (ID:52364) -> 172.31.46.109:4244 (host) to-stack FORWARDED (TCP Flags: ACK)

Follow Events in Real-Time

hubble observe --follow will follow the event stream for all connected clusters.

hubble observer --follow

Jul 15 09:51:29.874: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:45944 (ID:12210) -> kube-system/coredns-57d946db4c-54hgq:53 (ID:334) policy-verdict:L3-Only INGRESS ALLOWED (UDP)
Jul 15 09:51:29.874: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:45944 (ID:12210) -> kube-system/coredns-57d946db4c-54hgq:53 (ID:334) to-endpoint FORWARDED (UDP)
Jul 15 09:51:29.874: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:45944 (ID:12210) <- kube-system/coredns-57d946db4c-54hgq:53 (ID:334) to-overlay FORWARDED (UDP)
Jul 15 09:51:29.875: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:45944 (ID:12210) <- kube-system/coredns-57d946db4c-54hgq:53 (ID:334) to-proxy FORWARDED (UDP)
Jul 15 09:51:29.875: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:45944 (ID:12210) <- kube-system/coredns-57d946db4c-54hgq:53 (ID:334) dns-response proxy FORWARDED (DNS Answer  TTL: 4294967295 (Proxy coreapi.tenant-jobs.svc.cluster.local. AAAA))
Jul 15 09:51:29.875: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:45944 (ID:12210) <- kube-system/coredns-57d946db4c-54hgq:53 (ID:334) dns-response proxy FORWARDED (DNS Answer "10.100.228.206" TTL: 5 (Proxy coreapi.tenant-jobs.svc.cluster.local. A))
Jul 15 09:51:29.876: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:58748 (ID:12210) -> tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) policy-verdict:L3-Only EGRESS ALLOWED (TCP Flags: SYN)
Jul 15 09:51:29.876: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:58748 (ID:12210) -> tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) to-stack FORWARDED (TCP Flags: SYN)
Jul 15 09:51:29.876: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:58748 (ID:12210) -> tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) policy-verdict:L4-Only INGRESS ALLOWED (TCP Flags: SYN)

Troubleshooting HTTP & DNS

  • Suppose you have a CiliumNetworkPolicy that enforces DNS or HTTP policy. In that case, you can use the --type l7 filtering options for hubble to check our applications’ HTTP methods and DNS resolution attempts.
hubble observe --since 1m -t l7

Jul 15 09:51:48.189: tenant-jobs/crawler-64657c648d-kq885:34034 (ID:1486) -> kube-system/coredns-57d946db4c-sf42g:53 (ID:334) dns-request proxy FORWARDED (DNS Query api.twitter.com.svc.cluster.local. AAAA)
Jul 15 09:51:48.189: tenant-jobs/crawler-64657c648d-kq885:34034 (ID:1486) -> kube-system/coredns-57d946db4c-sf42g:53 (ID:334) dns-request proxy FORWARDED (DNS Query api.twitter.com.svc.cluster.local. A)
Jul 15 09:51:48.192: tenant-jobs/crawler-64657c648d-kq885:34034 (ID:1486) <- kube-system/coredns-57d946db4c-sf42g:53 (ID:334) dns-response proxy FORWARDED (DNS Answer RCode: Non-Existent Domain TTL: 4294967295 (Proxy api.twitter.com.svc.cluster.local. A))
  • You can use --http-status to view specific flows with 200 HTTP responses
hubble observe --http-status 200

Jul 15 09:52:39.618: tenant-jobs/coreapi-8689bfdc87-74h6f:40246 (ID:1966) <- tenant-jobs/elasticsearch-55f56859c9-pvpdh:9200 (ID:16826) http-response FORWARDED (HTTP/1.1 200 3ms (GET http://elasticsearch:9200/applicants/_search?request_cache=false))
Jul 15 09:52:39.622: tenant-jobs/recruiter-69f947cb7c-6ztrb:44192 (ID:16148) <- tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) http-response FORWARDED (HTTP/1.1 200 9ms (GET http://coreapi:9080/applicants))
Jul 15 09:52:39.946: tenant-jobs/coreapi-8689bfdc87-74h6f:40246 (ID:1966) <- tenant-jobs/elasticsearch-55f56859c9-pvpdh:9200 (ID:16826) http-response FORWARDED (HTTP/1.1 200 10ms (GET http://elasticsearch:9200/jobs/_search?request_cache=false))
Jul 15 09:52:39.952: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:50400 (ID:12210) <- tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) http-response FORWARDED (HTTP/1.1 200 18ms (GET http://coreapi:9080/jobs))
  • You can also show HTTP GET methods with --http-method
hubble observe --http-method GET

Jul 15 10:29:51.412: tenant-jobs/recruiter-69f947cb7c-6ztrb:37256 (ID:16148) <- tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) http-response FORWARDED (HTTP/1.1 200 6ms (GET http://coreapi:9080/applicants))
Jul 15 10:29:51.744: tenant-jobs/jobposting-7c77d8bd8d-rpl8c:33008 (ID:12210) -> tenant-jobs/coreapi-8689bfdc87-74h6f:9080 (ID:1966) http-request FORWARDED (HTTP/1.1 GET http://coreapi:9080/jobs)
  • To view DNS traffic for a specific FQDN, you can use the --to-fqdn flag
hubble observe --to-fqdn "*.github.com"

hubble observe --to-fqdn "*.github.com"
Jul 15 10:32:22.981: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) policy-verdict:L3-Only EGRESS ALLOWED (TCP Flags: SYN)
Jul 15 10:32:22.981: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) to-stack FORWARDED (TCP Flags: SYN)
Jul 15 10:32:22.983: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) to-stack FORWARDED (TCP Flags: ACK)
Jul 15 10:32:22.985: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) to-stack FORWARDED (TCP Flags: ACK, PSH)
Jul 15 10:32:23.025: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) to-stack FORWARDED (TCP Flags: ACK, FIN)
Jul 15 10:32:23.026: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) to-stack FORWARDED (TCP Flags: RST)
Jul 15 10:32:23.026: default/mediabot:43754 (ID:3004) -> api.github.com:443 (ID:16777222) to-stack FORWARDED (TCP Flags: RST)

Filter by Verdict

Hubble provides a field called VERDICT that displays one of FORWARDED, ERROR, or DROPPED for each flow. DROPPED could indicate an unsupported protocol within the underlying platform or Network Policy enforcing pod communication. Hubble can introspect the reason for ERROR or DROPPED flows and display the reason within the TYPE field of each flow.

hubble observe --output table --verdict DROPPED

Jul 15 10:33:28.403   default/mediabot:54980      support.github.com:443   policy-verdict:none EGRESS   DENIED    TCP Flags: SYN
Jul 15 10:33:28.403   default/mediabot:54980      support.github.com:443   Policy denied                DROPPED   TCP Flags: SYN
Jul 15 10:33:28.815   fe80::58ed:a5ff:fe75:dce5   ff02::2                  Unsupported L3 protocol      DROPPED   ICMPv6 RouterSolicitation
Jul 15 10:33:29.423   default/mediabot:54980      support.github.com:443   policy-verdict:none EGRESS   DENIED    TCP Flags

Filter by Pod or Namespace

  • To show all flows for a specific pod, filter with the --pod flag
hubble observe --pod default/server

Jul 15 10:33:43.235: default/client:35780 (ID:3623) <- default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: SYN, ACK)
Jul 15 10:33:43.235: default/client:35780 (ID:3623) <- default/server:80 (ID:14484) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Jul 15 10:33:43.236: default/client:35780 (ID:3623) <- default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Jul 15 10:33:43.236: default/client:35780 (ID:3623) <- default/server:80 (ID:14484) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
  • If you are only interested in traffic from a pod to a specific destination, combine --from-pod and --to-pod
hubble observe --from-pod default/client --to-pod default/server

Jul 15 10:34:15.360: default/client:59970 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: SYN)
Jul 15 10:34:15.361: default/client:59970 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK)
Jul 15 10:34:15.361: default/client:59970 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK, PSH)
Jul 15 10:34:15.363: default/client:59970 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK, FIN)
  • If you want to see all traffic from a specific namespace, specify the --from-namespace
hubble observe --from-namespace default

Jul 15 10:34:39.447: default/client:45940 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: ACK, RST)
Jul 15 10:34:39.447: default/client:45940 (ID:3623) -> default/server:80 (ID:14484) to-endpoint FORWARDED (TCP Flags: ACK, RST)
Jul 15 10:34:41.450: default/client:56308 (ID:3623) -> kube-system/coredns-57d946db4c-sf42g:53 (ID:334) to-overlay FORWARDED (UDP)
Jul 15 10:34:41.451: default/client:56308 (ID:3623) -> kube-system/coredns-57d946db4c-sf42g:53 (ID:334) policy-verdict:L3-Only INGRESS ALLOWED (UDP)
Jul 15 10:34:41.451: default/client:56308 (ID:3623) -> kube-system/coredns-57d946db4c-sf42g:53 (ID:334) to-endpoint FORWARDED (UDP)
Jul 15 10:34:41.452: default/client:45954 (ID:3623) -> default/server:80 (ID:14484) to-overlay FORWARDED (TCP Flags: SYN)
Jul 15 10:34:41.452: default/client:45954 (ID:3623) -> default/server:80 (ID:14484) policy-verdict:L3-Only INGRESS ALLOWED (TCP Flags: SYN)

Filter Events with JQ

To view filter events through the jq tool, you can swap the output to json mode. Visualize your metadata through jq, which will help you see more metadata around the workload labels like pod name/namespace assigned to both source and destination. This information is accessible by Cilium because it is encoded in the packets based on pod identities.

hubble observe --output json | jq . | head -n 50

{
  "flow": {
    "time": "2024-07-15T10:35:01.987421580Z",
    "uuid": "31b80f09-b5e0-4abb-b20d-cf2a33fd0138",
    "verdict": "FORWARDED",
    "ethernet": {
      "source": "96:54:e3:1f:60:da",
      "destination": "aa:ae:77:f0:b9:20"
    },
    "IP": {
      "source": "172.31.36.30",
      "destination": "172.31.36.67",
      "ipVersion": "IPv4"
    },
    "l4": {
      "TCP": {
        "source_port": 48008,
        "destination_port": 9080,
        "flags": {
          "SYN": true
        }
      }
    },
    "source": {
      "ID": 219,
      "identity": 12210,
      "namespace": "tenant-jobs",
      "labels": [
        "k8s:app=jobposting",
        "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=tenant-jobs",
        "k8s:io.cilium.k8s.policy.cluster=default",
        "k8s:io.cilium.k8s.policy.serviceaccount=default",
        "k8s:io.kubernetes.pod.namespace=tenant-jobs"
      ],
      "pod_name": "jobposting-7c77d8bd8d-rpl8c",
      "workloads": [
        {
          "name": "jobposting",
          "kind": "Deployment"
        }
      ]
    },
    "destination": {
      "ID": 613,
      "identity": 1966,
      "namespace": "tenant-jobs",
      "labels": [
        "k8s:app=coreapi",
        "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=tenant-jobs",
        "k8s:io.cilium.k8s.policy.cluster=default",

Hubble UI

Note: To obtain the helm values to install Hubble UI and access the Enterprise documentation, contact sales@isovalent.com.

The graphical user interface (Hubble-UI) utilizes relay-based Visibility to provide a graphical service dependency and connectivity map. Hubble-UI is enabled via helm charts. The feature is not enabled when you create a new cluster using Isovalent Enterprise for Cilium or upgrade to Isovalent Enterprise for Cilium.

Once the installation is complete, you will notice hubble-ui running.

kubectl get pods -n hubble-ui

NAME                         READY   STATUS    RESTARTS   AGE
hubble-ui-7b4c87fb48-2cpzz   2/2     Running   0          3d22h

Validate the installation

To access Hubble UI, forward a local port to the Hubble UI service:

kubectl port-forward -n hubble-ui svc/hubble-ui 12000:80

Then, open http://localhost:12000 in your browser: 

Select the namespace by default, and you will observe a service map and network event list. In this case, the pod mediabot (created in the previous test case) is trying to access support.github.com over port 443

Note- You can read more on hubble in a detailed blog post that is a 3-part series.

DNS Proxy HA

Cilium Enterprise supports deploying an additional DNS Proxy, Daemonset called cilium-dnsproxy that can be life-cycled independently of Cilium Daemonset.

What is Cilium DNS Proxy HA?

When Cilium Network Policies that make use of toFQDNs are installed in a Kubernetes cluster, the Cilium agent starts an in-process DNS proxy that becomes responsible for proxying all DNS requests between all Pods and the Kubernetes internal kube-dns service. Whenever a Cilium agent is restarted, such as during an upgrade or due to something unexpected, DNS requests from all Pods on that node do not succeed until the Cilium agent is online again.

When cilium-dnsproxy is enabled, an independently life-cycled DaemonSet is deployed. cilium-dnsproxy acts as a hot standby that mirrors DNS policy rules. cilium-agent and cilium-dnsproxy bind to the same port, relying on the Linux kernel to distribute DNS traffic between the two DNS proxy instances. This allows you to lifecycle either cilium or cilium-dnsproxy Daemonset without impacting DNS traffic.

Installation of DNS-Proxy HA using helm

Note-

  • DNS Proxy High Availability relies on configuring the cilium-config ConfigMap with external-dns-proxy: true and to deploy the DNS proxy component.

Once the installation is complete, you will notice cilium-dnsproxy running as a Daemonset, and also the Pods are up and running:

kubectl get ds -A

NAMESPACE       NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
cilium-system   cilium                   2         2         2       2            2           kubernetes.io/os=linux            75m
cilium-system   cilium-dnsproxy          2         2         2       2            2           <none>                            27s
cilium-system   cilium-license-manager   2         2         2       2            2           <none>                            75m
cilium-system   cilium-node-init         2         2         2       2            2           kubernetes.io/os=linux            75m
cilium-system   hubble-enterprise        2         2         2       2            2           <none>                            6d4h
kube-system     aws-node                 0         0         0       0            0           io.cilium/aws-node-enabled=true   11d


kubectl get pods -A -n cilium-system | grep "cilium-dnsproxy-*"

cilium-system   cilium-dnsproxy-kkptf             1/1     Running   0              43s
cilium-system   cilium-dnsproxy-n4l9p             1/1     Running   0              43s

Validate DNS-Proxy HA

  • In line with our Star Wars theme examples, you can use a simple scenario where the Empire’s mediabot Pods need access to GitHub to manage the Empire’s git repositories. The Pods shouldn’t have access to any other external service.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-sw-app.yaml
pod/mediabot created
  • Apply DNS Egress Policy. The following Cilium network policy allows mediabot Pods to only access api.github.com
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-matchname.yaml
ciliumnetworkpolicy.cilium.io/fqdn created
  • Testing the policy, you can see that mediabot pod has access to api.github.com 
kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/1.1 200 OK
  • Send packets continuously in a loop.
while true; do kubectl exec mediabot -- curl -I -s https://api.github.com | head -1; sleep 10; done
  • Simulate a failure scenario where in the cilium-agent Pods are not up and running and the traffic still goes through the cilium-dnsproxy-* Pods
  • cilium-agent Pods are restarted as a part of the test
  • Traffic is not disrupted and continues to flow through the cilium-dnsproxy Pods

Tetragon

Tetragon provides powerful security observability and a real-time runtime enforcement platform. The creators of Cilium have built Tetragon and brought the full power of eBPF to the security world.

Tetragon helps platform and security teams solve the following:

Security Observability:

  • Observing application and system behavior such as process, syscall, file, and network activity
  • Tracing namespace, privilege, and capability escalations
  • File integrity monitoring

Runtime Enforcement:

  • Application of security policies to limit the privileges of applications and processes on a system (system calls, file access, network, kprobes)

Tetragon has been specifically built for Kubernetes and cloud-native infrastructure but can be run on any Linux system. Sometimes, you might want to enable Process Visibility in an environment without Cilium as the CNI. The Security Observability and Runtime Enforcement provided by the Hubble-Enterprise Daemonset can operate in a standalone mode, de-coupled from Cilium as a CNI.

Installation of Tetragon using helm

Note To obtain the helm values to install Tetragon Enterprise and access Enterprise documentation, contact sales@isovalent.com.

Once the installation is complete, you will notice Tetragon running as a Daemonset, and also, the Pods are up and running:

kubectl get ds -A

NAMESPACE       NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
cilium-system   cilium                   2         2         2       2            2           kubernetes.io/os=linux            75m
cilium-system   cilium-dnsproxy          2         2         2       2            2           <none>                            27s
cilium-system   cilium-license-manager   2         2         2       2            2           <none>                            75m
cilium-system   cilium-node-init         2         2         2       2            2           kubernetes.io/os=linux            75m
cilium-system   hubble-enterprise        2         2         2       2            2           <none>                            6d4h
kube-system     aws-node                 0         0         0       0            0           io.cilium/aws-node-enabled=true   11d

kubectl get pods -A -n kube-system | grep "hubble-enterprise"

cilium-system   hubble-enterprise-fpk2q           2/2     Running   0               6d4h
cilium-system   hubble-enterprise-r2wcf           2/2     Running   0               6d4h

Validate Tetragon

You can use our Demo Application to explore the Process and Networking Events:
  • Create a namespace
kubectl create namespace tenant-jobs
  • Deploy a demo application. Verify that all Pods are up and running.
kubectl get pods -n tenant-jobs

NAME                             READY   STATUS    RESTARTS      AGE
coreapi-8689bfdc87-74h6f         1/1     Running   3 (78m ago)   79m
crawler-64657c648d-kq885         1/1     Running   0             79m
elasticsearch-55f56859c9-pvpdh   1/1     Running   0             79m
jobposting-7c77d8bd8d-rpl8c      1/1     Running   0             79m
kafka-0                          1/1     Running   0             79m
loader-68bdcc9d77-6bhgm          1/1     Running   1 (78m ago)   79m
recruiter-69f947cb7c-6ztrb       1/1     Running   0             79m
zookeeper-774bffb67f-x58cg       1/1     Running   0             79m
  • You can view all the Pods in the tenant-jobs namespace in hubble-ui
  • You can examine the Process and Networking Events in two different ways:
    • Raw Json events
      • kubectl logs -n kube-system ds/hubble-enterprise -c export-stdout -f
    • Enable Hubble UI– The second way is to visualize the processes running on a certain workload by observing their Process Ancestry Tree. This tree gives you rich Kubernetes API, identity-aware metadata, and OS-level Process Visibility about the executed binary, its parents, and the execution time until the Container Runtime has started the container.
    • While in a real-world deployment, the Hubble Event Data would likely be exported to an SIEM or other logging datastore, in this quickstart, you will access this Event Data by redirecting the logs of the export-stdout container of the hubble-enterprise pod:
      • kubectl logs -n kube-system ds/hubble-enterprise -c export-stdout > export.log
  • From the main Hubble UI screen, click on the tenant-jobs namespace in the list. Then, in the left navigation sidebar, click Processes.
  • To upload the exported logs, click Upload on the left of the screen:
    • Note: This manual Tetragon event export is not required when used with Hubble Timescape.
  • Use the file selector dialog to choose the events.log generated earlier and select the tenants-job namespace from the namespace dropdown.
  • Here, you can get a brief overview of a security use case that can easily be detected and be interesting to visualize. By using Hubble UI and visualizing the Process Ancestry Tree, you can detect a shell execution in the crawler-YYYYYYYYY-ZZZZZ Pod that occurred more than 5 minutes after the container has started. After clicking on the crawler-YYYYYYYY-ZZZZZ pod name from the Pods selector dropdown on the left of the screen, you will be able to see the Process Ancestry Tree for that pod:
  • The Process Ancestry Tree gives us:
    • Rich Kubernetes Identity-Aware Metadata: In the first row, you can see the name of the team or namespace and the specific application service to be inspected.
    • OS-Level Process Visibility: You can see all the processes that have been executed on the inspected service or were related to its Pod lifecycle 
    • DNS Aware Metadata: You can see all the external connections with the exact DNS name as an endpoint made from specific processes of the inspected service.
Enabling DNS Visibility

Outbound network traffic remains a major attack vector for many enterprises. For example, in the above example, you can see that the crawler service reaches out to one or more services outside the Kubernetes cluster on port 443. However, the identity of these external services is unknown, as the flows only show an IP address.

Cilium Enterprise can parse the DNS-layer requests emitted by services and associate that identity data with outgoing connections, enriching network connectivity logs.

To inspect the DNS lookups for Pods within a namespace, you must apply a network policy that tells Cilium to inspect port 53 traffic from Pods to Kube-DNS at Layer 7. With this DNS Visibility, Hubble flow data will be annotated with DNS service identity for destinations outside the Kubernetes cluster. The demo app keeps reaching out to Twitter at regular intervals, and now both the service map and flows table for the tenant-jobs namespace shows connections to api.twitter.com:

Enabling HTTP Visibility

Like enabling DNS Visibility, you can also apply a Visibility policy to instruct Cilium to inspect certain traffic flows at the application layer (e.g., HTTP) without requiring any changes to the application itself.

In this example, you’ll inspect ingress connections to services within the tenant-jobs namespace at the HTTP layer. You can inspect flow details to get application-layer information. For example, you can inspect HTTP queries that coreapi make it to elasticsearch:

Network Policy

In this scenario, you will create services, deployments, and Pods (created in the above step (Observing Network Flows with Hubble CLI)   and then use the Hubble UI’s network policy editor and service map/flow details pages to create, troubleshoot, and update a network policy.

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15/examples/minikube/http-sw-app.yaml
Apply an L3/L4 Policy (Optional as this is covered in the L3/ L4 Policy section)
  • You can start with a basic policy restricting deathstar landing requests to only the ships that have a label (org=empire). This will not allow any ships that don’t have the org=empire label to even connect with the deathstar service. This simple policy filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), often called an L3/L4 network security policy.
  • The above policy whitelists traffic sent from Pods with the label (org=empire) to deathstar Pods with the label (org=empire, class=deathstar) on TCP port 80.
  • To apply this L3/L4 policy, run:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.13/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
  • If you run the landing requests again, only the tiefighter Pods with the label org=empire will succeed. The xwing Pods will be blocked.
kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
Ship landed
  • The same request ran from an xwing pod will fail:
kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing

This request will hang, so press Control-C to kill the curl request or wait for it to time out. As you can see, Hubble-UI reports that the flow will be dropped.

  • Using the policy editor, click on the denied/dropped flows and add them to your policy.
  • You need to download the policy and apply/update it again.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: rule1
  namespace: default
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: deathstar
  ingress:
    - fromEndpoints:
        - matchLabels:
            org: empire
      toPorts:
        - ports:
            - port: "80"
              protocol: TCP
    - fromEndpoints:
        - matchLabels:
            k8s:app.kubernetes.io/name: xwing
            k8s:class: xwing
            k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name: default
            k8s:io.kubernetes.pod.namespace: default
            k8s:org: alliance
      toPorts:
        - ports:
            - port: "80"
kubectl apply -f rule1.yaml
ciliumnetworkpolicy.cilium.io/rule1 configured
  • Once the updated policy is applied, run the same request from a xwing pod, and that should pass
Enabling TLS Visibility

Tetragon’s Network Security Observability provides deep insights into network events, such as tls events containing information about the exact TLS connections, including the negotiated cipher suite, the TLS version, the source/destination IP addresses, and the connection ports made by the initial process. In addition, tls events are also enriched by Kubernetes Identity-Aware metadata (Kubernetes namespace, pod/container name, labels, container image, etc).

By default, Tetragon does not show connectivity-related events. TLS Visibility requires a specific tracing policy and a CRD, which we will apply.

Apply a TLS Visibility CRD:

Note- Contact sales@isovalent.com to get access to the TLS Visibility CRD.

  • Let’s see the events we get out of the box. You can use the same Pods that were created in the previous step.
  • Create a TLS Visibility Policy
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: tls-visibility-policy
spec:
  parser:
    tls:
      enable: true
      mode: socket
      selectors:
        - matchPorts:
            - 443
    tcp:
      enable: true
  • Apply the Policy:
kubectl apply -f tls_parser.yaml
  • Send traffic to see the TLS ciphers and versions being used. From the xwing pod shell, try a simple curl to google.com and use openssl to check ciphers:
curl --silent --output /dev/null --show-error --fail https://www.google.com

openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES128-SHA
CONNECTED(00000003)

openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES256-SHA
CONNECTED(00000003)
  • Check the events in a different terminal:
kubectl exec -it -n kube-system daemonsets/hubble-enterprise -c enterprise -- \
  hubble-enterprise getevents -o compact --pods xwing

🚀 process default/xwing /usr/bin/openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES128-SHA
🔌 connect default/xwing /usr/bin/openssl TCP 10.241.0.20:54552 => 104.154.89.105:443
🔐 tls     default/xwing /usr/bin/openssl 104.154.89.105:443  TLS1.2 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
💥 exit    default/xwing /usr/bin/openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES128-SHA SIGINT
🧹 close   default/xwing /usr/bin/openssl TCP

10.241.0.20:54552 => 104.154.89.105:443 tx 441 B rx 2.4 kB
🚀 process default/xwing /usr/bin/openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES256-SHA
🔌 connect default/xwing /usr/bin/openssl TCP 10.241.0.20:45918 => 104.154.89.105:443
🔐 tls     default/xwing /usr/bin/openssl 104.154.89.105:443  TLS1.2 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
💥 exit    default/xwing /usr/bin/openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES256-SHA 0
🧹 close   default/xwing /usr/bin/openssl TCP 10.241.0.20:45918 => 104.154.89.105:443 tx 494 B rx 2.5 kB
🚀 process default/xwing /usr/bin/openssl s_client -connect badssl.com:443 -cipher DHE-RSA-AES256-SHA
🔌 connect default/xwing /usr/bin/openssl TCP 10.241.0.20:45526 => 104.154.89.105:443
🔐 tls     default/xwing /usr/bin/openssl 104.154.89.105:443  TLS1.2 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  • You can also view the above results in a tabular format by using ‘jq’:
kubectl exec -n kube-system daemonsets/hubble-enterprise -c enterprise -- \
  hubble-enterprise getevents -o json --pods xwing | jq -r '.tls.negotiated_version | select ( . != null)'

TLS1.2
TLS1.2
TLS1.2
File Enforcement

Tetragon Enterprise monitors files using information about the kernel’s internal structures. To configure File Enforcement, apply an example File Enforcement Policy, which blocks any write access on /etc/passwd and /etc/shadow and blocks deleting these files. This policy applies to host files and all Kubernetes Pods in the default Kubernetes namespace.

Note- contact sales@isovalent.com to get access to the specific file enforcement policies.

Using one of the already existing Pods, you’ll see events from a test pod on the default Kubernetes namespace when trying to edit the /etc/passwd file: 

{"process_exec":{"process":{"exec_id":"YWtzLWF#####################3MwMDAwMDA6MTM5OT################jE0", "pid":212614, "uid":0, "cwd":"/", "binary":"/usr/bin/vi", "arguments":"/etc/passwd", "flags":"execve rootcwd clone", "start_time":"2023-12-05T10:49:10.927995393Z", "auid":4294967295, "pod":{"namespace":"default", "name":"xwing", "labels":["k8s:app.kubernetes.io/name=xwing", "k8s:class=xwing", "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default", "k8s:io.cilium.k8s.policy.cluster=default", "k8s:io.cilium.k8s.policy.serviceaccount=default", "k8s:io.kubernetes.pod.namespace=default", "k8s:org=alliance"], "container":{"id":"containerd://af3a386beef49967b092cab401347861ff41b000719fa4d33d401a8b91e45393", "name":"spaceship", "image":{"id":"docker.io/tgraf/netperf@sha256:8e86f744bfea165fd4ce68caa05abc96500f40130b857773186401926af7e9e6", "name":"docker.io/tgraf/netperf:latest"}, "start_time":"2023-12-05T10:48:24Z", "pid":13}, "pod_labels":{"app.kubernetes.io/name":"xwing", "class":"xwing", "org":"alliance"}}, "docker":"af3a386beef49967b092cab40134786",
Process Visibility

Tetragon Enterprise uses information about the internal structures of the kernel to provide process Visibility. Run kubectl to validate that Tetragon Enterprise is configured with process Visibility enabled:

kubectl exec -n kube-system ds/hubble-enterprise -c enterprise -- hubble-enterprise getevents

{"process_exit":{"process":{"exec_id":"aXAtMTcyLTMxLTQ2LTEwOS5hcC1zb3V0aGVhc3QtMi5jb21wdXRlLmludGVybmFsOjk2MjUwNTM3Nzk5MDI2NzoxNTk0Njkx", "pid":1594691, "uid":0, "cwd":"/", "binary":"/bin/wget", "arguments":"server", "flags":"execve rootcwd", "start_time":"2024-07-15T10:51:37.838236577Z", "auid":4294967295, "pod":{"namespace":"default", "name":"client", "labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default", "k8s:io.cilium.k8s.policy.cluster=default", "k8s:io.cilium.k8s.policy.serviceaccount=default", "k8s:io.kubernetes.pod.namespace=default", "k8s:name=client", "k8s:post=eks-wireguard-cilium"], "container":{"id":"containerd://49778fac295c480cd2280d3e0ce8f7372cee760e043efbdae6a9c4ebfbdd3605", "name":"client", "image":{"id":"docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7", "name":"docker.io/library/busybox:latest"}, "start_time":"2024-07-15T09:32:59Z", "pid":2358}, "pod_labels":{"name":"client", "post":"eks-wireguard-cilium"}}, "docker":"49778fac295c480cd2280d3e0ce8f73", "parent_exec_id":"aXAtMTcyLTMxLTQ2LTEwOS5hcC1zb3V0aGVhc3QtMi5jb21wdXRlLmludGVybmFsOjk2MjUwNTM3Njk0MDAxMDoxNTk0Njkx", "tid":1594691}, "parent":{"exec_id":"aXAtMTcyLTMxLTQ2LTEwOS5hcC1zb3V0aGVhc3QtMi5jb21wdXRlLmludGVybmFsOjk2MjUwNTM3Njk0MDAxMDoxNTk0Njkx", "pid":1594691, "uid":0, "cwd":"/", "binary":"/bin/sh", "arguments":"-c \"wget server\"", "flags":"execve rootcwd clone", "start_time":"2024-07-15T10:51:37.837185880Z", "auid":4294967295, "pod":{"namespace":"default", "name":"client", "labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default", "k8s:io.cilium.k8s.policy.cluster=default", "k8s:io.cilium.k8s.policy.serviceaccount=default", "k8s:io.kubernetes.pod.namespace=default", "k8s:name=client", "k8s:post=eks-wireguard-cilium"], "container":{"id":"containerd://49778fac295c480cd2280d3e0ce8f7372cee760e043efbdae6a9c4ebfbdd3605", "name":"client", "image":{"id":"docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7", "name":"docker.io/library/busybox:latest"}, "start_time":"2024-07-15T09:32:59Z", "pid":2358}, "pod_labels":{"name":"client", "post":"eks-wireguard-cilium"}}, "docker":"49778fac295c480cd2280d3e0ce8f73", "parent_exec_id":"aXAtMTcyLTMxLTQ2LTEwOS5hcC1zb3V0aGVhc3QtMi5jb21wdXRlLmludGVybmFsOjk1Nzc4NjcyMDM3ODY0NDoxNTc4ODcy", "tid":1594691}, "status":1}, "node_name":"ip-172-31-46-109.ap-southeast-2.compute.internal", "time":"2024-07-15T10:51:37.843411095Z"}

Conclusion

Hopefully, this post gave you an overview of how and why you would deploy Isovalent Enterprise for Cilium in the AWS Marketplace and enable end-users with Enterprise features. If you have any feedback on the solution, please share it with us.

You can schedule a demo with our experts if you’d like to learn more.

Try it Out

Amit Gupta
AuthorAmit GuptaSenior Technical Marketing Engineer

Related

Blogs

Isovalent Enterprise for Cilium on EKS & EKS-A in AWS Marketplace

Isovalent Enterprise for Cilium is now available in the AWS marketplace.

By
Amit Gupta
Blogs

Cilium in EKS-Anywhere

This tutorial will do a deep dive into how to bring up an EKS-A cluster then upgrading the embedded Cilium with either Cilium OSS or Cilium Enterprise to unlock more features

By
Amit Gupta

Industry insights you won’t delete. Delivered to your inbox weekly.