“At any given moment, you have the power to say, this is not how the story will end.”—Christine Mason Miller. Let’s resume where we left off while deploying Isovalent Enterprise for Cilium from the AWS marketplace. This tutorial teaches you how to enable Enterprise features in an Elastic Kubernetes Service (EKS) cluster running Isovalent Enterprise for Cilium from the AWS marketplace on an EKS cluster.
What is Isovalent Enterprise for Cilium?
Isovalent Cilium Enterprise is an enterprise-grade, hardened distribution of open-source projects Cilium, Hubble, and Tetragon, built and supported by the Cilium creators. Cilium enhances networking and security at the network layer, while Hubble ensures thorough network observability and tracing. Tetragon ties it all together with runtime enforcement and security observability, offering a well-rounded solution for connectivity, compliance, multi-cloud, and security concerns.
Why Isovalent Enterprise for Cilium?
For enterprise customers requiring support and usage of Advanced Networking, Security, and Observability features, “Isovalent Enterprise for Cilium” is recommended with the following benefits:
Advanced network policy: Isovalent Cilium Enterprise provides advanced network policy capabilities, including DNS-aware policy, L7 policy, and deny policy, enabling fine-grained control over network traffic for micro-segmentation and improved security.
Hubble flow observability + User Interface: Isovalent Cilium Enterprise Hubble observability feature provides real-time network traffic flow, policy visualization, and a powerful User Interface for easy troubleshooting and network management.
Multi-cluster connectivity via Cluster Mesh: Isovalent Cilium Enterprise provides seamless networking and security across multiple clouds, including public cloud providers like AWS, Azure, and Google Cloud Platform, as well as on-premises environments.
Advanced Security Capabilities via Tetragon: Tetragon provides advanced security capabilities such as protocol enforcement, IP and port whitelisting, and automatic application-aware policy generation to protect against the most sophisticated threats. Built on eBPF, Tetragon can easily scale to meet the needs of the most demanding cloud-native environments.
Service Mesh: Isovalent Cilium Enterprise provides sidecar-free, seamless service-to-service communication and advanced load balancing, making deploying and managing complex microservices architectures easy.
Enterprise-grade support: Isovalent Cilium Enterprise includes enterprise-grade support from Isovalent’s experienced team of experts, ensuring that any issues are resolved promptly and efficiently. Additionally, professional services help organizations deploy and manage Cilium in production environments.
Why AWS marketplace?
AWS Marketplace is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. In AWS Marketplace, you can find, try, buy, and deploy the software and services needed to build new solutions and manage your cloud infrastructure. The catalog includes solutions for different industries and technical areas, free trials, and consulting services from AWS partners. Included among these solutions are Kubernetes application-based container offers. These offers contain applications that are meant to run on Kubernetes clusters such as Elastic Kubernetes Service (EKS).
Pre-Requisites
The following prerequisites need to be taken into account before you proceed with this tutorial:
Access to AWS marketplace. Create a new account for free.
The Cilium operator requires the following EC2 privileges to perform ENI creation and IP allocation.
An EKS cluster is running Isovalent Enterprise for Cilium from the AWS marketpace.
Cilium CLI: Cilium Enterprise provides a Cilium CLI tool that automatically collects all the logs and debug information needed to troubleshoot your Cilium Enterprise installation. You can install Cilium CLI for Linux, macOS, or other distributions on their local machine(s) or server(s).
Hubble CLI: To access the observability data collected by Hubble, you can install the Hubble CLI. You can install Hubble CLI for Linux, macOS, or other distributions on their local machine (s) or server (s).
Layer 3/ Layer 4 Policy
When using Cilium, endpoint IP addresses are irrelevant when defining security policies. Instead, you can use the labels assigned to the Pods to define security policies. The policies will be applied to the right Pods based on the labels, irrespective of where or when they run within the cluster.
The layer 3 policy establishes the base connectivity rules regarding which endpoints can talk to each other.
The layer 4 policy can be specified independently or in addition to the layer 3 policies. It restricts an endpoint’s ability to emit and/or receive packets on a particular port using a protocol.
You can take a Star Wars-inspired example in which there are three microservices applications: deathstar, tiefighter, and xwing. The deathstar runs an HTTP web service on port 80, which is exposed as a Kubernetes Service to load-balance requests to deathstar across two pod replicas. The deathstar service provides landing services to the empire’s spaceships so that they can request a landing port. The tiefighter pod represents a landing-request client service on a typical empire ship, and xwing represents a similar service on an alliance ship. They exist so that you can test different security policies for access control to deathstar landing services.
Validate L3/ L4 Policies
Deploy three services: deathstar, xwing, and firefighter
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15/examples/minikube/http-sw-app.yaml
service/deathstar created
deployment.apps/deathstar created
pod/tiefighter created
pod/xwing created
Kubernetes will deploy the Pods and service in the background.
Running kubectl get Pods,svc will inform you about the progress of the operation.
kubectl get pods,svc
NAME READY STATUS RESTARTS AGE
pod/client 1/1 Running 0 23h
pod/deathstar-54bb8475cc-4gcv4 1/1 Running 0 3m9s
pod/deathstar-54bb8475cc-lq6sv 1/1 Running 0 3m9s
pod/server 1/1 Running 0 23h
pod/tiefighter 1/1 Running 0 3m9s
pod/xwing 1/1 Running 0 3m9s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/deathstar ClusterIP 10.0.114.36 <none>80/TCP 3m9s
service/kubernetes ClusterIP 10.0.0.1 <none>443/TCP 4d1h
Check basic access
From the perspective of the deathstar service, only the ships with the label org=empire are allowed to connect and request landing. Since you have no rules enforced, both xwing and tiefighter can request landing.
You can start with the basic policy restricting deathstar landing requests to only the ships that have a label org=empire. This will not allow any ships that don’t have the org=empire label to even connect with the deathstar service. This simple policy filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), often called an L3/L4 network security policy.
The above policy whitelists traffic sent from any Pods with the label org=empire to deathstar Pods with the label org=empire, class=deathstar on TCP port 80.
You can now apply this L3/L4 policy:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
If you run the landing requests, only the tiefighter Pods with the label org=empire will succeed. The xwing Pods will be blocked.
This request will hang, so press Control-C to kill the curl request or wait for it to time out.
HTTP-aware L7 Policy
Layer 7 policy rules are embedded into Layer 4 Example rules and can be specified for ingress and egress. A layer 7 request is permitted if at least one of the rules matches. If no rules are specified, then all traffic is permitted. If a layer 4 rule is specified in the policy, and a similar layer 4 rule with layer 7 rules is also specified, then the layer 7 portions of the latter rule will have no effect.
To provide the strongest security (i.e., enforce least-privilege isolation) between microservices, each service that calls deathstar’s API should be limited to making only the set of HTTP requests it requires for legitimate operation.
For example, consider that the deathstar service exposes some maintenance APIs that random empire ships should not call.
Cilium can enforce HTTP-layer (i.e., L7) policies to limit the firefighter pod’s ability to reach URLs.
Validate L7 Policy
Apply L7 Policy—An example policy file extends our original policy by limiting tiefighter to making only a POST /v1/request-landing API call and disallowing all other calls (including PUT /v1/exhaust port).
Update the existing rule (from the L3/L4 section) to apply L7-aware policy to protect deathstar
As this rule builds on the identity-aware rule, traffic from Pods without the label org=empire will continue to be dropped, causing the connection to time out:
DNS-based policies are useful for controlling access to services outside the Kubernetes cluster. DNS acts as a persistent service identifier for external services, for example, by Google, and internal services, such as database clusters running in private subnets outside Kubernetes. CIDR or IP-based policies are cumbersome and hard to maintain as the IPs associated with external services can change frequently. The Cilium DNS-based policies provide an easy mechanism to specify access control, while Cilium manages the harder aspects of tracking DNS to IP mapping.
Validate DNS-Based Policies
In line with our Star Wars theme examples, you can use a simple scenario where the Empire’s mediabot Pods need access to GitHub to manage the Empire’s git repositories. The Pods shouldn’t have access to any other external service.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-sw-app.yaml
pod/mediabot created
Apply DNS Egress Policy. The following Cilium network policy allows mediabot Pods to only access api.github.com
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-matchname.yaml
ciliumnetworkpolicy.cilium.io/fqdn created
Testing the policy, you can see that mediabot has access to api.github.com but doesn’t have access to any other external service, e.g., support.github.com
kubectl exec mediabot -- curl -I -s https://api.github.com |head -1
HTTP/1.1 200 OK
kubectl exec mediabot -- curl -I -s https://support.github.com |head -1
curl: (28) Connection timed out after 5000 milliseconds
command terminated with exit code 28
This request will hang, so press Control-C to kill the curl request or wait for it to time out.
Combining DNS, Port, and L7 Rules
The DNS-based policies can be combined with port (L4) and API (L7) rules to restrict access further. In our example, you can restrict mediabot Pods to access GitHub services only on port 443. The toPorts section of the policy below achieves the port-based restrictions and DNS-based policies.
Validate the combination of DNS, Port, and L7-based Rules
Cilium supports the transparent encryption of Cilium-managed host traffic and traffic between Cilium-managed endpoints using WireGuard® or IPsec. In this tutorial, you will learn about Wireguard.
When WireGuard is enabled in Cilium, the agent running on each cluster node will establish a secure WireGuard tunnel between it and all other known Nodes.
Packets are not encrypted when destined to the same node from which they were sent. This behavior is intended. Encryption would provide no benefits, given that the raw traffic can be observed on the node anyway.
Validate Wireguard Encryption
To demonstrate Wireguard encryption, users can create a client pod spun up on one node and a server pod spun up on another in EKS.
The client does a “wget” towards the server every 2 seconds.
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client 1/1 Running 0 20s 172.31.46.22 ip-172-31-46-109.ap-southeast-2.compute.internal <none><none>server 1/1 Running 0 61s 172.31.27.246 ip-172-31-20-222.ap-southeast-2.compute.internal <none><none>
Run a bash shell in one of the Cilium Pods with kubectl -n kube-system exec -ti ds/cilium -- bash and execute the following commands:
Check that WireGuard has been enabled (the number of peers should correspond to the number of Nodes subtracted by one):
By default, WireGuard-based encryption only encrypts traffic between Cilium-managed Pods. You can also enable node-to-node encryption, which encrypts node-to-node, pod-to-node, and node-to-pod traffic.
Validate Wireguard Encryption
Run a bash shell in one of the Cilium Pods with kubectl -n kube-system exec -ti ds/cilium -- bash and execute the following commands:
Check that WireGuard has been enabled (NodeEncryption: Enabled)
Check that node-to-node traffic is sent via the cilium_wg0 tunnel device.
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-104-238.ap-south-1.compute.internal Ready <none> 131m v1.29.3-eks-ae9a62a 192.168.104.238 <none> Amazon Linux 25.10.219-208.866.amzn2.x86_64 containerd://1.7.11
ip-192-168-141-217.ap-south-1.compute.internal Ready <none> 131m v1.29.3-eks-ae9a62a 192.168.141.217 <none> Amazon Linux 25.10.219-208.866.amzn2.x86_64 containerd://1.7.11
Traffic is encrypted between the two Nodes through the WireGuard interface. You can observe that Cilium Health requires port TCP 4240 for all the Nodes for Cilium Health monitoring, and this traffic is also encrypted between the Nodes.
192.168.104.238.53264: Flags [.], ack 4081575545, win 509, options [nop,nop,TS val 3501711868 ecr 2677193498], length 013:38:40.999783 IP 192.168.104.238.53264 >192.168.141.217.4240: Flags [.], ack 1, win 502, options [nop,nop,TS val 2677208603 ecr 3501697020], length 013:38:41.074585 IP 192.168.141.217.4240 >192.168.104.238.38798: Flags [.], ack 3421044590, win 509, options [nop,nop,TS val 4095391564 ecr 136823413], length 013:38:41.074616 IP 192.168.104.238.38798 >192.168.141.217.4240: Flags [.], ack 1, win 502, options [nop,nop,TS val 136838515 ecr 4095376640], length 0
Ingress
Cilium uses the standard Kubernetes Ingress resource definition with an ingressClassName of cilium.
Cilium allows you to specify the load balancer mode for the Ingress resource:
dedicated: The Ingress controller will create a dedicated load balancer for the Ingress.
shared: The Ingress controller will use a shared load balancer for all Ingress resources.
Each Ingress controller mode has its benefits and drawbacks. The shared mode saves resources by sharing a single LoadBalancer Service across all Ingress resources in the cluster. In contrast, the dedicated mode can help to avoid potential conflicts (e.g., path prefix) between resources.
Note:Ingress helm values are available when you create an EKS cluster from the AWS marketplace, as listed in the prerequisites above.
Validate Ingress in EKS
Create a demo app in which the ingress configuration routes traffic to backend services from the bookinfo demo microservices app from the Istio project.
You’ll see a LoadBalancer service automatically created for this ingress when you get the list of services.
For EKS, you will observe an FQDN for the external IP rather than an actual IP address.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cilium-ingress-basic-ingress LoadBalancer 10.100.42.179 a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com 80:30493/TCP,443:31109/TCP 9s
details ClusterIP 10.100.160.211 <none>9080/TCP 8m7s
kubernetes ClusterIP 10.100.0.1 <none>443/TCP 3h3m
productpage ClusterIP 10.100.71.44 <none>9080/TCP 8m7s
ratings ClusterIP 10.100.106.195 <none>9080/TCP 8m7s
reviews ClusterIP 10.100.142.166 <none>9080/TCP 8m7s
The external IP address is populated into the Ingress
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
basic-ingress cilium * a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com 80 100s
You can also check the respective port mappings created in AWS for the Load Balancer.
Send HTTP Requests- Check (with curl or in your browser) that you can make HTTP requests to that external address. The / path takes you to the home page for the bookinfo application.
curl --fail -s http://a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com/details/1 -v
* Trying 3.108.107.12:80...
* Connected to a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com (3.108.107.12) port 80(#0)> GET /details/1 HTTP/1.1
> Host: a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com
> User-Agent: curl/7.81.0
> Accept: */*
>* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< server: envoy
< date: Tue, 16 Jul 2024 09:29:36 GMT
< content-length: 178< x-envoy-upstream-service-time: 3<* Connection #0 to host a7381f04d19b14629a443ad3aeacee80-1387183035.ap-south-1.elb.amazonaws.com left intact{"id":1,"author":"William Shakespeare","year":1595,"type":"paperback","pages":200,"publisher":"PublisherA","language":"English","ISBN-10":"1234567890","ISBN-13":"123-1234567890"}
As you keep adding Ingress resources, the Cilium Ingress controller will create dedicated LoadBalancer Services for each.
kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS
PORTS AGE
default basic-ingress cilium * ae433be6727054783896c9895876fb6f-1617631252.ap-south-1.elb.amazonaws.com 80 43s
default basic-ingress1 cilium * a4b53b55d28ed45ca82285dc23fae422-1118518200.ap-south-1.elb.amazonaws.com 80 6s
default basic-ingress2 cilium * af797a4519fc84884a2bf09a64a3a157-2068600342.ap-south-1.elb.amazonaws.com 80 29s
Optional: Deploy an ingress with LoadBalancer type as Shared. Notice how the Ingress controller will use a shared load balancer for all Ingress resources when you add ingress.
One of the additional benefits of using Cilium is its extremely efficient data plane. It’s particularly useful at scale, as the standard kube-proxy is based on a technology – iptables – that was never designed with the churn and the scale of large Kubernetes clusters.
Note-
Delete the kube-proxy Daemonset and configmap. For existing installations running kube-proxy as a DaemonSet, remove it using the commands below.
kubectl -n kube-system delete ds kube-proxy
kubectl -n kube-system delete cm kube-proxy
Remove kube-proxy iptable entries from each node
With root permissions, ensure that iptable entries pertinent to kube-proxy are removed.
This is done to clear out AWS-VPC-CNI-related rules.
The user needs to configure Cilium to point to the Kube API endpoint; otherwise, the cluster will be in an unhealthy state. Make sure the cluster is configured with these helm values when Cilium is installed.
Verify that the NodePort service has been created:
kubectl get svc my-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx NodePort 10.100.170.109 <none>80:31170/TCP 7s
With the help of the cilium service list command, validate that Cilium’s eBPF kube-proxy replacement created the new NodePort services under port 31170: (Truncated O/P)
Hubble’s CLI extends the Visibility that is provided by standard kubectl commands like kubectl get pods to give you more network-level details about a request, such as its status and the security identities associated with its source and destination.
The Hubble CLI can be leveraged to observe network flows from Cilium agents. Users can observe the flows from their local machine workstation for troubleshooting or monitoring. For this tutorial, users can see that all hubble outputs are related to the tests that are done above. Users can try other tests and see the same results with different values as expected.
Setup Hubble Relay Forwarding
Use the kubectl port forward to hubble-relay, then edit the hubble config to point at the remote hubble server component.
Hubble status can check the overall health of Hubble within your cluster. If using Hubble Relay, a counter for the number of connected Nodes will appear in the last line of the output.
hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 8,190/8,190 (100.00%)Flows/s: 31.18Connected Nodes: 2/2
View Last N Events
hubble observe displays the most recent events based on the number filter. Hubble Relay will display events over all the connected Nodes:
Suppose you have a CiliumNetworkPolicy that enforces DNS or HTTP policy. In that case, you can use the --type l7 filtering options for hubble to check our applications’ HTTP methods and DNS resolution attempts.
Hubble provides a field called VERDICT that displays one of FORWARDED, ERROR, or DROPPED for each flow. DROPPED could indicate an unsupported protocol within the underlying platform or Network Policy enforcing pod communication. Hubble can introspect the reason for ERROR or DROPPED flows and display the reason within the TYPE field of each flow.
To view filter events through the jq tool, you can swap the output to json mode. Visualize your metadata through jq, which will help you see more metadata around the workload labels like pod name/namespace assigned to both source and destination. This information is accessible by Cilium because it is encoded in the packets based on pod identities.
Note: To obtain the helm values to install Hubble UI and access the Enterprise documentation, contactsales@isovalent.com.
The graphical user interface (Hubble-UI) utilizes relay-based Visibility to provide a graphical service dependency and connectivity map. Hubble-UI is enabled via helm charts. The feature is not enabled when you create a new cluster using Isovalent Enterprise for Cilium or upgrade to Isovalent Enterprise for Cilium.
Once the installation is complete, you will notice hubble-ui running.
kubectl get pods -n hubble-ui
NAME READY STATUS RESTARTS AGE
hubble-ui-7b4c87fb48-2cpzz 2/2 Running 0 3d22h
Validate the installation
To access Hubble UI, forward a local port to the Hubble UI service:
Select the namespace by default, and you will observe a service map and network event list. In this case, the pod mediabot (created in the previous test case) is trying to access support.github.com over port 443
Note- You can read more on hubble in a detailed blog post that is a 3-part series.
DNS Proxy HA
Cilium Enterprise supports deploying an additional DNS Proxy, Daemonset called cilium-dnsproxy that can be life-cycled independently of Cilium Daemonset.
What is Cilium DNS Proxy HA?
When Cilium Network Policies that make use of toFQDNs are installed in a Kubernetes cluster, the Cilium agent starts an in-process DNS proxy that becomes responsible for proxying all DNS requests between all Pods and the Kubernetes internal kube-dns service. Whenever a Cilium agent is restarted, such as during an upgrade or due to something unexpected, DNS requests from all Pods on that node do not succeed until the Cilium agent is online again.
When cilium-dnsproxy is enabled, an independently life-cycled DaemonSet is deployed. cilium-dnsproxy acts as a hot standby that mirrors DNS policy rules. cilium-agent and cilium-dnsproxy bind to the same port, relying on the Linux kernel to distribute DNS traffic between the two DNS proxy instances. This allows you to lifecycle either cilium or cilium-dnsproxy Daemonset without impacting DNS traffic.
Installation of DNS-Proxy HA using helm
Note-
DNS Proxy High Availability relies on configuring the cilium-config ConfigMap with external-dns-proxy: true and to deploy the DNS proxy component.
To enable DNS Proxy HA and access the Enterprise documentation, contact sales@isovalent.com.
Once the installation is complete, you will notice cilium-dnsproxy running as a Daemonset, and also the Pods are up and running:
kubectl get ds -A
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cilium-system cilium 22222 kubernetes.io/os=linux 75m
cilium-system cilium-dnsproxy 22222<none> 27s
cilium-system cilium-license-manager 22222<none> 75m
cilium-system cilium-node-init 22222 kubernetes.io/os=linux 75m
cilium-system hubble-enterprise 22222<none> 6d4h
kube-system aws-node 00000 io.cilium/aws-node-enabled=true 11d
kubectl get pods -A -n cilium-system |grep"cilium-dnsproxy-*"cilium-system cilium-dnsproxy-kkptf 1/1 Running 0 43s
cilium-system cilium-dnsproxy-n4l9p 1/1 Running 0 43s
Validate DNS-Proxy HA
In line with our Star Wars theme examples, you can use a simple scenario where the Empire’s mediabot Pods need access to GitHub to manage the Empire’s git repositories. The Pods shouldn’t have access to any other external service.
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-sw-app.yaml
pod/mediabot created
Apply DNS Egress Policy. The following Cilium network policy allows mediabot Pods to only access api.github.com
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.15.6/examples/kubernetes-dns/dns-matchname.yaml
ciliumnetworkpolicy.cilium.io/fqdn created
Testing the policy, you can see that mediabot pod has access to api.github.com
Simulate a failure scenario where in the cilium-agent Pods are not up and running and the traffic still goes through the cilium-dnsproxy-* Pods
cilium-agent Pods are restarted as a part of the test
Traffic is not disrupted and continues to flow through the cilium-dnsproxy Pods
Tetragon
Tetragon provides powerful security observability and a real-time runtime enforcement platform. The creators of Cilium have built Tetragon and brought the full power of eBPF to the security world.
Tetragon helps platform and security teams solve the following:
Security Observability:
Observing application and system behavior such as process, syscall, file, and network activity
Tracing namespace, privilege, and capability escalations
File integrity monitoring
Runtime Enforcement:
Application of security policies to limit the privileges of applications and processes on a system (system calls, file access, network, kprobes)
Tetragon has been specifically built for Kubernetes and cloud-native infrastructure but can be run on any Linux system. Sometimes, you might want to enable Process Visibility in an environment without Cilium as the CNI. The Security Observability and Runtime Enforcement provided by the Hubble-Enterprise Daemonset can operate in a standalone mode, de-coupled from Cilium as a CNI.
Installation of Tetragon using helm
Note–To obtain the helm values to install Tetragon Enterprise and access Enterprise documentation, contactsales@isovalent.com.
Once the installation is complete, you will notice Tetragon running as a Daemonset, and also, the Pods are up and running:
kubectl get ds -A
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cilium-system cilium 22222 kubernetes.io/os=linux 75m
cilium-system cilium-dnsproxy 22222<none> 27s
cilium-system cilium-license-manager 22222<none> 75m
cilium-system cilium-node-init 22222 kubernetes.io/os=linux 75m
cilium-system hubble-enterprise 22222<none> 6d4h
kube-system aws-node 00000 io.cilium/aws-node-enabled=true 11d
kubectl get pods -A -n kube-system |grep"hubble-enterprise"cilium-system hubble-enterprise-fpk2q 2/2 Running 0 6d4h
cilium-system hubble-enterprise-r2wcf 2/2 Running 0 6d4h
Enable Hubble UI– The second way is to visualize the processes running on a certain workload by observing their Process Ancestry Tree. This tree gives you rich Kubernetes API, identity-aware metadata, and OS-level Process Visibility about the executed binary, its parents, and the execution time until the Container Runtime has started the container.
While in a real-world deployment, the Hubble Event Data would likely be exported to an SIEM or other logging datastore, in this quickstart, you will access this Event Data by redirecting the logs of the export-stdout container of the hubble-enterprise pod:
From the main Hubble UI screen, click on the tenant-jobs namespace in the list. Then, in the left navigation sidebar, click Processes.
To upload the exported logs, click Upload on the left of the screen:
Note: This manual Tetragon event export is not required when used with Hubble Timescape.
Use the file selector dialog to choose the events.log generated earlier and select the tenants-job namespace from the namespace dropdown.
Here, you can get a brief overview of a security use case that can easily be detected and be interesting to visualize. By using Hubble UI and visualizing the Process Ancestry Tree, you can detect a shell execution in the crawler-YYYYYYYYY-ZZZZZ Pod that occurred more than 5 minutes after the container has started. After clicking on the crawler-YYYYYYYY-ZZZZZ pod name from the Pods selector dropdown on the left of the screen, you will be able to see the Process Ancestry Tree for that pod:
The Process Ancestry Tree gives us:
Rich Kubernetes Identity-Aware Metadata: In the first row, you can see the name of the team or namespace and the specific application service to be inspected.
OS-Level Process Visibility: You can see all the processes that have been executed on the inspected service or were related to its Pod lifecycle
DNS Aware Metadata: You can see all the external connections with the exact DNS name as an endpoint made from specific processes of the inspected service.
Enabling DNS Visibility
Outbound network traffic remains a major attack vector for many enterprises. For example, in the above example, you can see that the crawler service reaches out to one or more services outside the Kubernetes cluster on port 443. However, the identity of these external services is unknown, as the flows only show an IP address.
Cilium Enterprise can parse the DNS-layer requests emitted by services and associate that identity data with outgoing connections, enriching network connectivity logs.
To inspect the DNS lookups for Pods within a namespace, you must apply a network policy that tells Cilium to inspect port 53 traffic from Pods to Kube-DNS at Layer 7. With this DNS Visibility, Hubble flow data will be annotated with DNS service identity for destinations outside the Kubernetes cluster. The demo app keeps reaching out to Twitter at regular intervals, and now both the service map and flows table for the tenant-jobs namespace shows connections to api.twitter.com:
Enabling HTTP Visibility
Like enabling DNS Visibility, you can also apply a Visibility policy to instruct Cilium to inspect certain traffic flows at the application layer (e.g., HTTP) without requiring any changes to the application itself.
In this example, you’ll inspect ingress connections to services within the tenant-jobs namespace at the HTTP layer. You can inspect flow details to get application-layer information. For example, you can inspect HTTP queries that coreapi make it to elasticsearch:
Network Policy
In this scenario, you will create services, deployments, and Pods (created in the above step (Observing Network Flows with Hubble CLI) and then use the Hubble UI’s network policy editor and service map/flow details pages to create, troubleshoot, and update a network policy.
Apply an L3/L4 Policy (Optional as this is covered in the L3/ L4 Policy section)
You can start with a basic policy restricting deathstar landing requests to only the ships that have a label (org=empire). This will not allow any ships that don’t have the org=empire label to even connect with the deathstar service. This simple policy filters only on IP protocol (network layer 3) and TCP protocol (network layer 4), often called an L3/L4 network security policy.
The above policy whitelists traffic sent from Pods with the label (org=empire) to deathstar Pods with the label (org=empire, class=deathstar) on TCP port 80.
To apply this L3/L4 policy, run:
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.13/examples/minikube/sw_l3_l4_policy.yaml
ciliumnetworkpolicy.cilium.io/rule1 created
If you run the landing requests again, only the tiefighter Pods with the label org=empire will succeed. The xwing Pods will be blocked.
This request will hang, so press Control-C to kill the curl request or wait for it to time out. As you can see, Hubble-UI reports that the flow will be dropped.
Using the policy editor, click on the denied/dropped flows and add them to your policy.
You need to download the policy and apply/update it again.
Once the updated policy is applied, run the same request from a xwing pod, and that should pass
Enabling TLS Visibility
Tetragon’s Network Security Observability provides deep insights into network events, such as tls events containing information about the exact TLS connections, including the negotiated cipher suite, the TLS version, the source/destination IP addresses, and the connection ports made by the initial process. In addition, tls events are also enriched by Kubernetes Identity-Aware metadata (Kubernetes namespace, pod/container name, labels, container image, etc).
By default, Tetragon does not show connectivity-related events. TLS Visibility requires a specific tracing policy and a CRD, which we will apply.
Send traffic to see the TLS ciphers and versions being used. From the xwing pod shell, try a simple curl to google.com and use openssl to check ciphers:
Tetragon Enterprise monitors files using information about the kernel’s internal structures. To configure File Enforcement, apply an example File Enforcement Policy, which blocks any write access on /etc/passwd and /etc/shadow and blocks deleting these files. This policy applies to host files and all Kubernetes Pods in the default Kubernetes namespace.
Note- contact sales@isovalent.com to get access to the specific file enforcement policies.
Using one of the already existing Pods, you’ll see events from a test pod on the default Kubernetes namespace when trying to edit the /etc/passwd file:
Tetragon Enterprise uses information about the internal structures of the kernel to provide process Visibility. Run kubectl to validate that Tetragon Enterprise is configured with process Visibility enabled:
Hopefully, this post gave you an overview of how and why you would deploy Isovalent Enterprise for Cilium in the AWS Marketplace and enable end-users with Enterprise features. If you have any feedback on the solution, please share it with us.
You can schedule a demo with our experts if you’d like to learn more.
Amit Gupta is a senior technical marketing engineer at Isovalent, powering eBPF cloud-native networking and security. Amit has 22+ years of experience in Networking, Telecommunications, Cloud, Security, and Open-Source. He has previously worked with Motorola, Juniper, Avi Networks (acquired by VMware), and Prosimo.io (acquired by Palo Alto Networks). He is keen to learn and try out new technologies that aid in solving day-to-day problems for operators and customers.
He has worked in the Indian start-up ecosystem for a long time and helps new folks in that area outside of work. Amit is an avid runner and cyclist, and also spends considerable time helping kids in orphanages.
This blog shows how to unlock the feature set for your EKS-A clusters by upgrading the embedded Cilium. We will also add the KuberVirt Operator and deploy VM's on the cluster.
By
Amit Gupta
Industry insights you won’t delete. Delivered to your inbox weekly.
