Isovalent + Splunk for Integrated Vulnerability Mitigation

New: Join the Isovalent + Splunk webinar

Step into the next era of Kubernetes security with Isovalent and Splunk! Register and learn how to bring Isovalent's Runtime Security into Splunk workflows.

Get the webinar link

As security requirements multiply, data continues painting the picture of where SecOps and DevOps teams need support in runtime observability and proactive threat response.

In The State of Cloud Security Platforms and DevSecOps report for 2024, 74% of respondents reported they “cannot remediate security issues fast enough to prevent incidents” and 94% are searching for “better context for efficient remediation and faster response”. We see this consistently, SecOps and DevOps needing support for highly-scaled observability and enforcement, with workflows made easy.

Isovalent and Splunk have joined to tackle these problems with distributed monitoring and policy enforcement. Bringing the best of Isovalent’s eBPF-based security data, with the insights and workflows from Splunk’s leading security analytics platform, into a ready-to-win rollout for SecOps and DevOps teams.

Together, Isovalent + Splunk brings teams deeply integrated compensating runtime controls, threat detection of known behaviour, distributed exploit protection, and application fingerprinting for policy generation. If there is a specific use case you want to learn more about, feel free to navigate directly below or reach out for a personal demo.

Isovalent offers a full stack for container and multicloud networking, which includes CNI, an ingress controller, and service mesh, as well as Hubble and Tetragon for runtime security and observability.
— Andrew Green – Analyst GigaOm

Isovalent is the standard in Container Networking

Isovalent named as leader in GigaOm Sonar Report

To the Kernel of Truth

Despite often feeling on an island, we see similar problems emerge across organizations. It’s extremely important for Security teams to observe all applications and processes at runtime, and apply policies that immediately trigger if violating events are invoked.

In the last decade it’s become increasingly clear, the solution to the observability and policy enforcement problem should come from the kernel. eBPF has emerged as the answer to this problem space, allowing for flexible and dynamic control directly in-kernel.

Highlighted in the 2024 State of eBPF report by the Linux Foundation and eBPF Foundation, eBPF has seen longstanding adoption among industry leaders. For instance, “since 2017, Meta has processed every packet going into its data centers with eBPF, as does Google for most of its data center traffic.” eBPF is the now standard technology being used by Meta, Android, Microsoft, Google, CloudFlare, Netflix, and other leaders for core production networking, security, observability and tracing.

Importantly, as the creators of eBPF, Isovalent’s runtime security platform optimizes the kernel level integration to keep overhead extremely low, doesn’t require changes to the application (allowing for policies to be applied on the fly without downtime), and gathers Kubernetes identity metadata for clear context about process ancestries and system calls.

eBPF’s unique vantage point inside the operating system allows Isovalent to efficiently extract this workload identity alongside network transmission data (L3, L4, and L7) from a Kubernetes environment, all with zero changes to the applications code. The kernel also provides the ideal location for enforcement use cases, to block or override malicious processes before they execute.

How to guide for host-based Kubernetes visibility

Learn how Isovalent's lightweight eBPF sensor captures K8s telemetry down to the binary, tying process to network data with no application changes.

Monitoring and Alerting Bad Behavior

Security teams grapple with analyzing vast amounts of data to pinpoint potential threats. Isovalent has built out a comprehensive set of security observability queries for Splunk, creating an out-of-the-box Threat Detection dashboard of the most common known suspicious behaviors to alert on.

Monitoring and alerting on kernel events give Security Teams critical visibility into system activity.

Did a specified process perform a read/write to a sensitive file?
Did the process send or receive data on a socket?
Did it mount a file system? Escalate privileges to root?
Attempt to perform a container escape?
Load a new driver?
Modify system time or firewall rules?
Attempt to exploit a known CVE?
and so much more visibility fundamental to good security.

The Splunk dashboard for Threat Detection highlights known suspicious behaviors. This means security teams can quickly focus on genuine risks, reducing the chance of security incidents. SecOps teams pull and apply these prebuilt queries to monitor for the most common known threats around sensitive system calls, late process executions, malicious kubectl execs, new DNS names, namespace changes, new external destinations, and more.

Notably, oversight is built into the alerting workflow, allowing admins to easily mark ‘known-bad’ behaviors as acceptable for specific workloads, minimizing false positives and building out layers of efficiency. Imagine you have a workload that knowingly relies on a sensitive system call, the SecOps team can tag the violating workload as an approved exception to remove the policy from alerting on that target workload.

Let’s explore a known threat behavior related to suspicious process executions. Despite predictable patterns in container initialization, detecting suspicious process executions in Kubernetes runtime is challenging. Processes running post-initialization could be benign (like cronjobs) or indicate a compromise, especially if the binary isn’t part of the container init.

Here, Isovalent provides a pre-built Splunk query to alert on shell executions occurring more than 5 minutes after container initialization by comparing execution timings with expected initialization periods.

In the use case above, the timing of a process allows us to detect anomalous behaviour and alert on shell executions outside the expected container initialization period. Each line in the above dashboard is just the tip of the iceberg for runtime context that Isovalent provides. Additionally, SecOps teams familiar with Splunk’s Search Processing Language (SPL) can apply discrete tagging on the Isovalent data, for precise and efficient filtering of security-significant events.

With Kubernetes aware metadata, the rich eBPF telemetry allows SecOps teams to click deeper into an event. Teams are able to answer the earlier questions we covered about each process; such as, what were the parent processes to each shell execution? What binaries are running on the target workload? What versions of these binaries? Any sensitive files accessed? Was there any weak TLS version or weak cipher used?

Isovalent and Splunk have also included risk scoring flags to reduce noise and keep only the important alerts surfacing up. This goes hand-in-hand with the manual feedback process from a human operator, whereby a SecOps team can manually mark specific applications or services to be ignored for a set of policies – as the violating process is actually part of expected behavior.

For example around risk scoring, a kubectl exec in isolation may be part of baseline behaviour. However, if we couple this with elevated privileges, access to sensitive files, or the process leading to a public internet network connection, then we can correlate that context to raise the risk score and generate an alert.

Generate audit trails for mission-critical applications

Investigate an example of a process tree built using Isovalent, mapping the processes leading to a data exfiltration attack.

Explore the process tree

Monitoring Access to Sensitive Files in Splunk

This example shows an integration with Splunk to track access to sensitive files while providing context of the access such as Kubernetes metadata, container image, binary, and user information.

See the TLS & FIM dashboards

Out-Of-The-Box Policy Enforcement for High Risk Events

Let’s address one of the most common issues teams face: remediating threats faster. Moving beyond monitoring and alerting, Isovalent and Splunk work together to bring robust out-of-the-box policy enforcement capabilities for high-risk events.

Security policies are proactive enforcement points, evaluating real time events to prevent unauthorized actions before they compromise your systems. Isovalent’s predefined policies contain a comprehensive list of “known-bad” actions and system calls that are associated with security threats, such as privilege escalation or the creation of new Linux namespaces—common techniques used in container escapes.

By leveraging predefined policies for known vulnerabilities and high-risk configurations, security teams can effectively block threats without manually configuring each scenario. This approach streamlines the policy enforcement process, allowing teams to focus on strategic security initiatives rather than routine tasks.

Let’s look at an example with Apache Tomcat CVE 2020-9484, allowing for remote code execution under multiple conditions.

In the Splunk dashboard below, you can see the event details of the Tomcat attack being successfully exploited on a Kubernetes workload without a mitigating enforcement policy in place. Here, there is no runtime policy enforcement point to block the attack, but Isovalent still captures the full attack flow.

We observe that the attacker executed a bash script on the target workload, tomcat, with Isovalent detailing the complete parent process flow and capturing the arguments of each executed binary. Later, we’ll use these parent processes and arguments to construct an ancestry tree of the attack.

Isovalent’s rich runtime data gives us the basis and flexibility to enforce granular policies right where they matter most in the process flow.

In the next scenario, we apply an enforcement policy. With the Isovalent runtime policy active, the vulnerable Java binaries inside the tomcat workload are not able to execute any other processes (e.g: the Java binary cannot “shell out”). The Splunk dashboard confirms the attack mitigation, providing detailed insights to reconstruct the entire process flow.

In the above dashboard, we see the policy block-cve-2020-9484 protected the tomcat container from being exploited by CVE-2020-9484. The enforcement policy blocks the vulnerable Java binaries from executing any other processes, thus preventing late process executions after container initialization. Notably, this targeted control keeps normal functions of the application unhindered, allowing the application to operate as expected while surgically addressing the exploit threat.

Looking a layer deeper, SecOps teams can also reconstruct the full context of the attack. Below is the process tree for the Apache Tomcat CVE exploit attempt, teams quickly see the:

Kubernetes metadata (tomcat container, tomcat-92c2c pod, ip-192-168-50-41.us-west-1.compute.internal namespace)
Container runtime initialization steps with systemd and containerd processes
Pod process tree (processes, arguments, timestamps, network information) with the attackers catalina.sh bash script, through to the data exfiltration attempt by reading the content of the /etc/passwd file.

The workflow of rich Isovalent data into Splunk analysis lends itself to building a feedback loop of policy enforcement and policy improvement. Isovalent collects identity-aware data, feeds that data into Splunk for analysis, and SecOps teams use that data to improve runtime policies.

Where SecOps Meets DevOps

Notably, security requirements have permeated across traditional divisions, now often finding themselves in the hands of DevOps teams. Here, we expand on a solution that support DevOps teams building faster and safer with Isovalent and Splunk.

An integration with Elastic Container Registry (ECR) applies policies across containerized environments, maintaining security without hindering DevOps workflows.

In order to keep developer workflows moving smoothly, Isovalent integrates with ECR to monitor container images as they are added to the registry, and helps SecOps teams apply compensating controls if they are deployed.

Isovalent’s out-of-the-box policies for known CVE’s allows DevOps teams to push out and maintain workloads that are protected by runtime policies. Meaning that the DevOps teams have a way to immediately apply compensating controls to production workloads, without the requirement of producing a perfect golden container image or Linux VM image.

Take another example workflow where a new registry image is vulnerable to CVE-2020-9484, exploiting Apache Tomcat. This image needs to be deployed in production today, and the DevOps team has to find a way for the exploit to be mitigated. Isovalent and Splunk can help:

The new image is added to ECR.
Container registry flags the image for CVE-2020-9484.
The image is deployed to production.
Isovalent immediately captures the identity metadata of the vulnerable image running on Pod A, sending this to Splunk.
The Splunk dashboard shows Pod A running a vulnerable image.
Splunk recommends that the Isovalent policy template for CVE-2020-9484 is updated to have Pod A as the target endpoint.
The updated policy is manually applied to target Pod A.
An event is generated in Splunk for an attempt to exploit CVE-2020-9484, showing the Isovalent runtime policy blocked the attack. Includes all the rich metadata needed for audit and forensics.

This process keeps developer workflows uninterrupted (no need to remove or delete the vulnerable image), while keeping production runtime secure with the appropriate compensating control.

Creating a Detailed Model of Application Behavior

Another core use case is leveraging deep observability to form an application baseline, and then understand when baseline behaviour has changed. The baseline can be established in two ways: explicit or implicit. In explicit modeling, a SecOps team manually reviews existing behavior and sets the baseline for expected events. In implicit modeling, the runtime sensor observes behavior for a defined period to learn ‘known-good’ behavior, and then understands what is anomalous to that baseline in the future.

In both cases, baseline policies are easily generated from fingerprints to lockdown processes outside of the ‘known-good’ set of behaviors. This model allows security teams to detect deviations from expected behavior.

An example flow follows:

Isovalent Runtime Security collects detailed runtime events around existing processes, network connections, file I/O activity, and syscalls.
Isovalent exports events to Splunk.
In Splunk, browse each workload by label to drill down into individual applications, visualizing the telemetry of process ancestries and system calls.
From this baseline behavior, least-privilege policies are recommended for a given workload to lockdown which binaries can execute.
The platform owner applies the baseline policy to the target workload.
Policy violations are generated by Isovalent, and surfaced into Splunk.

This foundation not only hardens production workloads, but also supports efficient policy generation, laying the groundwork for a feedback loop for better policies. With a known baseline in expected behaviour, SecOps teams are generating and manually applying policies that block potentially malicious processes outside of the ‘known-good’ workflow.

Top 7 Runtime Observability Use Cases

Platform and security teams have quickly adopted Isovalent for flexible eBPF-based security observability and runtime enforcement. Isovalent is the standard for eBPF-based security observability, and let’s look at what that means in practical terms for the use cases teams are successfully solving.

Explore the top runtime observability use cases

Want to learn more?

From distributed exploit protection and policy enforcement, to deep observability and application behavior modeling, Isovalent and Splunk deliver production-ready solutions that tackle the core issues plaguing teams around context and speed.

The right context keeps SecOps and DevOps teams agile and proactive, safeguarding production applications with minimal disruption. Through the use cases above, we see clearly why teams are choosing Isovalent for the right foundation in security visibility and control. And where deep integrations with Splunk allow teams to move entire organizations towards faster and safer workflows.

Ready to modernize your security strategy? Reach out for a personalized demo or explore our solution briefs to learn how Isovalent helps global leaders scale and succeed.