How to Upload Kubernetes Packet Captures to S3 With Isopcap

Christian Hernandez

Published: October 29, 2026 | Updated: October 30, 2026

A packet trace has always been the common language between networking, platform, application, and security teams, the one artifact everyone can open when logs and metrics stop explaining a problem. Isopcap brings that artifact into Kubernetes, and this release makes it easy to get into the hands of the people who need it.

Isopcap brings packet capture into Kubernetes, letting teams request captures through a custom resource. It records traffic from the selected workloads and writes PCAPng files for analysis with tools such as Wireshark or tcpdump.

Isopcap has been evolving quickly, with new capabilities making packet capture easier to share and use across teams. Here are a few highlights:

  • A shared home for every capture. Flip one flag and completed captures mirror straight to any S3-compatible bucket, so networking, application, and security teams pull traces from one durable, well-known place instead of hunting across nodes. The local storage is reclaimed once the file is safely uploaded, so continuous capture no longer quietly fills the node disk.
  • Confidence that a capture is complete before anyone reads it. Each capture is sealed with an atomic manifest that only appears once every packet file is in place, so a watcher, automation job, or downstream analysis tool never picks up a half-written trace.
  • Output your platform can build on. Ability to build your own tooling and automation can watch the bucket and act on each capture's metadata, files, and packet counts without taking on Isopcap as a dependency.
  • A documented OpenShift installation path. Installing on OpenShift is now a documented, deterministic workflow, which makes Isopcap straightforward to roll out on enterprise clusters.

Isopcap captures traffic where workloads run, but the engineers analyzing those captures may not have access to the nodes storing them. Uploading completed captures to a shared S3-compatible bucket lets teams retrieve traces without node access.

This blog will explore writing captures into object storage. Starting with a capture, watch the PCAPng file ship to an S3-compatible bucket, reclaim the disk space on the node, and then pull the trace back down from a workstation with s3cmd. An optional upload sidecar handles that handoff, uploading each completed PCAPng file from the node to the bucket.

Sending Packet Captures to Object Storage with Isopcap | Technical Demo Video

Why a central bucket for packet captures

Once a capture finishes, the PCAPng file is stored on the node where the workload ran. The workflow then looks like this:

  • A PacketCapture runs where the workload runs.
  • The completed PCAPng file is written to node-local storage.
  • The file is copied to an S3-compatible bucket.
  • A workstation or a separate process retrieves the trace with standard tools.

The bucket removes node access from the analysis step. Platform teams continue to request captures through Kubernetes, while the teams investigating the incident retrieve the completed trace with standard tools. The walkthrough below shows this with the shop workload.

Where Isopcap puts its files

Before any object storage, it helps to understand where the agent writes captures.

In this example, --output-base-path was used and points to a node-local hostPath, so captures remain on the node until they are uploaded. Isopcap can also write to a shared ReadWriteMany persistent volume, or to the agent pod’s filesystem when no persistent volume is configured.

container path   /var/lib/isopcap/captures      (agent writes here)
node path        /var/lib/isopcap-captures       (hostPath, set at install)

You can see which path the agent is using:

$ kubectl -n isopcap-system get ds isopcap-agent \
    -o jsonpath='{.spec.template.spec.volumes[?(@.name=="captures")].hostPath.path}{"\n"}'
/var/lib/isopcap-captures

As of v0.7.0 the agent lays files out per node, <base>/<namespace>/<capture>/<node>/<file>:

/var/lib/isopcap-captures/default/http-capture/kind-worker2/
  default_shop-backend_eth0_gen1_20260921-212354.pcap    # the capture
  default_shop-backend_eth0_gen1_20260921-212354.json    # per pod/iface sidecar
  manifest.json                                          # per-node manifest

The <node> segment is important because every agent mounts the same output volume. Without it, filenames and manifests from different nodes would collide. Two things follow from this layout:

  1. The output is whole binary artifacts, not a log stream. A .pcap is a finished PCAPng file, closed when the capture ends.
  2. The manifest is the completion signal. The agent writes manifest.json last only once the capture has finished on that node. A running capture has no manifest yet, so nothing downstream ever ships a half-written trace.

Kubernetes Packet Capture with Isopcap

Learn to capture and analyze live Kubernetes network traffic with Isovalent's Isopcap: install it, run targeted PacketCapture captures against labeled workloads, narrow them with BPF filters, and inspect the PCAPng files with tcpdump and termshark.

Meet rclone

The upload sidecar for Isopcap is built on rclone, an open-source command-line tool for moving files between local disk and cloud storage. It speaks S3 along with dozens of other backends, and it is a natural fit for pcap files. A .pcap is a finished binary artifact, so the job is to move the exact bytes of a completed file to an object key, confirm it landed, and then reclaim the space on disk. That is precisely what rclone does well.

Inside the sidecar, rclone copies the pcap objects to the bucket, runs rclone check --one-way to confirm every object arrived intact, and only then deletes the local directory. Nothing is removed from the node until rclone has verified the copy in object storage, so a capture is never lost in transit.

The sidecar keys off the manifest, not a timer. It copies the pcap files first and publishes the manifest to the bucket in a separate final pass, so the manifest never appears in object storage before the packets it declares complete.

The environment

This example will test with Isopcap v0.7.0 and its captures written to a hostPath on each node. Object storage is Garage, a single-pod S3-compatible store used for this example, with a bucket named isopcap-pcaps, and the S3 client for verification is s3cmd.

To make it realistic, a capture against a shop workload to help debug an issue will be ran, the kind of "the app team is seeing intermittent HTTP drops" report where the packets tell the real story.

Here is the flow of the example:

inline-image-0

Turning on the upload sidecar

Enabling the sidecar can be done using the new upload feature, which can be enabled via helm. First, the credentials. The sidecar reads its S3 keys from a Secret in the release namespace, with the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys:

$ kubectl -n isopcap-system create secret generic s3-credentials \
    --from-literal=AWS_ACCESS_KEY_ID="$KEY_ID" \
    --from-literal=AWS_SECRET_ACCESS_KEY="$SECRET"

Using helm, enable the upload feature and set the upload values pointing at the bucket:

helm upgrade --install isopcap oci://registry.isovalent.com/isopcap \
  --version 0.7.0 \
  --namespace isopcap-system \
  --set upload.enabled=true \
  --set upload.endpoint="http://garage.garage.svc.cluster.local:3900" \
  --set upload.bucket="isopcap-pcaps" \
  --set upload.region="garage" \
  --set upload.credentialsSecret="s3-credentials"

A few of the defaults are worth calling out, because they are what make this a shared troubleshooting store that teams can consume without node access, no extra flags needed:

  • upload.prefix=pcap places objects at s3://isopcap-pcaps/pcap/<ns>/<capture>/<node>/…, so the bucket layout mirrors the on-disk layout.
  • upload.deleteAfterUpload=true removes local files after rclone check verifies them, which is the disk-reclaim behavior.
  • upload.forcePathStyle=true is required for most S3-compatible stores, Garage included, since they do not do per-bucket DNS.
  • upload.intervalSeconds=15 is how often the sidecar scans for completed manifests.

See the official rclone documentation for more details.

For a private CA in front of the endpoint, upload.tls.caConfigMap points at a ConfigMap holding the certificate. Garage here is plain HTTP inside the cluster, so no CA is needed.

Once the upgrade rolls out, each agent pod runs two containers. The second one is the sidecar:

$ kubectl -n isopcap-system get pods \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'
isopcap-agent-7vk9r     agent upload
isopcap-agent-8zg8b     agent upload
isopcap-agent-rqh6w     agent upload

Running a capture and watching it ship

The capture is an ordinary PacketCapture. This one targets the shop workload, filters to TCP port 80, and stops after 30 seconds or 10 MiB, whichever comes first:

apiVersion: isopcap.isovalent.com/v1alpha1
kind: PacketCapture
metadata:
  name: http-capture
  namespace: default
spec:
  selector:
    matchLabels:
      app: shop
  filter: "tcp port 80"
  maxDuration: "30s"
  maxSize: "10Mi"

Apply it and watch the per-node status. The shop pods run on kind-worker2, so that is the node that captures packets. The other nodes report "No matching pods on this node" and complete immediately:

$ kubectl apply -f http-capture.yaml
packetcapture.isopcap.isovalent.com/http-capture created

$ kubectl get pcapstatus -l isopcap.isovalent.com/capture=http-capture -n default
NAME                              CAPTURE        NODE                 PHASE       REASON                          PACKETS   DROPS
http-capture-kind-control-plane   http-capture   kind-control-plane   Completed   No matching pods on this node   0         0
http-capture-kind-worker          http-capture   kind-worker          Completed   No matching pods on this node   0         0
http-capture-kind-worker2         http-capture   kind-worker2         Completed                                   1734      0

The capture on kind-worker2 recorded 1734 packets with no drops. The moment it finishes, the files exist on that node's disk, including the freshly written manifest:

$ docker exec kind-worker2 find /var/lib/isopcap-captures -type f
/var/lib/isopcap-captures/default/http-capture/kind-worker2/default_shop-deadend_eth0_gen1_20260921-212354.pcap
/var/lib/isopcap-captures/default/http-capture/kind-worker2/manifest.json
/var/lib/isopcap-captures/default/http-capture/kind-worker2/default_shop-backend_eth0_gen1_20260921-212354.json
/var/lib/isopcap-captures/default/http-capture/kind-worker2/default_shop-frontend_eth0_gen1_20260921-212354.pcap
/var/lib/isopcap-captures/default/http-capture/kind-worker2/default_shop-deadend_eth0_gen1_20260921-212354.json
/var/lib/isopcap-captures/default/http-capture/kind-worker2/default_shop-backend_eth0_gen1_20260921-212354.pcap
/var/lib/isopcap-captures/default/http-capture/kind-worker2/default_shop-frontend_eth0_gen1_20260921-212354.json

Within one scan interval, the sidecar copies those objects to the bucket, verifies them, and removes the local copy. Check the same node again and it is empty, the node disk reclaimed:

$ docker exec kind-worker2 find /var/lib/isopcap-captures -type f
$

No output means every file was uploaded, verified, then unlinked. The durable copy now lives in the bucket.

The manifest as the completion signal

Before verifying the pcaps, look at the object the sidecar publishes last. Pull the manifest for this node's capture:

$ s3cmd get s3://isopcap-pcaps/pcap/default/http-capture/kind-worker2/manifest.json -
{
  "capture": "http-capture",
  "namespace": "default",
  "node": "kind-worker2",
  "phase": "Completed",
  "filter": "tcp port 80",
  "startTime": "2026-09-21T21:23:54Z",
  "endTime": "2026-09-21T21:24:24Z",
  "files": [
    {
      "key": "default/http-capture/kind-worker2/default_shop-backend_eth0_gen1_20260921-212354.pcap",
      "podNamespace": "default",
      "podName": "shop-backend",
      "interface": "eth0",
      "sizeBytes": 93904,
      "packetsCaptured": 749
    },
    {
      "key": "default/http-capture/kind-worker2/default_shop-deadend_eth0_gen1_20260921-212354.pcap",
      "podNamespace": "default",
      "podName": "shop-deadend",
      "interface": "eth0",
      "sizeBytes": 11688,
      "packetsCaptured": 118
    },
    {
      "key": "default/http-capture/kind-worker2/default_shop-frontend_eth0_gen1_20260921-212354.pcap",
      "podNamespace": "default",
      "podName": "shop-frontend",
      "interface": "eth0",
      "sizeBytes": 105468,
      "packetsCaptured": 867
    }
  ]
}

This makes the bucket a reliable collaboration store rather than a folder full of files that might or might not be complete. The manifest carries the capture identity, the node it ran on, the filter, start and end times, and one entry per pcap file with pod, interface, size, and packet counts. The three files add up to the 1734 packets the status reported, which is a nice cross-check that nothing was lost in transit.

Because the manifest is published only after every pcap it lists is already in the bucket, a consumer can watch for manifest.json, and its appearance guarantees the whole capture is present and readable.

Pulling the trace from a workstation

Now the payoff, from the perspective of whichever teammate needs the packets. They never touch a node. They point s3cmd at the bucket and list what is there:

$ s3cmd ls s3://isopcap-pcaps/pcap/default/http-capture/kind-worker2/
2026-09-21 21:24     93904   .../default_shop-backend_eth0_gen1_20260921-212354.pcap
2026-09-21 21:24     11688   .../default_shop-deadend_eth0_gen1_20260921-212354.pcap
2026-09-21 21:24    105468   .../default_shop-frontend_eth0_gen1_20260921-212354.pcap
2026-09-21 21:24      1020   .../manifest.json

Pull the frontend trace and open it with any standard tool:

$ s3cmd get s3://isopcap-pcaps/pcap/default/http-capture/kind-worker2/default_shop-frontend_eth0_gen1_20260921-212354.pcap ./frontend.pcap

$ tcpdump -n -r ./frontend.pcap
14:23:54 IP 10.244.2.34.55248 > 10.244.2.77.http: Flags [P.], ... length 91: HTTP: GET / HTTP/1.1
14:23:54 IP 10.244.2.77.http > 10.244.2.34.55248: Flags [P.], ... length 207: HTTP: HTTP/1.1 200 OK
14:23:54 IP 10.244.2.200.http > 10.244.2.34.33985: Flags [R.], seq 0, ack 521900245, win 0, length 0

The trace shows a TCP reset (Flags [R.]) alongside the healthy 200 OK, which is the evidence that explains the intermittent drops. Whoever needs a trace can pull it straight from the bucket, with no node access, because the capture was already waiting in shared object storage.

Summary

Isopcap v0.7.0 turns packet capture into something a whole team can share. The agent writes each capture into its own per-node directory and seals it with an atomic manifest.json, and the opt-in rclone sidecar mirrors completed captures to any S3-compatible bucket, verifies them, and reclaims the node disk behind them.

For collaborative operations, that bucket becomes the single durable home for every capture. Platform engineering runs captures as Kubernetes state, the manifest guarantees each one is complete before anyone reads it, and NetOps, application, and security engineers pull standard PCAPng files from one well-known location with the tools they already trust. Packet capture goes back to being the shared troubleshooting language it always was, without anyone having to chase down a node.

Schedule a Demo With Our Engineers

Learn how our solution map to your challenges.

/authors/christian-hernandez

Christian Hernandez, Technical Marketing Engineer