A packet trace has always been the common language between networking, platform, application, and security teams, the one artifact everyone can open when logs and metrics stop explaining a problem. Isopcap brings that artifact into Kubernetes, and this release makes it easy to get into the hands of the people who need it. Isopcap brings packet capture into Kubernetes, letting teams request captures through a custom resource. It records traffic from the selected workloads and writes PCAPng files for analysis with tools such as Wireshark or tcpdump. Isopcap has been evolving quickly, with new capabilities making packet capture easier to share and use across teams. Here are a few highlights: A shared home for every capture. Flip one flag and completed captures mirror straight to any S3-compatible bucket, so networking, application, and security teams pull traces from one durable, well-known place instead of hunting across nodes. The local storage is reclaimed once the file is safely uploaded, so continuous capture no longer quietly fills the node disk. Confidence that a capture is complete before anyone reads it. Each capture is sealed with an atomic manifest that only appears once every packet file is in place, so a watcher, automation job, or downstream analysis tool never picks up a half-written trace. Output your platform can build on. Ability to build your own tooling and automation can watch the bucket and act on each capture's metadata, files, and packet counts without taking on Isopcap as a dependency. A documented OpenShift installation path. Installing on OpenShift is now a documented, deterministic workflow, which makes Isopcap straightforward to roll out on enterprise clusters. Isopcap captures traffic where workloads run, but the engineers analyzing those captures may not have access to the nodes storing them. Uploading completed captures to a shared S3-compatible bucket lets teams retrieve traces without node access. This blog will explore writing captures into object storage. Starting with a capture, watch the PCAPng file ship to an S3-compatible bucket, reclaim the disk space on the node, and then pull the trace back down from a workstation with s3cmd. An optional upload sidecar handles that handoff, uploading each completed PCAPng file from the node to the bucket. Why a central bucket for packet captures Once a capture finishes, the PCAPng file is stored on the node where the workload ran. The workflow then looks like this: A PacketCapture runs where the workload runs. The completed PCAPng file is written to node-local storage. The file is copied to an S3-compatible bucket. A workstation or a separate process retrieves the trace with standard tools. The bucket removes node access from the analysis step. Platform teams continue to request captures through Kubernetes, while the teams investigating the incident retrieve the completed trace with standard tools. The walkthrough below shows this with the shop workload. Where Isopcap puts its files Before any object storage, it helps to understand where the agent writes captures. In this example, --output-base-path was used and points to a node-local hostPath, so captures remain on the node until they are uploaded. Isopcap can also write to a shared ReadWriteMany persistent volume, or to the agent pod’s filesystem when no persistent volume is configured. You can see which path the agent is using: As of v0.7.0 the agent lays files out per node, <base>/<namespace>/<capture>/<node>/<file>: The <node> segment is important because every agent mounts the same output volume. Without it, filenames and manifests from different nodes would collide. Two things follow from this layout: The output is whole binary artifacts, not a log stream. A .pcap is a finished PCAPng file, closed when the capture ends. The manifest is the completion signal. The agent writes manifest.json last only once the capture has finished on that node. A running capture has no manifest yet, so nothing downstream ever ships a half-written trace. Meet rclone The upload sidecar for Isopcap is built on rclone, an open-source command-line tool for moving files between local disk and cloud storage. It speaks S3 along with dozens of other backends, and it is a natural fit for pcap files. A .pcap is a finished binary artifact, so the job is to move the exact bytes of a completed file to an object key, confirm it landed, and then reclaim the space on disk. That is precisely what rclone does well. Inside the sidecar, rclone copies the pcap objects to the bucket, runs rclone check --one-way to confirm every object arrived intact, and only then deletes the local directory. Nothing is removed from the node until rclone has verified the copy in object storage, so a capture is never lost in transit. The sidecar keys off the manifest, not a timer. It copies the pcap files first and publishes the manifest to the bucket in a separate final pass, so the manifest never appears in object storage before the packets it declares complete. The environment This example will test with Isopcap v0.7.0 and its captures written to a hostPath on each node. Object storage is Garage, a single-pod S3-compatible store used for this example, with a bucket named isopcap-pcaps, and the S3 client for verification is s3cmd. To make it realistic, a capture against a shop workload to help debug an issue will be ran, the kind of "the app team is seeing intermittent HTTP drops" report where the packets tell the real story. Here is the flow of the example: Turning on the upload sidecar Enabling the sidecar can be done using the new upload feature, which can be enabled via helm. First, the credentials. The sidecar reads its S3 keys from a Secret in the release namespace, with the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys: Using helm, enable the upload feature and set the upload values pointing at the bucket: A few of the defaults are worth calling out, because they are what make this a shared troubleshooting store that teams can consume without node access, no extra flags needed: upload.prefix=pcap places objects at s3://isopcap-pcaps/pcap/<ns>/<capture>/<node>/…, so the bucket layout mirrors the on-disk layout. upload.deleteAfterUpload=true removes local files after rclone check verifies them, which is the disk-reclaim behavior. upload.forcePathStyle=true is required for most S3-compatible stores, Garage included, since they do not do per-bucket DNS. upload.intervalSeconds=15 is how often the sidecar scans for completed manifests. See the official rclone documentation for more details. For a private CA in front of the endpoint, upload.tls.caConfigMap points at a ConfigMap holding the certificate. Garage here is plain HTTP inside the cluster, so no CA is needed. Once the upgrade rolls out, each agent pod runs two containers. The second one is the sidecar: Running a capture and watching it ship The capture is an ordinary PacketCapture. This one targets the shop workload, filters to TCP port 80, and stops after 30 seconds or 10 MiB, whichever comes first: Apply it and watch the per-node status. The shop pods run on kind-worker2, so that is the node that captures packets. The other nodes report "No matching pods on this node" and complete immediately: The capture on kind-worker2 recorded 1734 packets with no drops. The moment it finishes, the files exist on that node's disk, including the freshly written manifest: Within one scan interval, the sidecar copies those objects to the bucket, verifies them, and removes the local copy. Check the same node again and it is empty, the node disk reclaimed: No output means every file was uploaded, verified, then unlinked. The durable copy now lives in the bucket. The manifest as the completion signal Before verifying the pcaps, look at the object the sidecar publishes last. Pull the manifest for this node's capture: This makes the bucket a reliable collaboration store rather than a folder full of files that might or might not be complete. The manifest carries the capture identity, the node it ran on, the filter, start and end times, and one entry per pcap file with pod, interface, size, and packet counts. The three files add up to the 1734 packets the status reported, which is a nice cross-check that nothing was lost in transit. Because the manifest is published only after every pcap it lists is already in the bucket, a consumer can watch for manifest.json, and its appearance guarantees the whole capture is present and readable. Pulling the trace from a workstation Now the payoff, from the perspective of whichever teammate needs the packets. They never touch a node. They point s3cmd at the bucket and list what is there: Pull the frontend trace and open it with any standard tool: The trace shows a TCP reset (Flags [R.]) alongside the healthy 200 OK, which is the evidence that explains the intermittent drops. Whoever needs a trace can pull it straight from the bucket, with no node access, because the capture was already waiting in shared object storage. Summary Isopcap v0.7.0 turns packet capture into something a whole team can share. The agent writes each capture into its own per-node directory and seals it with an atomic manifest.json, and the opt-in rclone sidecar mirrors completed captures to any S3-compatible bucket, verifies them, and reclaims the node disk behind them. For collaborative operations, that bucket becomes the single durable home for every capture. Platform engineering runs captures as Kubernetes state, the manifest guarantees each one is complete before anyone reads it, and NetOps, application, and security engineers pull standard PCAPng files from one well-known location with the tools they already trust. Packet capture goes back to being the shared troubleshooting language it always was, without anyone having to chase down a node.