BIG TCP Over IPv4 With Cilium Introduction What we need to do is group packets together to reduce the number of packets that the CPU has to process as traffic moves through the Linux networking stack. We already had technologies like GRO and TSO, which let us aggregate traffic into bigger packets, but they were limited to around 64 kilobytes because of the size field in the IP header. BIG TCP gives us a way to go beyond that 64 kilobyte limit. In the Cilium demo environment we’re going up to 192 kilobytes, which effectively triples the maximum packet size we can handle in the stack. Demo time In my demo environment I have a small Kind cluster running Cilium 1.14, and so far I have not enabled BIG TCP. One of the key requirements is a recent Linux kernel because this is a very new feature. In my environment I’m running kernel 6.4. What I’m going to do is run performance testing before enabling BIG TCP, then enable the feature, run the same tests again, and compare the results. For the network performance testing I’m using netperf, with a server and a client running on different nodes. Network performance testing without BIG TCP First we run the baseline tests without BIG TCP so we can see the packet rate and latency before changing anything. Then I enable BIG TCP in the Cilium configuration, restart the relevant pods, and rerun the same netperf tests. With BIG TCP enabled, you can see the improvement very clearly. Latency drops, the packet-processing efficiency improves significantly, and in this demo we move from roughly five thousand packets per second to around eight thousand packets per second. So that’s BIG TCP over IPv4 with Cilium. It’s a powerful optimization for high-bandwidth environments, but you do need a recent Linux kernel to take advantage of it.