Cilium BGP Graceful Restart

Transcript

Welcome to this video on BGP Graceful Restart on Cilium. Alright, it’s a new feature that is coming with CNI 1.14, and it makes your BGP configuration more robust. If you look at this diagram, it’s going to explain it a little bit why, but we will also look at the demo with and without Graceful Restarts.

BGP Graceful Restart has been around for a long time in the world of BGP and routing protocols, but we are introducing it into Cilium. The way BGP is done with Cilium right now is by running a BGP Daemon within the Cilium agent. So, you can see when your BGP session is up and running, you’ve got, you know, you can advertise your pod IP ranges, you can advertise your service IPs to your peers, and it works great. But, for example, you can see that if I look at the routing table of my peering router, I can see the routes at 10.0.0.24.

Now, what happens when I restart the Cilium agentm when I do an upgrade? Because the BGP daemon is embedded within Cilium, as soon as I restart the agent, the BGP session will go down, and prefixes will be gone. We’ll see this in the demo in the video. That’s not ideal. We want to be able to separate the control plane from the data plane. When you reboot Cilium agents or do an upgrade, you don’t want an interruption to the traffic. What Graceful Restart is able to do is, for a period of time while you’re restarting the agents, your peers will be able to keep the routes in their routing table. For example, you know, say 20 seconds because we know that after 10-12 seconds, the agent would have restarted. So, that’s really a way to gracefully recover from a BGP event or a restart of your agents, and that enables us to separate the control plane from the data plane.

Now, let’s have a look at a demo.

So, if we start with my environment again, I get my peering here. I’ve got one BGP session between Cilium and its remote node, and I’m going to connect to the node to show some of the information we see from the side of the peer with Cilium.

Right now, Graceful Restart is enabled on the peer as a helper, which means it will be keeping the BGP routes learned from Cilium while the Cilium agent is restarting. If and when Cilium is enabling Graceful Restart, right now, the remote Graceful Restart mode is disabled, which means on Cilium, we’re not using Graceful Restart just yet.

If we look at the routing table, you can see that we’ve got the route learned from Cilium, and it’s installed in the routing table. Okay, it’s again, it’s working great.

Now, let’s see what happens when I restart the Cilium agent again. We don’t have Graceful Restart enabled.

So, BGP is gone, and the BGP session goes in Active. The route to 10.0.0.0/24 is gone from the routing table. So as soon as I restart the agent, we get this interruption, which again, is not ideal.

So, let’s enable Graceful Restart to just see the difference in behavior.

The Graceful Restart configuration is within the Cilium BGP peering policy manifest, and it’s a per-neighbor configuration.

So, we’re going to enable Graceful Restarts, and that’s going to renegotiate the session with our peers. And we have a restart time of 20 seconds, which means that our peers need to keep the routes for 20 seconds. And then, if we still haven’t heard back from Cilium during that time, they will kind of tear down that relationship with Cilium.

Let’s save this and apply it.

This time, still negotiating right now.

The session is now established, and you can see that the remote Graceful Restart mode is now in restart. So our Cilium has enabled Graceful Restart with a restart time of 20 seconds. So, again, it means peers should keep the routes for 20 seconds while Cilium is restarting.

Now, we’re going to have to be pretty quick, but we’re going to again execute Cilium daemon-set restart and just see the difference in behavior.

So, what we want is we want to show the IP routes, and we want to show the BGP relationship.

Again, right now, you see it’s established. So, let’s restart the daemon, and we’re going to check the routes.

And so, what happens is, no, I restarted the agent, but this time, the route is still there in the routing table, and you can see we are NSF. You know, the session is not just active, it’s non-stop forwarding. So, we are still keeping the routes in the routing table. We are still forwarding traffic while the Cilium agent is restarting. And you can see how long we are keeping these routes up, up to 20 seconds.

Now the session is re-established. The Cilium agent is back up and running, and the session has been restarted. You know, it’s normal again.

I’m going to show you one more time what happens because, during this kind of restart period, you can see the session, the route, is marked as “S,” which is stale. And I guess that’s just a way to identify routes that are not healthy as such because we know that there’s an event happening on the other side, which is why we’ve got Graceful Restart enabled here. But, within these 20 seconds, we’re still forwarding the traffic. And then, a few seconds later, against the agent, restart has been successful. The BGP session has been properly reestablished, and the session is marked as valid.

And that’s it. That’s just again to show you how we can handle the Cilium agent restart more gracefully and not interrupt traffic by using BGP Graceful Restart. That’s it. Thanks very much for watching