New features come in new versions of Cilium. But sometimes, there is the desire to have new features, or even bugfixes also incorporated into older versions - this is called a backport. In this blog post, we shall share some insights on how backports happen in Cilium, and provide some examples for the various types of backports that have been implemented in Cilium. What are Cilium backports? The main development of Cilium takes place in the main branch of the Cilium repository. This is where the bleeding edge development takes place: new features, new capabilities, new ways to solve the networking problems of tomorrow are added here. Once these contributions are deemed stable enough, a new version of Cilium is branched off and released. Major releases, such as 1.14.0 are focused on additional features, whereas minor releases, such as 1.14.1 are focused on patches for issues and CVEs. This most recent major version - 1.14 as of writing - is what Cilium users deploy when they want to have the most up-to-date version of Cilium running in their environment. However, sometimes users can't or don't want to update to the most recent version. There are various reasons why customers might not be able to update to the current Cilium version: Compliance Dependencies further down the tool chain No time to perform the update due to competing priorities Not enough resources to upgrade every few months And others. But what if a user requires a patch that was incorporated into the most recent version of Cilium - but still wants to stay on an older version? That is where backports are coming in. With backports, part of the Cilium code base, such as a given feature or a specific bugfix are taken from the most recent version, and incorporated into an older version. For example, recently, there was a race condition in clusters with ipv4/ipv6 dual-stack nodes that could lead to broken policies. While this affected only Cilium 1.13, it was fixed in the main branch, on Cilium 1.14. The Cilium 1.13 users, who couldn’t update to 1.14, were happy to hear that indeed a backport of the fix in 1.14 was done, fixing the issue in Cilium 1.13.8. 🎉 Unfortunately, backporting is harder than it looks, let’s dive into why it can be a complex undertaking. Why are Cilium backports so tricky? Unfortunately, when a new fix is introduced, bringing it to an existing released version of Cilium can be complex. Backports take an incredible amount of work, you need to find a way to stitch in those new capabilities into the old code, you have to fit it around existing constructs that are already well established in the older versions such as verification and testing frameworks. Furthermore, you need to ensure that future forwards compatibility is not impeded, so that updates to later versions can still be performed. Given the complexity of introducing backports to older software versions, most engineers understandably want to work on newer features and releases, combined with the fact that backporting new capabilities are not typically prioritised by the broader community. This is because the unique value of the backport is rarely recognised by a larger number of users, as it’s available in the newer versions. Often, backports are not attempted by most, due to the lack of time, skills and perceived value recognition. However that doesn’t mean it’s not possible! Larger companies, such as Datadog, who have a team of Cilium experts, backport features to their own forked and self-managed Cilium OSS branch. This does provide them the flexibility they require to operate their platform and still take advantage of improvements in later CIlium versions, but as discussed, it also means they not only have to manage the backports themselves, but also take on the responsibility of merging into forward facing versions of Cilium OSS from the community each time a new release is available. If you want to dive into Datadog’s journey with Cilium, and understand running a cloud native platform at scale, I highly recommend the below recording from CiliumCon North America in 2023. What are Enterprise backports provided by Isovalent? As the creators of Cilium, at Isovalent we have a large engineering team dedicated to building and implementing Cilium’s code base, and an experienced customer success team focused on understanding customers needs and outcomes. As part of their ongoing enterprise support contract, Isovalent Enterprise for Cilium customers will benefit from backports to address their specific issues to older supported Cilium versions. From an architectural perspective, we can help guide the customer through the decision making process to ensure that the implementation of a backport is the correct path to address their specific needs and constraints. We will also provide on-going support to older Cilium versions and installations, ensuring backports compatibility long-term. This benefits the wider community and not just Isovalent customers - Isovalent have contributed a number of backports to both Cilium OSS and Enterprise offerings, covering areas such as security CVE patches and major bug fixes. What are examples of Cilium backports? At the start of the blog post, I highlighted a recent backport that fixed a race condition that could lead to broken network policies. Let’s now look at two larger backport efforts from Isovalent for our enterprise customers. Backporting new Cilium components When Cilium was donated to CNCF, its network observability component Hubble was developed in later versions and then donated alongside Cilium. This also meant that in the open source community both offerings could be developed upon not only by Isovalent, but the community also. One of the first major Hubble initiatives, as part of Cilium 1.8, to be developed after being donated, was the Hubble Relay component, giving centralised ability to pull data from each Cilium node in the Kubernetes cluster. This was an important feature to bring to Hubble, improving usability and scalability of the Cilium platform. However, for those existing users running older versions of Cilium, if they are unable to immediately move to a newer major version of Cilium, they couldn’t take advantage of this new feature. Listening to our enterprise customers, it was clear that Hubble relay was an important component for the success of their platforms, and the engineering team at Isovalent set to backport this new feature to previous editions of Isovalent Enterprise for Cilium. Backporting this feature was a complex task, as the first iteration of the Hubble server was standalone and not integrated with Cilium agent, as we see in current versions. This meant that information sharing between Cilium agent and Hubble components was not available in older versions, and a new method had to be designed and implemented, pulling the node data from the Kubernetes API instead. Of course, implementing a new data access method in a backported feature, also meant that further consideration had to be given when it comes to forward compatibility of the software when customers want to upgrade to later versions. Backporting Cilium and Hubble security features The final example of a backport is that of Hubble TLS. This feature was first introduced in Cilium OSS 1.9, improving the security of the platform. Due to Hubble’s recent addition to Cilium, development of the code and supporting artifacts, was moving at rapid pace, this also led to more complex backports, one area was the refactoring of helm charts to deploy and configure Cilium and it’s components, which was massively overhauled in Cilium 1.9. These helm chart changes needed to also be backported into Cilium 1.8, to enable the configuration of Hubble TLS, which not only updates the running configuration of the containers associated with Hubble, but also creates kubernetes components such as certificates and secrets. Both of these backports provided Isovalent enterprise customers further stability and new features to their existing deployments, without having to undertake a wider upgrade program to a new major version of software. As an enterprise customer, you can continue to benefit from the developments made in both Cilium OSS and enterprise releases, however as your environment becomes more complex, or if for example, you are constrained by regulatory requirements, you can work with Isovalent on roadmap development to ensure that you are able to benefit from the changes, fixes and improvements from newer releases into the software versions that underpin your platform. Conclusion In an enterprise environment, it is well understood that keeping up-to-date is a key part of securing and maintaining a healthy platform. However as we have covered in this blog post, it’s not always viable for enterprises to upgrade to the latest major versions of software, they may find they are restricted by regulations or other considerations, but still require a certain feature, bugfix or CVE patch. Backport’s provide an important method of ensuring stability and security for many Cilium use cases. As you have learned, a number of Isovalent Enterprise for Cilium customers today benefit from our expertise in this area, ensuring that their business cases are met, and they continue with a vendor-supported platform that meets their needs. If you are interested in learning more or discussing Cilium roadmap items and potential backport projects, reach out to our team today!