Introduction All right I think I'll get going if a few people jump in a few minutes we'll we'll probably catch on so hey good morning thanks for coming or good afternoon or wherever you happen to be so I'm going to talk of the how the hive came to be the history of eBPF and how it fundamentally changed networking and tracing so I'll give you some of my insights and sort of experiences working on this stuff for the last actually it's almost 15 years. Now it seems like so I don't jump in so but first before we get going a couple housekeeping issues and logistics so if you join. You're muted okay when you join the zoom if you want to ask questions you can use the chat or the question and answers window either one will work. I'll I'll feel free to ask questions as I go along I will kind of glance over there at the questions in the chat as I go along. And if I miss it I might get a note that says do it but then at the end too we'll have time for questions as well so if you have kind of longer form questions or more general questions feel free to wait till then it's recorded and so. and we'll have a recording that you all the registered people can get a hold of all right sounds good so who am I recognize some of the names there and the participants but if I you don't know me and I don't know you I'm John Fastabend I'm a Tetragon lead for our recently open. Source project on Tetragon here I've been a Cilium maintainer for from very early on I think pre 1.0 release. I've worked on the Linux kernel for quite a long time now 15 20 years and I've been working on bpf since very. Early on as well I think 2015 or 2016 or something and we'll go into it and the sort of plan today is to kind of go over the history. Of bpf provide some insights on kind of how I view it and the different kind of pieces that I personally find quite interesting and hopefully everyone else. Will as well there'll be a little bit of technical stuff too but they'll also be kind of a little overview so what is agenda. So I want to talk a bit about the early days and the early days in this context is not 30 years ago when brand jacobson and steve mccain published the original bpf paper but more like 2014 when linux gets the bpf super powers right then we call this super powers we'll go into that a little. Bit and why folks call it the linux super powers also in this kind of era we have a bpm backend and all these other great things going on and then after that we'll talk about early adopters and kind of time box that and 2017 2018 2019 kind of where all these folks were grabbing bpf kind of early adopters it was still kind of rapidly being developed people? Were trying out new kind of new ways of doing networking and with BPF new ways of doing tracing with bpf and releasing a lot of really interesting and innovative. Tooling and projects on top of epf and then the last piece I'll talk about where we're at today where we kind of have. This community of bpf folks running through ebp io we call it the hive here so we'll go through that kind of three. Stages as I block them out so first I really want to just kind of jump back to what we were doing. In 2013 so if you were in networking kind of what did the landscape look like in 2013 this is kind of pre. Bbpf like before the original bpf patch submission landed in Linux kernel code and these are just some things that were top of mind for me I'm sure I. Missed something so apologize if I missed what you were working on in 2013 right but I think it's interesting so. Obs 2.0 was released in october of 2013 and so what did we get with obs we got open flow 1.1 we had Linux kernel. Support 10. 10. so there you go before the evening we have 4.x series. And they drop support for 2632 which I find it kind of interesting if you're working on the linux. Kernels these are pre 2 6 32 days or kind of back there aways if you come from the telco space at the time I was working on some telco pieces the etsy nfv stuff will probably ring a bill there was this release one release two. So early days of nfv what we see we're working on is how do we get virtualized resource management how do we default tolerance how do we have all these new. Protocols running in these telco environments so kind of early days of that stuff and we had dvdk and these user mode stacks were running. Around and the other not just dvdk but the other ones as well and then if you're focused on the Linux kernel kind of not in this virtual. Switch space you might have been working on ip tables right we had one for release and I actually went to this net. Filter conference which I was in 2014 this is a developer only conference so you get kind of the kernel developers. That are working on ip tables or some of the user-based system pieces they're working on ip tables there was a few folks working on obs. There and I'll talk a little bit about it in the next slide but stephen heminger who is a maintainer for some of the switching software and some of the linux kind of core code gave this talk about the switch apocalypse okay we're going to go into that in a bit but the gist is how do we deal with all of these new ideas coming into. Networking inside this framework of the Linux kernel I like to point these two sort of software release points out which I also. Find pretty interesting so you have rels7 just released on july 2014 and to give you some context. Just if you think 2014 from kind of a networking world and it seems a long time ago but if you fast forward. To today rel7 is just going to be end of life and Right. So, think about that everything. That was released in 2014 is still in production today Right. So, if you think about what are the designs what are the architectures what are the networking stacks look like pre-2014 that's what's in real seven right and of course they've done some add-ons and fix ups. But fundamentally that's the base of rel seven okay and just as a quick note if you happen to be running roll seven you have two years to update so maybe get working. but to kind of juxtaposition that against we also had the Kubernetes initial commit in june 2014. so you have like these two pieces of software right there's. linux distribution that's stable and robust and and has this huge light long lifespan right but maybe doesn't have the latest and greatest features especially when you think about today in 2022 and then you have Kubernetes this new piece of software that was released by google from their original internal stuff right and then got a whole bunch of additional users pulled in right and on the right here we kind of see the picture what did our networks look like. A cartoon of the network that I personally would be working on in this kind of time frame right you have a you have your physical network where you're worried about. Switching and ieee specs rfcs and then you have your network and then on top of that you have your bridge right it's an open virtual switch et cetera and then you have vms. And that's the model that we were kind of working on inside the Linux kernel at the time and like I said I would mention a little bit about the switch apocalypse. So what was happening here is as people were trying to innovate in this bridge obs kind of virtual machine environment they continually adding? New components new features Right. So, it actually ended up with lots of different types of bridges that you could use we had a bridge we had an ovs we had. Some folks were using tc to do bridging light which is our traffic control piece of qos piece of linux Right. So, all of these pieces were trying to come together and add new features. But what would happen is no one block would solve someone's exact problem. Right. So, what you ended up with is lots and lots of features that were sometimes unused by certain people. If you had a specific use case you needed a certain niche piece of obs or you needed a piece of tc and a piece of the bridge but the code itself contained all of these things right and everything that everyone could possibly need and sort of to add to that you ended up with this mix of standardized features. So we're kind of in this era you think vxlan openflow 1.1 openflow 1.0 etc but then we also had a mix of sort of non-standard things like hardware. Offloads were coming into play for some of the srob use cases direct attach we have stt if you remember this one. This is a sort of a non-standard way to get around some of the limitations of the hardware offload for example. In the various protocols so we just had a lot of features coming in that were being added to all these pieces right. and then at the same time we're moving from this top model here where we have the virtual machines the network stack running the bridge running. Obs down to this container model and you think Kubernetes and docker at the time and cloud-native and service mesh later on right. where you're going from two kind of things are colliding here different model so you're changing how ips are allocated they become dynamic. Configure the history of eBPF Right instead of a virtual machine where they were still dynamic but at a lot lower scale right you're changing your entire configuration so you're no longer using. Qamu and kvm you're using the linux network stack directly and we go from just a few virtual machines right you may be tens or twenties right and we thought maybe when I was working on the hardware at the time we thought okay 120 virtual machines should be plenty right like yes. Or I'll be we're how do we offload these things we set the limit at 120 right but now we're going to these container models where. The limits are very different right you're not talking about virtual machines you're talking about a small lightweight in the kernel a pod container that is sort of more. Efficient in some sense right and so we're kind of dealing with both of these things from the Linux kernel side but also from how do we. Deal with the networking in these environments Right. So, if you kind of put these two things side by side you have. These constraints that are coming from the networking world where we want to innovate create new architectures right we want new firewalls we want new load balancers new ids technologies. We're also bumping against performance problems right we went from these kind of relatively slower networks and now we want 10 40 100 gig and more right I was. Remember being at a few I trip leading me is what we're talking about 100 gigabit 400 gigabit etc keep pushing that performance boundary and at the same time inside the Linux kernel where you have to get consensus to put code in and it has to be sort of generally useful. you get this kind of environment where nothing is exactly matching the needs of your specific application right. And then so that's sort of the cons how things are working on the networking side but then there's also this sort of engineering model that's happening on the right it's kind of been in place on the linux side where if you want a new feature like what does that mean what does that involve what do you need to do. As a developer so somebody comes to myself or someone else who's working on the next kernel at the time and says I have this new feature that I need. Kubernetes right and you say okay well let me go and do that work and so you go in and you spend some amount of time in the Linux kernel you convince your colleagues that this is a good thing you convince the maintainers of your specific subsystem that this is a good thing you maybe go to a couple conferences and talk about it a few times right and eventually you get everyone to believe possibly with some changes and some generalizations so it's no longer exactly what you want but it's close enough. and you convince everybody that this is the code they want you submit it gets accepted right that's the best case right but now you have this problem right. You've got this code it's in the kernel and it's in the latest kernel right it's not in your customers linux distribution so how do you get them that code well. You have a couple options you can ship them a module right a kernel module and you say here's my kernel module but that very few people are going to say okay I'm going to take your kernel. Module right because kernel modules can. Crash the kernel right corrupt memory. Other subsystems crash the kernel for example so it's a very kind of high risk thing to be doing on the other hand they say okay well look I want this to go through the normal kernel flow I want the distributions to do all their testing all their vetting and all the sort of normal kind of things that we have become used to in the linux distribution. But now you're you're waiting a couple years in advance right a year out to two years now then you need to update your kernels. Deploy the history of eBPF As well right there's this kind of fleet deployment problem where you go from having the ability to just run user space software if it was a user space thing but because you're in the kernel. You need to reboot your systems somehow get that kernel on the nodes so what's gone from sort of a request. To development now has this very large feedback loop Right. So, if we want to innovate quickly and get that kind of software development to use our experience to feedback to update etc where we go back and forth and we look we can incrementally make better and better products or innovating more. interesting solutions we're stuck in this feedback loop that is really long times right we're talking years of things except for perhaps a few. Major players who can roll out new kernels but even if you're at a sort of data center a big data center that can do this it was still not something they would do frequently right. So you're you're kind of stuck in this cycle Right. So, what is our what is the kernels communities kind of answer to this was ebpa right. And what is eBPF this is what is on the edpf.o webpage and I kind of bolded the piece that I think is the most interesting is the ability to safely and efficiently extend the capabilities of the kernel so without requiring the change the kernel source code or to load kernel modules. Right. So, what that gives us is the ability to run these code that we just wrote we just got a request for we get that code. We can run it and give it to our customers they can try it with the sort of assurance that is not going to cause damage to their kernel right it's only going to be. Locked into that specific network feature or that specific tracing feature or security feature as we kind of extend the feature set of bpm. And this allows this innovation loop to start happening right I can write my code takes me a month or week or however long I give it to my customer or you. Long I give it to my customer or my internal customer if they're know my internal customer if they're internal they run it they say oh this is great but can you do this that and the other thing and I say sure or go. Back and we'll keep iterating and things get better at a much quicker pace than they would otherwise so if we think just as a quick sketch and I know there's a couple more. Webinars where we'll talk in more detail about this specific thing like how does the technical piece of it actually work but just to give you a high level view. Right. So, what does this mean it means I write a c program on the right there like I would any other kind of c program. Except for this c program is going to run in the bpf kernel right and so I compile it and then I load it when I load it's going. To go through this verifier stage where it ensures that the code is safe to run and what that means is it won't read random memory it won't write random memory it'll ensure ensures. That it can't crash that it can't run forever and deadlock my system and so on so there's a big list of safety properties that it verifies and then once it's loaded in the kernel I. Can hook all many different places in that kernel from the driver to the networking stack to the tcp stack to the file? System or even to the syscalls or even the user space piece so there's really a wide variety of places I can hook. Depending on my use case do I need to do I do something with the tcp stack do I really actually care about the networking layer 2 level. Layer 3 layer 4 or maybe the file system so on so it really allows me to target my use case. all right but that sort of gives you this picture of the fully formed bpf Right. So, at that point you're looking at kind of what is bpf look like today what it doesn't give you is the sense of how we developed all this stuff and so today I want to talk a little bit about the early days. How did we get here what were the challenges what were the really sort of milestones as we went along okay and so if we start what did we have? Before we had eBPF well we had yeah which we call classic ppf now and you can see that kind of most. Prominently if you look at tcp dump and I got a little example here tcp dump where I show like what is tcp dump actually run for filter code if you were. To say match port 80 of a tcp session right and you can see the assembly there and we don't need to walk through it line by line but. That is what it is so you can imagine that's kind of where we started okay simple jumps loads stores. Basic sort of instruction set. So we wanted to extend this daniel and alexa sent out some initial patch right that spent some time and working on this patch they. Built up this eBPF idea and they sent out a big rfc well the first thing. It was a lot of code right and it was a completely new idea and so it got a lot of good feedback but it eventually got rejected Right. So,. What is sort of the lesson that we can learn I think this applies to Linux kernel definitely but it's certain I think it applies to all sort of software projects Right. So, basically large? Changes they can be contentious they're difficult to consume right because they're so different but if you can break those large changes down into small incremental. Pieces that look familiar that are easy to review that people can understand without having to spend hours that you as the person building the speech you're. Spent getting your mental model working getting the architecture built up in your mind and not ideally on paper too so your colleagues can read it. But in the end if you take all of these small familiar changes you can build up something large like for example the eBPF infrastructure that we have today Right. So, that was rejected so then what was the next plan well exactly like we said. Incremental pieces of replacing the vpn classic vpn so you start by replacing the classic bpf and here you say well it's faster. It's more performant people understand these types of arguments right and then you slowly say well if I add this feature I can get this to work and if I add that feature I get this to work and so we slowly over the course of kind of 2014 this was built up right and so on the right there you see this patch that said okay applied and that was the initial start of kind of start date of eBPF in the kernel what's perhaps interesting is eBPF. Didn't land fully formed with a nice compiler and c code Right. So, after that first patch what happened if you wanted to write eBPF code you basically had to write bpf assembly this box here on the Right. So, people quickly it's really difficult to write large blocks of software this way Right. So, the next thing that landed. Was the llvm backend so this gives us the ability to write c code and this changes things dramatically Right. So, now instead of writing assembly code where you maybe have a few hundred lines at best and then now you have c code much easier to develop it. In linux 4.1 we got k-probe support all right and then now what this means is we are now able to attach bpf to k probes so. and if you're not familiar with whether k-pro is in linux it's an ability to attach a probe to a function call and so most function calls inside the Linux kernel or system calls cannot be attached with the k-probe and so now you have the ability to run bpf at pretty much arbitrary functions in the kernel okay now this is a really big jump for tracing so if. You imagine previously how did I trace or how did I observe some kernel functionality around a function is either there was a trace. Point there which some kernel developer or someone else had originally said okay I think people will want to observe this thing so I'll put something. There in the code right and not only did they do that they did that on the kernel version that you're running Right. So, you got to kind of so you're sort of stuck with what somebody else thought. You wanted and then also what sort of values that you would want on that okay probe okay this basically says okay now instead of. the history of eBPF observability requiring someone to think about this beforehand I can as a user of the system get observability anywhere in the kernel that I. think I would like it okay a really big change and we'll talk about some interesting things that came out of that I think first we'll just mention. Bcc all right it was created shortly after the logan landed and shortly after k probes Right. So, you get k-probes so I can hook anywhere and then you get llvm. So I can write code efficiently and now you have this new project called bcc its main goal and at least from my point of view is to create these programs that people can consume to hook all of these new places in the kernel Right. So, they give you a tool kit. Basically and there's also sort of a library with this as well so that if you want to go and write your own you have a sort of foundational place to start doing it. then we got networking tcbpf what this means connect and run bpf on every package in the system Right. So, this is a big deal if you think about those switches we were talking the switch apocalypse that was though I was mentioned in the beginning right everybody has their feature everybody wants to add it to the kernel well now. As a kernel developer I don't have to say okay do we want your feature or not you don't have to ask you can go in and build it in BPF and I think. Give it to your custom customers immediately Right. So, now instead of going to the kernel waiting for the kernel to get deployed waiting for your customers to update their kernel you're. Now writing your networking function inside tc and you can give it to your customers immediately and as a bonus we don't have. To agree right I no longer need consensus I don't have to convince everybody in the links kernel community that my feature is interesting. right I might even disagree with your feature entirely but that's the point here is that anybody can write the feature that they need that they believe. Developer impact In shortly after to address some of these performance concerns we get xdp and xcp. moved that hook point from the tc so from the software stack down into the driver Right. So, now instead of talking about all the overhead of a Linux kernel. That's optimized really for sockets at the end points right with traffic coming in we can get really high efficient load balancers firewalls ddos. Protection because we're running that code directly in the driver now so there's very little overhead basically you're talking about hardware. Copying the packet into the driver memory buffer and then this program runs on that buffer okay we're not allocating extra memory we're not doing. kind of all the overhead that a host operating system needs to do to deliver packets to sockets or operating down here at kind of the lowest possible. Software level okay and finally I just like to denote this with the release of 4.9 so 4.9 is. Interesting because it's supported by Cilium and it has been supported by cilium traditionally it's kind of the first kernel that it supports right and the reason is because it kind of brings together a good core set of the base features that you need to build really interesting networking. Functionality into the Linux kernel so that's sort of the early days we whizzed through it there in 10 minutes or so those are kind of the sort. Of basic building blocks are in place now so now we get to the point where we get people can start to build things and what are they building. So we talked about bcc so these early adopters I call them here because they are kind of in the 1560 out of the stuff was being built they were building. Software on top of it right it wasn't always fully formed but they were there building the software bcc came out and very quickly built up this toolkit of programs so that you could monitor almost any subsystem in the Linux kernel that you wanted if you want to monitor. The file system and you have file life file slower file top all these sort of command line tools that when you install bcc on your node you can log in and you. Can debug things you can start to see what is the file usage pattern all these kind of new insights that you get about your system that you didn't have? Before right and I can personally relate to this right you look at execs to see all of the things that are executing on your system all of a sudden you find things that you didn't know. Exactly right or you have tcp top monitoring all your tcp connections and you go hey why am I talking to this server every five minutes what's going on there so you get lots of interesting insights about what your system is doing but also. If you're debugging something you want to know why is the cpu slow I can do start doing things like monitor. Usage stack traces out of the system so on oops all of this led brendan greg at scale. 16 I think he probably said this before but I recall the talk he gave at scale 16 where he said super powers are coming to linux right what did he mean by these. Superpowers what were they Right. So, I listed a few of the things but I sort of summarize as some of these superpowers right I put at the top of the list ability to hook kernel functions right you can hook anywhere that gives you the ability to build all these tools and you don't need a kernel patch to do it just go in say you want to monitor your file things you. Can either use this tool I was written by the bcc team or you can write your own it's not too terribly difficult you best probably? Better to start with bcc and then decide what you need and go from there but you can do it we're getting multiple architectures right we're not. Just x86 anymore we're arm x86 you'll get power pc and risk and so on coming into play as well. We have stack traces now this is super interesting if you want to make flame graphs you've seen some of the flame graphs that brendon greg talks about as well you want to see where your cpu time spent etc we're starting to build up a really solid list of data structures. Inside bpf we have arrays of hashes lrs lpms queues etc this allows you to build start building complicated software. Btf came out about this time this is it's a type BPF type format if you're not familiar what this does is it allows. You to run your program on lots of different kernels so if you think about what is the internal structure of a kernel look like they're sort of different between different kernel. Versions and what btf does is that load time so that you as the author of a program don't have to worry about it lets you run your program on lots of different kernels so when you say I want to reference look at the task structure the structure inside linux that every has it backs every process and I. Want to look at the pit inside that or something pick your favorite field what this will do is let you run your program. on any of the kernel and it'll transparently at load time find that for you so that you don't have to as an author say this only works on 419 this only works on five four so on and so forth right we started hitting the limit of complexity in our programs we were using too many instructions so. Instruction limits went from a smaller number to one million so now you can have really large programs relatively large compared to how they were before. And we got bounded loops so now you can start to do things like walk lists of headers right walk lists of task structures and so on block the file system. so on and so forth so this is really I kind of summarize this as the 5 4 lts kernel and you start to really get a powerful bpf. Engine in place and then on top of this we've got some sort of some innovative things going on here bpf. Trace came out about this time this is a tracing language so a dsl for tracing Okay. So, you don't need to know how to write c code and bpf loaders and all the kind of things that these system people like myself find quite interesting but if you're just if you're an ops. Person and your focus is I need to get into these systems and I need to understand this kind of my environment then bpf trace is a good. Avenue for you to go right it gives you a high level language you dsl for tracing you can get a lot of interesting feedback quickly. And the quality of the bc's tools continue to get better they're sort of more and more of them as we go along. And I dropped this in here because I think it's an interesting point and I remember watching the reinvent talk from by brandon and greg where he said all. Right netflix has over 150 000 instances and oh by the way every one of them has probably at least 14 active bpm programs. Right. So, at this point the tracing is not an experiment it's not something that people use just for debugging this is kind of a real. Production ready widely deployed infrastructure piece of core infrastructure at some of these main. Data centers and then if we switch over so that's the tracing side if we switch over to the networking side for example we start to see all kinds of things happening Right. So, instead of and I this is my sort of view is instead of before when you were programming the data path think of obs when you write a rule into a table or think teach if you're thinking like the bridge you write an entry into the bridge instead of programming the data path you're. Actually going to create the data path and so what we see is a whole series of different types of databases coming online so we have the Cilium. Cni which I worked on obs has a bpf version p4 if you come from that kind of switching programmability side. This is basically a dsl but it also has a BPF backend that folks worked on if you're more in the user space. We have af xdp this is a way to couple bpf with user space stacks such as dvdk and then we have a lot of specialty things lvl load balancing important and proxies and so on right and I think what's really interesting. Here is the kernel developers didn't have to okay every one of these Right. So, you're running and you're able to innovate in your in your space with your very specific product knowledge or your technology knowledge that you have and you bring. It to the problem set Right. So, at this point time we release 1.0 and april 18th this is the cilium cni it was foundationally built on bpf Right. So, at the core of Cilium is eBPF and a very early cni I the first one to sort of base everything on eBPF from a connect containers viewpoint as part of this right we had to build up a whole lot of a bunch of different. Primitives that didn't exist before so on the networking side you can get something that probably looks familiar redirects checksums rewrite. Packets extended string packets for adding headers removing headers cilium pushed features into xdp Right. So, instead of going into this linux. Stack you're now down in the drive we're working on load balancers firewalls ddos af xdp as we mentioned came in that time and we also do some socket based. Stuff so now instead if you really care about socket stuff you don't have to wait for the linux stack to process the packet you can go and put your bpf program at the socket. Sort of interesting ideas and I think quite useful around this time as well we start to see other people katron which is the facebook load balancer it's a c plus library but it's also something they deploy right and this takes the idea of putting. A load balancer down into xtp for very high performance and then coupling it with the linux stack for other things that you might need maybe your you have your fast path and your slow path for example I pulled this little quote here out of the blog post that they had in their open source release of this and I think it's interesting so you've taken what was traditionally the realm of hardware fpgas and now because the software is. Getting fastened up flexible enough you start to see some of those things that have traditionally been done in hardware moving into. Software right and not only like dedicated software stacks but software stacks that are integrated into a linux operating system. this has a series of advantages it runs on your commodity hardware you don't need special hardware but it also coexists with other services. So you have your normal stack but you just add this in you can update it right if it's a hardware feature you traditionally I will be a little bit. More tricky to update right you can't just do a hardware update without downtime and it's easy to debug an instrument. If you want to know what's going on in your xdp load balancer or you just add a bpf debugging statement there. Some other interesting things that fell out at this time is the L4 drop this is a ddos that came from clouds there and the graph there I think on the right is really the key takeaway here what you. See is by putting a ddos in xdp they're able to absorb these ddos. Attacks in their production network with very little overhead and very little lost pockets so we see kind of at the spike there. Right in here you see that they get hit a ddos attack the eight million packets per second are dropped cpu usage in the sort of purple color there goes. Up slightly right ten percent or so you can study that the links are at the bottom by the way you're sort of like that piques your interest you can go and read the whole. Blog post where they go into much more detail the other interesting thing that I note here I'm not sure that they noted in their blog but you avoid vendor. Lock-in lock-in Right. So, instead of requiring a specific nic to do this if you were doing those sort of hardware features. Because it's based on xdp and by this time most of the major networking nyx would support xdp. You could very easily run the same program on a whole host of different nics different vendors and so on I think this is an interesting case point as well because not only did they build the ddos piece they shortly after a couple years later. Decided that they needed a new load balancer and they again looked at xdp Right. So, you see it's sort of a successful use case with ddos let's. Repeat it with L4 load balancing and here's what you get as you see on the left side of the white line in the graph you see the previous or xdp right and so what's that showing is the load on a whole bunch of different nodes. Right and you see the sort of wider differences between them because the fairness is not quite as good. Right and so they turn on their alpha low bouncer and xdp all of a sudden you get one tight grouping basically all of this kind of differences in fairness are reduced and then on the right we have a another interesting note now we're starting to see xdp pipelines. Right. So, not only just a single thing in xdp you see here they have a pipeline L4 tdos control L4 balancing. So if I were to sort of summarize this and take away what can we take away from this right there's a lot of traditional I think. When we started doing this we took especially when we went to virtual machines we took sort of traditional middle boxes and moved them into the kernel all right you think of obs it's a virtual switch there's a real hardware component out there and then we moved it into this virtual switch environment right and we very almost literally right there's a one-to-one correspondence between the two Right. So, you can have an obs and hardware you can have obs and software right and they can do the same thing maybe the software is a little bit more feature-rich but they're both. Running open flow for example and then when we think about it what how do we benchmark these things how do we test them we talk about per packets. We talk about throughput and so on but sorry I jumped ahead of time but now if. We think about this as we go to bpf we're in the os and so we started to see different types of switches different types of load balancers so. Instead of looking at kind of what a middle box would have done in the hardware where they see every packet we see things like let's just do load balancing at the socket layer socket. Layer load balancing or socket layer firewalling where if a rocket tries to connect to it. Connect to an endpoint that they shouldn't be allowed to connect to I just stopped the socket from connecting right and what we've done there is we've. Changed the equation somewhat so now instead of talking about per packets we've changed the scale we're talking about per sockets Right. So, bpf allows. These sort of new kind of innovations to happen right new types of firewalls new types of load. Balancers and so on so if I just some high level takeaways one thing you can do here right is reframe your problem so. Before we'd always said how fast do we do packets how many packets per second can we push so one way to reframe that the problem is at least for local traffic you can say well let me just. Move that little dancer into the socket I mean it's kind of reframe it as not allowing sockets to connect to things the other one we've sort of used. Successfully is we've moved the solution Right. So, previously if you look at that top kind of original story with virtual machines the obs was in the stack. it was part of the Linux kernel instead with xdp we have moved that problem and put it into the driver right I think in 2014 if you said well we're going to have bpf code. Running the driver people would say okay that's interesting right probably but that's not how linux does things right eBPF allows. For these types of things so what we do when we do that then we get these kind of solutions load balancing. from cloud fair load balancing tron and facebook ddos mitigation also saw that at cloudflare as well some of the things that cilium cni is doing as well with ingress controls and so on the other thing that I think is. Really important as well is you're changing who's solving the problem to some degree right the people that have the most direct knowledge of the problem can now go and solve the problem on their kernel right the content experts can now write bpf programs and work or perhaps work with people who are the BPF experts very tightly and build this kind of really nice feedback loop where before you sort of kicked it to the kernel developers in a year or two you got your result. Right now you can work very tightly you can get this really great feedback loop going and just one more and this is the Cilium example here and basically an xdp load balancer for cilium and if. You look on the left you can see a L4 lp xdp so this is the cpu used by the load balancer in cilium and then at some point we see the ipvs take over so ipvs would have been the traditional way. Of doing load balancing in the Linux kernel to do load balancing and Linux kernel. By removing that problem into the driver we get this sort of dramatic improvement right if you want to read about that the link is down there as well. Another interesting example previously if you want to work on tcp stacks you had to work on the Linux kernel because that's where tcp stacks would have run in sort of this Kubernetes in a traditional computer linux stack right where you have the linux stack is the sockets Kubernetes on top. So but now if you want through bpf you can write your own congestion algorithm if you'd like now you better know how to write congestion algorithms because. That's what you're doing but this allows folks to really innovate again right like you can deploy this to some set of your servers and say how does my new. Algorithm do compared to what exists so that's sort of the early adopters let's do a quick update of where we are. Today so in 2021 we have the announcement of the eBPF foundation. These sort of initial group of initial foundation members you have facebook google I surveillance. If we look at what that is today and now it's growing and you have a whole host of members right on different levels here but you can see in many of these. If you go back to some of our examples you'll see that they were listed in the early adopters as well. We have a healthy ecosystem Right. So, you go from application space you have bcc cilium bpf trace falco katron so on and I think really notably as we now have multiple operating systems supported Linux kernel and windows Right. So, we've gone from sort of a linux only technology which I've been talking about earlier to now a linux and windows technology right we have lots of library support. Across different languages go see rust and then we have a whole host of emerging applications and go to eBPF.org you get a longer list. The other thing I just wanted to point you on one of the last slides is it's not like epo evap is done Right. So, in 2020 google added lsm which brought. In security this allows you to allow and deny different actions in the kernel openssl 3.0 was released just last. Year and had ktls and bpf support so you could work with bpf inside ssl sessions we're always working on the sort of cutting edge of security with the spectre and so on all that happened to 29 2020 cloud providers are coming on board you now have aws and azure who. Both support xdp sysmon for linux is now was released by microsoft and then the features the ability to do new and interesting things inside BPF just continues to grow we have spin lock support timer support. Iterators if you think of like a c plus iterator all of these kinds of things are now safe pointers is a good example of something that's that's I think really. Interesting on the bpf side and then Cilium continues to grow I see them we get new projects there was first we did Hubble. Which was the observance networking stack observability piece we recently released Tetragon if you want more interesting and more info on that there's a link there to the github page and we've also done a few other webinars about it as well so let me give you the summary just kind of the high level here we. Started this journal in 2014 alexa and daniel released the initial patch and very early on a lot of folks jumped. In to start working on this we're grateful a handful of people also saw that it has sort of the next wave of things and jumped on and started building really innovative technology on top of it and like we said pcc all the cool tracing. Netflix deploying it and we've got Cilium cmi tron cloudflare building load balancers and so on right and then the other one I just want to point out is it doesn't seem to be slowing down right it actually seems to be accelerating we're getting more users and more features and the everything continues to expand so I think it's a really exciting at least I find it really exciting I hope everyone here also is excited about this growth. So with that if you want to learn you so with that if you want to learn some more things we'll have another know some more things we'll have another deep dive technical deep dive coming up they're listed here. for eBPF and Cilium there's a book what is eBPF that you can look up there's a security book if you're interested more on the security side. Go to the link myself.com resources there and events and you can get all of the details. There's an eBPF summit coming up september 28th and 29th I've gone to the last two and they're really interesting you'll get lots of great talks from folks I highly. Recommend going to take a look it's really interesting stuff and with that I'll say thank you I you and with that I'll say thank you I also want to make sure we give know I also want to make sure we give. Credit to all the folks that have been working on this stuff so thanks to daniel and alexei and all the maintainers david miller and I. wanna get some Cilium team the whole BPF group of folks and like I said all the adopters couldn't couldn't have. Done all this work without them so in many ways I'm just that just the messenger today but yeah also thank you for coming and if there's any questions I'm I'm happy to answer them now I'll take a look. I'll leave this slide up it looks like there's a couple over here okay. Why is no BPF tracing tools for ip and net devices on the picture early adopters okay no tracing tools for at the ip and the net device layer inside the bcc toolkit they could have been there there's. Definitely ways to trace at that level if you have a specific example that you're thinking of that I missed. I'm sorry about that but yeah I definitely didn't mean to exclude ip and that device tracing as well will the recording be shared after. The session yes for the if you registered with an email my understanding is you'll get an email with the recording. And one more question does Cilium Hubble gather any information from monitoring from xdp and what exactly if so after all eBPF and xcp are mostly. Used for fast routing background and so on so I think the short answer is yes it gets information from the multiple layers of the Cilium. Stack the exact details of that I think the webinar that joe is doing on Cilium. We'll probably cover some of that and then I think it's a technical deep dive next one I'll probably talk a little bit about more the technical aspects of how you would. Go about making that happen but I think the short answer is yes I off top my head I'm not exactly sure what you my head I'm not exactly sure what know what. fields would be exported by xtp but you could imagine things like I want to know every flow that my load balancer. Is going to redirect Right. So, I want to know statistics about my load balancer how many flows exist in that load balancer what are those points? What protocols are being used for example maybe I want to see all my DNS entries or something so those are sort of made up ideas I'll stop my head having worked on it? They're slightly informed but I don't know the exact details right now but yeah I would encourage you to go to the next webinar I think the Cilium one will give you. Probably more detail about those specific topics. anything else. Okay if there's nothing else you can focus on maybe 30 seconds to ask going once going twice alright. Well with that I think everyone for coming I really appreciate it and I think we'll hope to see you at the next. webinar on bpf technical on deep dive I'll definitely be there and as well as the pacillion one and have a good rest of your week or afternoon or evening wherever you're at thank you.