I'm going to make an introduction without much ado, talking about high-speed packet filtering and Cloudflare from the London office.
Welcome to DEF CON, and my pleasure to introduce you to Gabriel Tobler-Tinn.
Thank you.
Okay, good morning. Thanks, everyone, for joining me today.
Today, we talk about past, present, and future of high-speed packet filtering on Linux.
So, let's get started.
About me, I'm a system engineer at Cloudflare.
I work in the DDoS mitigation team in the London office.
For the ones of you who don't know what Cloudflare is,
Cloudflare is a security and performance company which offers a reverse proxy CDN service.
So, when a site is on Cloudflare, all its traffic is routed through our edge network first,
and then it's proxied back to the origin servers.
This means we can provide different services,
such as caching, content optimization, and, most importantly, for the sake of this talk,
we can provide DDoS mitigation.
We have a fairly large network.
We see a lot of traffic every day, and also a lot of malicious traffic.
And, in fact, on a daily basis, we have to deal with hundreds of different DDoS attacks.
To give you some numbers, on a normal day, we see attacks ranging from 50 to 100 million packets per second,
and from 50 to 250 gigabits per second.
And, we have also seen much bigger attacks.
These graphs, for example, show one of the biggest TCP-SIM flows we recorded a few months ago.
The green line represents the volume of the attack traffic,
while the small yellow line at the bottom represents the larger traffic.
And the job of my team is to basically drop all the malicious traffic before it hits our servers,
without affecting in any way the small portion of yellow traffic.
So, let's start from iptables.
In the beginning, we were relying simply on iptables to drop traffic.
And, iptables is great for many reasons, because it's a well-known tool.
It's relatively easy to interface with it from different programming languages.
It has this nice concept of tables that hooks into different parts of the network stack and chains.
It's also well integrated with the Linux kernel, and it has support for BPF management.
Which means it's possible to use BPF bytecode to specify how a rule will be matched.
And, this gives iptables a reasonable flexibility.
And, speaking of BPF, in the last few years, we've been developing a set of utilities called BPF tools,
which allows to generate on the fly BPF bytecode to match a specific class of packets.
In this example, I'm basically generating BPF bytecode to match a specific class of TCP-SIM packets.
And, without going too much into the details, the weird string that you can see passed through the BPFgen command
is called pofsignature, and it's basically a short description of all the interesting fields of a TCP-SIM packet.
And, the way we used to use iptables is the following.
We use the usual iptables syntax to express most of the mitigation logic,
and we resort to BPF to express those bits that cannot be expressed by just using iptables.
And, this works great.
But, there is a problem.
And, there is a problem.
With iptables, we can't handle big packet flutes.
And, last time we checked, we could, like, do two, three million packets per second on a single box,
leaving no CPU at all to the user space applications.
And, this is, for us, a problem, because on our network, each server, which serves HTTP and DNS traffic,
also participates in DDoS mitigation.
So, we can't just afford to spend a whole box just mitigating DDoS attacks.
There are a few Linux alternatives, but we didn't really consider any of them.
And, the reason is that we are not just trying to squeeze some more million packets per second of DDoS mitigation capacity.
We'd like to have a solution that we can use, that uses as little CPU as possible to filter packets at line rate.
So, before discussing our current setup, I'd like to give you a quick introduction about the path of a network packet in the Linux kernel.
And, this is to make it clear why it's so expensive to filter packet by just using iptables.
So, first, let me define a couple of important data types that I will refer later during the talk.
So, when I speak about packet buffers, the one at the bottom, I'm just referring to memory pages which contain the actual network packet.
When I'm speaking about SKBuff, I'm speaking about a big data structure that contains a lot of memory.
So, basically, it's just a structure that keeps the context about a specific packet while the packet is being processed by the network stack.
And, when I'm speaking about Eric Strings, the one on the nick diagram, I'm speaking about data structures used by the network card to keep track of all the packet buffers that are actually in use.
Okay, so how does Linux receive packets?
Linux uses a mechanism based on polling.
So, basically, periodically, Linux invokes the NetRx action and goes through the list of network cards.
And, for each network card, it checks if there are new packets.
And, the important point of this slide is that just the network driver has to do a lot of expensive operation, memory operation, to deal with the packets.
It has to DMA and map the packet, allocate an SKBuffer, and allocate a new page for a new packet that will replace the old one.
And, actually process the packet.
And, in the end, free all these data structures.
So, there is a lot of memory pressure just from the point of view of the driver.
So, I want to really quickly go through this code trace of the network stack receiving a packet.
Just to give you, again, a quick sense of how many operations the network stack does when it receives a packet.
And, so, why it's so expensive to actually drop them.
So, as I said, NetRx action is called periodically, which goes through...
Which goes the driver function.
And, we have to allocate SKBuffers.
And, there is some generic receiver flow processing.
And, this is repeated for each packet.
Then, the driver has to allocate new pages for the new packet that will arrive.
And, there is IP header processing, IP tables, row contract, and more contracting, routing decision.
Finally, we encounter the IP tables chain, which is fairly late in this whole list, of course.
And, finally, layer 4 protocol handling and IOM meeting, other things that we don't care here.
So, what's the point of this?
The point is that we should not say that IP tables is low.
We should say that IP tables is just executed too late into the network stack.
And, we have to do all these actions for each packet that we are going to drop.
And, that's why it's so expensive to drop packets.
And, so, we should move to our current solution, which is based on user space upload.
And, when I say user space upload...
I mean offloading to user space the task of dropping the packets.
So, user space offload is based on a technique called kernel bypass.
Kernel bypass means taking some of the network card receive and transmission rings, mapping them in user space, and allowing a user space program to deal with them.
So, basically, partially the network card is detached from the network stack, from the Linux network stack.
And, the user space program can take control of that.
So, it can write packets from there and reads packets from there.
And, one of the reasons this technique is used is to implement user space network stacks.
Because a user space program can just read and write packets from the NIC directly without having to deal with the kernel.
But, this technique is also useful to filter packets.
Because you can map a receive ring in user space.
And, a user space program can start reading all the packets.
From there, without any interference from the kernel.
And, then the user space program can filter, actually, what are just buffers of packets.
So, the way this works is by just selectively offloading the attack traffic to a specific receive queue.
And, then after that, a user space program will be able to read the packets from the queue.
And, if the packet is a good one, it will just be re-injected into the network stack.
And, if it is a malicious one, there is nothing to do, really, because of the way this framework works.
So, instead of having to allocate and de-allocate packet buffers, like the Linux kernel does,
this framework usually uses circular buffers.
So, there is no real work to do in case we are dropping packets.
And, since, in case of big floods, we are dropping, like, 90%, 99% of the traffic,
this is a big save in terms of performance.
So, we use a technique called partial kernel bypass, because we keep the network card,
most of the receive rings attached to the kernel network stack.
And, we bypass only a single or a couple of receive rings.
And, we use what is called an RxFlowHash rule.
So, basically, we try to steer all the potential traffic to a RxQueue.
In this case, RxQueue.
And, then we filter it.
And, when I say potential bad traffic, I mean, for example, traffic that is targeting a specific destination IP.
So, we can leave the kernel work on the normal traffic, as usual,
and we can filter in user space the bad traffic.
So, there are a bunch of, like, different frameworks to do this,
that do exactly, more or less, the same thing.
And, in fact, from the user space point of view,
the way a user space filter is done is the following.
So, a user space program keeps asking the network card for new packets.
And, as soon as there is a new packet, the program goes through a list of rules.
And, if any of them is a drop, the packet is dropped, which means we do nothing.
And, if a packet is not a match for the rule, it means it's a good one,
and so we reinject it in the network stack.
So, this solution works great, because we can, like, filter six to eight million packets per second
on a single packet.
Compared to two, three million packets on a whole box.
So, this is a great performance gain.
But, unfortunately, there are some limitations about this solution,
because legit traffic has to be reinjected,
and this can be, sometimes, expensive, depending on the framework that we are using.
One or more cards have to be reserved completely for this kind of thing,
because the way we do it is by busy polling the NIC,
and this is because we want to keep the latency as low as possible.
And, also, you have to pay a lot of kernel space and user space content switches
every time you move from kernel to your filtering user space program.
And so, the future, XDP, or Express Data Path.
So, XDP is a new technology recently introduced in the Linux kernel,
and it's a concrete alternative to IP tables or user space offload.
And the idea is to filter network packets as soon as they are received by the network card.
So, using an eBPF program, which will take as input a packet,
and it will produce as output what is called an XDP action,
which can be XDP pass, so the packet should proceed to the network stack,
XDP drop, so the packet should be dropped,
and there are even other actions and other cool things that you can do with XDP,
but that we don't care today.
So, if you remember the five long slides call trace,
this is where XDP is run.
It's just at the beginning, just as soon as you receive a packet inside the network driver.
So, before even allocating SKBs, you run your eBPF program,
and you decide if you want to leave the packet flowing through the network stack,
or you want to just drop it.
And if we take a look at one of the drivers that implemented XDP,
we can see how the XDP drop actions is actually similar to what we are already doing with user space offload.
Because, basically, there is nothing to do with it.
If you want to drop the packet, you just leave, you just do nothing.
You just leave the packet buffer where it is in the network card ring,
and you go to the next packet.
So, there is no memory allocation, the allocation cost at all.
Yeah, so XDP has the same advantages as user space offload.
There is no kernel processing involved,
and there is no memory allocation and allocation cost,
no DMMA map and map cost,
which is a very expensive operation.
But also, it has a couple of other advantages,
because you can use eBPF to program your filtering logic,
and there is no need to re-inject packets,
because if a packet is not going to be dropped,
it will just go through the network stack the usual way.
So, let me spend a couple of words about eBPF.
So, eBPF is kind of new.
I think it is two years old technology.
In the Linux kernel, which is gaining more and more traction
in multiple subsystems in the Linux kernel,
for example, in the tracing subsystem.
And it's an extension to the classical BPF,
and it's close to a real CPU architecture,
and in fact, it's also jitted on many different architectures.
But the great things of eBPF is that
it provides safe memory access guarantees and time-bounded execution.
So, when you load an eBPF program,
in your kernel, you know that it will be as safe
as running a program in user space,
because the program cannot access random region in the kernel,
and the program is guaranteed to terminate,
because no backward jumps are allowed.
In fact, every time you load an eBPF program,
the kernel will run a verifier,
and if your program does not meet any of these guarantees,
it is just rejected.
And also, eBPF provides shared maps
with user space, so you can have some kind of state
shared between your XDP program and user space.
And another great thing is that there is an LLVM backend compiler,
so basically you can write your C program
and compile it to eBPF and then load it in the kernel,
which is super handy.
And, okay, so this is not an XDP tutorial,
but I just wanted to show you a quick sample of an XDP program.
In this case, we are...
This is a super simple example.
An XDP program is the function that is called on each packet.
It receives as input a context,
which is just a couple of pointers
to the begin and the end of the packet.
And then we can just access the packet
using the Linux kernel data structures
that are already there.
And so an important thing to notice
is that we are actually making sure
that the program is not reading past the buffers,
the packet buffer boundaries.
Otherwise, it would be not possible,
not even possible to load the kernel, the program,
because the kernel would comply.
And then we can go on and check
that the packet is an AP packet
and access the AP header and make sure, again,
that we are not writing past the buffers
and return an action based on the filtering logic that we use.
Okay, the second thing is maps.
So I talked about maps,
and they are super simple to use.
So you define it.
You define the size of the key of the values
and the size of the map.
And then using BPF helpers,
you can just access a specific key
and set the value,
and so it's just like a normal hash table,
and it's shared with user space.
And the great thing is that we can, of course,
auto-generate XDP programs
because it's just C.
So it's easy to write a script
and actually generate C code for you automatically.
And in this case, I'm generating the same,
the program for the same puff signature
I showed you before with BPF tools.
But instead of, like, outputting cryptic BPF bytecode,
we can output actual C code
and then compile it to eBPF and just load it.
So I will not go through the whole code,
but you can just,
you can access different header,
the IP header,
and the TCP fields that you are interested in.
So if you are curious and you want to try it,
the first thing to keep in mind
is that XDP is a new technology, as I said,
so it requires supporting drivers.
So as initially only the Mellanox card supported it,
and now that at each kernel release,
there are new drivers that get support for XDP.
But starting from Linux 4.12,
there is a new feature called generic XDP,
so even if you don't have one of these nice network cards,
you can just run XDP on your laptop
and just play with it and see how it works.
There are some samples in the kernel directory,
so in slash samples slash BPF,
you will find some XDP programs,
and there are also a couple of libraries
that will help you load,
and manage the XDP programs.
And that's all for me.
So questions?
Yes?
How many filters can you load with XDP?
So you...
Okay.
How many filters can you load with XDP?
XDP uses eBPF.
eBPF has an art code size of 4096 instructions,
so as long as you can stay in that size of the program,
you can chain multiple XDP functions,
you can do whatever you want.
You can even change that limit if you know what you're doing.
But the idea is that you should try to keep the eBPF program short
because it's running in kernel space,
so you don't want to hang the whole kernel while processing packets.
Do you make a distinction for V4 packets and V6 packets?
Are those two separate programs or is it the same program?
This is up to you,
but you can, for example,
start processing the different header
and then check which version of IP you're dealing with
and then call another function
because you can...
it's just C,
so you can chain multiple functions
as you would write in C.
Did you explore mangle and raw before going this whole route?
Did you explore mangle and raw tables and IP tables
before jumping out and doing the direct?
If I did...
Are the mangle and raw chains
that happened before processing...
No.
The two IP tables chain happened after XDP.
Okay.
Okay.
No more questions,
so thank you.