Our next talk epitomizes what this village is all about.
To put it bluntly, packet hacking doesn't happen without packets.
And also, PCAPSR didn't happen.
Chris Arbella and Pete Anderson have spent decades working with
Fortune 500 knockin' socks
to implement advanced packet analysis solutions,
better packet pipelines,
and to get more from those packets.
It is my pleasure to introduce to you
Chris Arbella and Pete Anderson
with their talk, Packet Acquisition, Building the Haystack.
Take it away!
Hi everybody, and welcome to Packet Acquisition, Building the Haystack.
I'm Chris Arbella.
And I'm Pete Anderson.
And we've been designing large-scale enterprise packet acquisition
capture solutions for a long time.
Most of that is mostly Pete, if you can't tell.
So what are we going to talk about today?
So most of you have probably used Wireshark,
you know what a packet capture file is.
But when you look at a big environment,
you might wonder, where does that data come from?
How can I get it from all the different strategic points
in a large environment?
And that's what we're going to talk about today.
How do you instrument your environment for packet capture?
And there's various different topics.
We'll start with local packet capture.
Then we're going to talk about how do you scale it out?
How do you cap-span your environment?
And then what are some of the other considerations
you have to deal with in the enterprise?
And then at the end, we'll talk a little bit about cloud and containers,
because those are some of the newer topics that we're all dealing with
in the packet capture world today.
All right.
So before we dive in to talking about how we're going to build out
packet capture infrastructure, I want to talk a little bit about
what are packets good for?
Why do we want to capture them in the first place?
Why do we care?
Exactly. Why should you care about this?
Why do you want to build one of these out?
So what are they good for?
So some of you may have heard the expression,
packets or it didn't happen.
Basically, they are a ground truth.
You see something in a packet trace, that's what happened.
They're definitive information.
And most attacks, exploits, breaches, etc.,
are going to involve traffic moving over the network,
packets going over the network.
And something I always say is that packets can tell you something about everything.
So I'm not going to read through all of these,
but there's a lot of different use cases.
And I always like to use the analogy of logs.
So a lot of you have probably dealt with log files and log aggregations.
Very similar to packets in that they're good for all different types of things.
They can answer all different types of questions.
And actually, some of the challenges and trade-offs are pretty similar as well.
So there's a bunch of different use cases for packets,
a bunch of different types of packet analysis.
The one that most of you are probably familiar with is offline packet analysis.
So when someone gives you a PCAP or you've gathered a PCAP,
and you use a tool like Wireshark to go and parse through that PCAP file.
But there's also real-time analysis solutions.
Things that do shallow packet inspection,
where you're basically looking at the DCP IP5 tuple,
what ports are talking, what IP addresses are talking.
And honestly, if you're doing that, just use NetFlow.
But then it gets more interesting once you start getting into deep packet inspection,
where you're actually looking at the application payload.
You're looking at L5, L6, L7.
And especially when you start becoming session aware.
So if you've used Snort, Siricata, Zeek,
once you have something that can tell you when a connection has actually been established.
And even getting to the point where you can do full reassembly,
as in the case of doing file extraction from, whether it's HTTP or SMB, picket poisoning.
Kind of the flip to that offline packet analysis is continuous packet capture,
where you're just writing packets to disk as fast as you can.
You have a rolling buffer, and you just keep filling those disks with raw packets.
A good example of this is Google's open source stenographer,
which again is meant for high speed packets to disk.
So a little bit about the basics. Where do we start?
We kind of already mentioned some of these technologies,
but when you're just grabbing packets from a single host, from your local host,
most common you're going to use something like TCP dump, Tshark, InGrep, or Wireshark.
You're actually using a GUI.
There are some other ways of getting packets off of a host,
RPCAPD or remote PCAP, it's a part of PCAP, I believe.
But it actually lets you take hosts from a local interface,
and then direct them to some remote destination.
These are good for one-offs, troubleshooting, forensics, when you have a host isolated.
But this doesn't really scale to an actual enterprise deployment.
Alright, so speaking of scale, how do we scale out?
And the thing we're going to talk about first is identifying critical networks.
So before you start implementing anything, by far the most important step
is to identify what you care about and what you want to capture.
Enterprise networks are very complicated.
If you remember, I used the log analogy before.
You're never going to capture every packet. It's impossible.
It's just like you're never going to write every single log out on every host in your environment,
and then ship all that over somewhere in the store. Very similar.
So we have to target key network segments.
And the questions to think about are, where are your critical assets?
Where is your critical data stored? And how does that get accessed?
And you want to focus on those network paths and capture traffic there.
Because if you have a breach, if you have an attacker, if you have lateral movement,
that's the stuff they're trying to get to, and that's where you really want to focus your attention.
Some common examples of those critical assets, things in our experience,
Active Directory or any authentication service, both internal within your network for your users,
as well as for your customers. So you run the website,
making sure that you actually are monitoring the authentication services for that website.
Get repos, and actually tied to get repos, anything tied to the CICD pipeline.
There was some exports around publicly exposed Jenkins endpoints, as an example.
File shares, databases and data lakes, any sort of PCI network, even mainframes.
Basically, does sensitive data live there? Is it something that is critical to your business making money?
It is probably a critical network.
So after you've picked out what your critical networks are,
we have some prioritization on where we want to get the data from.
How do we actually start gathering data from those networks?
And there's really two common ways of doing it.
The first is taps. The second is spans. A tap is actually purpose-built hardware.
It's going to physically break the link between two different pieces of network infrastructure
and take a passive copy from that individual link.
So again, it's tied to everything. You'll need one tap for every link that you're trying to get packets from.
On the span side, it's a configuration on network infrastructure, typically a switch.
It's switched port analyzer, what span stands for.
You'll also hear mirroring or monitor sessions. Those are all the same phrase.
But it's taking a digital copy from the data plane of the network infrastructure and directing that copy to another location.
So taps, they're less common than spans, and that's mostly because of the cons of taps.
So I mentioned it's actually tied to individual links, which means that for every link you want to monitor,
you actually need another tap. So you end up having to pay for many, many different taps,
which also has an administrative burden, both from the outage required when you break that wire,
as well as just how many individual taps that you have and you have to cable.
On the other hand, because it's tied to the individual link, because it's actually a physical tap into that data feed,
you end up with the best possible data feed. You can actually see on the screen how a fiber tap works,
where you're actually slightly bending a cable and using refraction through that cable to pull out some percentage of the light passing over that fiber.
So additionally, because it's tied to the physical infrastructure, there's no impact on any sort of switches, any sort of routers, anything like that.
Yeah, so taps don't cause packet loss and they preserve packet order and timing perfectly.
So where should you tap? There's typically a couple of different places that we see it.
One is transit links, so north-south boundary into and out of a data center, for example.
You'll also often see it between link layers or links between layers in a traditional three-tier network architecture, whether that's cord distribution or distribution to access.
And in spine-leaf networks, you're actually going to most commonly see it on the link between the spines and the leaves themselves.
And the reason for these different points of tapping, they give you the most bang for your buck.
You'll have the most complete data set for critical network junctures.
So spans. Spans are what you're going to run into most of the time, and they're what you're going to use most of the time.
Taps are actually great, and in a perfect world, you would tap stuff all over the place.
The problem is they're really expensive, and Chris mentioned some of the other cons.
So we're going to use spans. Now, why are spans so great? Because guess what? You've already got them.
So the vast majority of managed switches support at least some type of span port veering technology.
They are, indeed, inexpensive and ubiquitous, but it's extremely flexible, too.
So when you think about a tap, you're tapping a link. With spans, the basic idea is you're spanning a port, but there are a lot of more advanced configurations around VLAN, filtering, and other things along those lines.
And you can target traffic very specifically with a span.
So, sounds like spans are great. Why don't we just always use them? So there are a couple of downsides.
The data can be lower fidelity. You really have to watch out for over-subscribing your destination and having what I sometimes call observation drops, which is essentially a packet loss on the span destination.
Occasionally, you'll see some negative impact on switch performance. It's pretty rare. It can happen. It's less common than it used to be.
Now that switch hardware has gotten more powerful, but every once in a while, you'll have a switch vendor make an implementation change, which I'm sure some of you are familiar with, that can cause issues.
But generally, that's pretty rare. And overall, this is the approach that you're going to use the majority of the time, at least in most environments.
I'd say 98% of the environments that I work in are all based off of spans versus caps.
So let's talk a little bit about the types of spans. So you may hear these different terms. Span, RSpan, ERspan. What do they all mean?
So a span, or a port mirror, that is going to be the most common span type, and that is going to be local to a single switch.
So a simple example. I want to span all the traffic going in and out of port 1, and I'm going to send it all to port 2, and I'm going to put my packet analysis, packet capture solution on port 2.
That's going to be your most common span type. RSpan, you're not going to run into very much, at least in my experience. I've been doing this for almost 20 years. I think I've run into RSpan like twice.
And both times it didn't work that well, so that's probably why I never run into it.
I've never had it.
Exactly. So basically, an RSpan is your source is on one switch, your destination can be on the other, and you use an L2 transport to get there.
We're going to talk about this later, but generally when you have a situation like that, an aggregation solution is much more preferable, and I think that's one of the big reasons we don't see RSpan.
ERspan is a little bit more common. It's somewhat similar to an RSpan, except that instead of a Layer 2 transport, it uses a Layer 3 transport.
So you can span stuff on one switch and send it to another, and it does that by encapsulating the spanned packets in a Layer 3 GRE tunnel.
In a traditional switch network, it's somewhat uncommon, but when you get into the VMware world, it is more common because the VMware distributed switch supports an ERspan, so you can essentially span virtual ports on that virtual distributed switch and then send it somewhere else.
It's also fairly common in Spineleaf networks, although that is a little bit different because although it's technically an ERspan, it doesn't leave the... you're not always going over the data network, it's usually using that underlay network that a lot of the Spineleaf implementations use, so it doesn't leave it that way.
The big caveat with ERspan is that it can put a lot of traffic on your network, right? If you're spanning everything in and out of a port and then you're encapsulating it and sending it over your IP network, you're literally doubling whatever that traffic is, now it's on your network twice.
The other thing you have to keep in mind is there is some encapsulation overhead, so you've got to make sure jumbo frames are supported everywhere, because otherwise packets are going to be too big.
I'll say I've seen, especially in VMware environments, sometimes you'll actually have dedicated monitoring interfaces, physical interfaces on the physical hosts that are dedicated for ERspan. That way you're avoiding putting monitor traffic onto the data networks.
Actually, Chris brings up a great point there. If I was designing an environment from scratch, I would absolutely make it a requirement to have a second 10 gig NIC in every single virtual host and have that connect to an out-of-band management network.
Then you could use ERspan to your heart's content without impacting your data network at all.
So this is a quick example, or a couple of examples, of how you might configure a SPAN session. And you can see a link to a documentation down below. This is Cisco specific. Most vendors are fairly similar though.
I will tell you, I've been doing this a long time, there are always slight little differences in the different software versions.
So it's always a good idea to Google, hey, I've got this particular switch with this version of iOS. If we're talking Cisco, obviously, if you have another vendor, do whatever the similar search would be and make sure you have the right syntax.
Cisco likes, at least, to change their syntax quite a bit from version to version.
It's funny, as I said, I've been doing this almost 20 years, and it's always looked pretty much like this, but the way you arrange your VLANs has changed over the years and things of that nature.
By the way, so one of the things we're going to be talking about here is you'll notice that in one case we have an example of spanning a port, in another one we have an example of spanning a VLAN, and then you'll notice how you can send the traffic to a destination.
So a lot of times, by the way, you're not just limited to one destination, you can do multiple destinations, and I'll talk about this a little more, but virtual interfaces are often supported.
You'll notice there's some directionality there as well, which is something else we'll talk about.
Alright, so, as I said, we were going to talk about some of the spam sources. So, at its most basic, right, a spam is going to be just spanning a port or multiple ports.
And when you think of an interface, right, you might think, hey, yes, it's just an Ethernet interface on a Switch.
But a lot of Switch vendors also support things like virtual interfaces and aggregate interfaces, so in the Cisco world, like a port channel, you can spam a port channel, essentially.
And so what's the use case of spamming a particular port? It's usually in a large environment to target a particular network path or link.
Now, if you have a very small environment where you have, you know, like one Switch, like I have one Cisco Switch in my home environment, right, so I just spam every port on the Switch.
That is going to be less common in an enterprise environment, but if you do have a small environment, it's a reasonable approach to just spam all the ports.
However, when you get to the enterprise, it's much more common to use a VLAN spam, right, because most modern networks, things are segmented by VLAN.
So your source is going to be a VLAN or multiple VLANs, right? Typically, when I do this, it's multiple VLANs.
I'll go in and we'll say, hey, where are all the critical servers? Let's spam all those VLANs, and that's a lot of times where we start.
How often do people actually know what lives on which VLANs?
Well, that varies, of course.
It's usually, in my case, a discovery where you'll start with a wide caster net wide and capture many, many VLANs and then prune off as you validate what traffic lives where.
And I've actually done the opposite approach to, you know, we'll start and we'll say, OK, we're going to start with these VLANs and then inevitably you'll go look for traffic and it's not there.
And then, you know, you add some more. So Chris brings up a good point, right? I mean, there's always going to be some tweaking.
It's not like, hey, you have to have everything set in stone on day one.
You'll want to take your best shot at it, what you think you need, and then refine your data from there.
So a couple more notes on this. With port spans, by the way, typically you're going to span in both directions, so ingress and egress.
With VLANs, you only need to span ingress. And in fact, some newer switches only support ingress.
So you won't even have the choice anyway. So you don't even have to worry about it.
And then the last thing I'll say, a couple of last things on VLANs.
So they're a really good way to target certain assets, which is the reason they're the most common method.
And then when we get into Spineleaf, there are some similar concepts in there.
They just have different names because, of course, we want to confuse everyone.
So what are some common span issues?
So number one is probably oversubscribed monitor reports.
And I'll give you the simple explanation of this. Believe it or not, you can have a single...
if you span a single 1GIG interface in both directions, you can completely overrun a destination 1GIG interface.
And you say, Pete, how is that possible?
Well, think of it this way, an interface sends traffic in both directions, right?
So you're not... a 1GIG interface can actually have a total of 2 gigabits per second of traffic, right?
If you combine the ingress and the egress.
However, your destination interface that's sending the traffic to your analysis solution, that can only send at 1 gigabit per second, right?
Now, in reality, is spanning a 1GIG port going to overrun another 1GIG port?
No, almost never, right?
So it's very rare for switch ports to consistently run maxed out.
But it is something to be aware of, right?
And this is where we're going to get into, you know, having multiple destinations and aggregation.
In an enterprise environment, your destination port should almost always be at least 10 gigs.
Because you're going to... just trust me, you'll run into way less problems.
Trying to span out 1GIG interfaces in most modern networks, unless you're in a really small environment, is... you know, you're looking for trouble there.
I mentioned this on the previous slide, but when you have an interface span, typically, you're going to want to do both directions.
If you don't, you're going to get unidirectional traffic.
And you'll be really confused because your packet trace won't make any sense.
And we do have a note here.
If you go back to my simple, like, super small network example, if you're spanning all the interfaces on a switch,
in that case, you could do... you could just do receive.
But most of the time, you're going to do both.
And then I mentioned this before, but another one is ER-span without JumboFrame support.
So I know some implementations of ER-span actually mark the packets do not fragment.
So that means if you try to send a full-size packet that's been encapsulated over a network without JumboFrame support,
well, it's not going to go anywhere. It can't be fragmented.
So where do we span?
So in a three-tier architecture, when I say three-tier architecture, this is your traditional core distribution access.
Those are typically going to be your three options.
So we'll start with the core.
I sometimes work with people that span at the core.
Depends on the network.
I'll get to where we usually span.
But when you look at the core, your north-south traffic is going to be there a lot of times.
So you can get all your north-south traffic.
But your east-west traffic can be pretty limited.
Sometimes some networks will really only get routed traffic and things like that.
The advantage of spanning at the core is it does not require a lot of cabling or port density.
Some networks use more of a collapsed core where a lot of traffic goes to the core.
In those cases, it can be good to span at the core.
But generally, I don't recommend only spanning at the core.
You're just going to miss too much east-west traffic.
Usually, the keyword I look for is if a network architect describes their network as flat,
then you probably can span at the core and get a decent amount of the traffic.
But again, as Pete mentioned, you're still going to be missing a significant amount.
So distribution. This, as we say, is often the sweet spot.
Typically, you're still going to get... so we mentioned north-south on the last slide.
A lot of times you get north-south anyway because you move down the stack.
So you may have to do a little supplementing of certain transit networks,
or maybe you grab lots of DMV traffic that's up closer to the core,
but most of you are north-south.
It's going to stop all the way at the bottom of this diagram anyways.
So you're going to get it regardless of the layer you span at.
And what's the advantage of the distribution over the core?
More east-west traffic, period.
You're still not going to get all of it because you won't get traffic that never leaves an access switch.
So if you have two servers that talk to each other on the same access switch,
it's not going to come up in the distribution layer.
But you'll get a lot of stuff, and it doesn't have incredibly onerous tabling requirements.
So even in large networks, a lot of times we're only talking 8, 12, 16 distribution switches.
So it's not like you're going to be running dozens and dozens and dozens of tables,
or requiring massive aggregation switches.
Finally, the access layer.
Now, like I mentioned in a perfect world with PaaS, this is another perfect world situation.
If I could just wave a wand, I would just spam it the access layer.
Because you're going to get the most traffic.
Because you'll get almost all your east-west traffic there.
Even hosts on the same switch.
The problem is it can require extensive tabling.
I mean, we have customers with dozens, hundreds of access switches.
And that's a lot of labor and work to cable all that up.
You say getting crazy?
I do know some organizations who spam all their TopperRack switches though.
And they get great coverage.
So you have to look at your environment, right?
It depends how sprawling it is, how many access switches you have.
Sometimes what it might make sense to do is not do all the access switches,
but if you have a certain critical one where there's certain access for it, then you can look at those.
I think that just gets back to inter-packet acquisition strategy design with a plan.
Know what you want to get at.
And even if you don't necessarily know where it lives, if you know what you want,
you can figure it out over time as you do some discovery.
All right, so we talked about 3Gear.
Now we're going to talk a little bit about SpineLink.
This example is going to be very ACI-specific.
Just because that's what I'm most familiar with.
But other vendors are pretty similar.
The most confusing thing about SpineLink is they decided to use all different terminology to talk about everything.
But in some ways, when you start digging into it, it actually can be a little easier than traditional three-tier networks.
Number one, there's not three tiers anymore. There's only two, right?
There's Spine switches and Leaf switches.
And specifically in ACI networks, there's three span types.
There's an access span. That is very similar to a traditional span.
So an access span is on the Leaf switch.
And Leaf switches essentially have two different types of ports.
They have access ports, which is traditional switch ports. You hang a server off it.
And they have uplink ports, or fabric ports.
And that's what talks to the Spine switches.
So an access span would be spanning only access ports.
And for those, your destination can be local on that switch, or it can be a remote using ER spans.
Now, as I mentioned before, while this is technically ER span, it's not exactly the same as, you know, just a, you know, ER span in a vacuum.
Because you're doing ER span, but you're going somewhere else on the fabric.
And the fabric already has its own transport IP network as part of it.
So it's some of the down... you don't have to worry about as many of the downsides of ER span when you're dealing with Spine switches.
That's the high level summary.
It's much less likely you're going to, you know, stomp on your other traffic.
And, you know, in modern timing networks, they're all going to support jumbo-brings.
So the second type of span is a fabric span.
That's going to be spanning the uplink port between the Leaf and the Spine.
One thing to note here is when you go to look at the packets, if you wanted to, like, pull them into Wireshark, they're going to be DX LAN encapsulated.
So you're going to have to use... you're going to have to decapsulate them before you analyze them.
And actually, if you go beyond Wireshark, it's all... and you're looking at this kind of traffic, your aggregation solution or your analysis solution would also need to be able to decapsulate that traffic.
This one always uses ER span.
And then the final one, which is the one I at least personally have used the most and dealt with the most, is what they call a tenant span.
And that spans one or more endpoint groups.
And it's sort of beyond the scope of this presentation to get in to what an endpoint group is.
It's somewhat similar to a VLAN, though.
And that's why it ends up being a pretty popular approach, right?
Because it's a good way to target assets.
And kind of nice in Spineleaf, it doesn't really matter.
You can span... this endpoint group can be on all different heap switches.
And because it all uses ER span, you can pull it all back to one spot and send it over to your analysis solution or your aggregation network.
So once we get beyond just three-tier architectures, Spineleaf architectures, we actually end up having to have some additional challenges in enterprise deployments.
And one of those is going to be virtual environments.
The challenge in a virtual environment is that not all of the network traffic is going to hit a physical network.
So the image on the screen, the three virtual hosts that are on host one, if they're talking to each other, that traffic is going to stay on host one, even though they're all hooked into the same virtual distributed switch.
However, we always recommend do your physical spanning and tapping first.
This is a problem that gets blown out of proportion. You often get 80 plus percent of the traffic that you're looking for.
And it's funny we use 80% here because I would say, not only is it like 80% of the traffic you're looking for, I'd say about 80% of the time when people are worried about this, it ends up not being a problem.
Pete mentioned already ER span, so virtual distributed switches support ER span for critical workloads. You can target individual interfaces and get that traffic off of the hypervisor host.
There's also virtual taps, which are installed on a host by host or VM by VM basis.
You start running into the management overhead of installing those virtual taps, but for hypercritical pieces of infrastructure and pieces of applications, this can be a very viable solution.
Another less obvious solution is to separate key VMs and make sure that they're on different hosts, forcing their communication to actually traverse a physical network that you already have monitoring in place for.
And that last solution, by the way, is it sounds kind of funny. Hey, yeah, we'll just put them on separate hosts. But turns out that that easiest choose part is pretty important, right? Because some people are like, wait, that's all I have to do. And I'm like, yep, that's all you have to do. And they're like, oh, well, that's easy.
So, you know, don't, you know, I think we might have mentioned earlier, don't let the perfect be the enemy of the good. And that's a good example of that.
Alright, so let's talk a little bit about span aggregation, right? So what happens when you get these large environments is you don't have to just tap and span all over the place, you got to find a way to bring all that traffic back somewhere. So you can feed it one or more analysis and capture solutions.
Right? So one thing to note about most of these analysis and capture solutions is they are not designed for port density. So a lot of them use expensive high end network cards, you know, that can cost, you know, over $10,000 each. So they're not going to have 16 of them sitting, you know, in the solution, right?
So we need to, we need some way, you know, if I'm spanning 8 or 16 different switches, how do I get that fed into my analysis solution that has, say, you know, 2 or 4 or 10 gig ports on it? So what's the solution? An aggregation switch. And if you're not familiar with aggregation switches, there are some big vendors that do it. So if you've heard of Gigamon or AppCon or HCM, they do it.
Some switch vendors like Arista, you can also just configure their switches to be an aggregation switch. And really, when we say it's an aggregation switch, I mean, there are some that are purpose built, but it is a switch. Like the hardware, that's what it is. It's just a switch, except instead of sending traffic over it, you're going to use it to aggregate all your span and tap feeds, right?
So essentially, what it does is, let's say I have that example where I have 8 switches span, I need to get it into 2 ports, it's just going to multiplex all that traffic from those 8 ingest ports and send it out, 2 of them, to my analysis solution, right? And in fact, most of them can feed multiple analysis solutions. So maybe you have high speed packet capture, you've got an IDS, right? You've got some, you know, network detection and response solution, you want to feed all those.
So nice thing is, you know, you just have one set of spans and taps, feed it into your aggregation switch, and then it can feed multiple solutions that are connected to it.
And a couple things to add as well. So if you are thinking about taps at any sort of enterprise scale, you have to be thinking about span aggregation as well. Like, it is not an option to not consider that, not factor that into your deployment.
Yeah, one thing we didn't mention about taps that maybe we should have, if you have a single 10 gig fiber tap, you might say, okay, I'm going to tap one link. So how many links do you actually have that feed your analysis solution?
It's not one, it's actually two, because the way taps work is they feed you the ingress and the egress on that link, and they separate it out into two different links.
And they do that because, as I mentioned before, it's a full duplex link, and taps want to do perfect preservation. They give you two outputs so that there is no packet loss at all possible.
So Chris is exactly right. I mean, when you're talking taps, you can be talking a huge amount of links if you do them at any scale at all.
One additional thing, we've mentioned ER-SPAN a couple of times. We've also mentioned VXLAN in the case of a Spineleaf deployment.
SPAN aggregation solutions are capable a lot of times of doing decapsulation to get all of your traffic into whatever the base level traffic pattern that you want to analyze.
So you want just the raw traffic that a host would see, for example. Great, fantastic. A lot of SPAN aggregation solutions can get it to that specific...
And that's really important because if you feed, let's say Spineleaf encapsulated VXLAN traffic to an analysis solution and it's not decapsulated and that solution can't decapsulate it, it's not going to do anything for you.
You're going to be like, what are these IP addresses? Because they're all going to be the underlay network and it's not going to be your actual host IP addresses.
So some other things aggregation switches do, filtering traffic, they do deduplication, which we're going to talk about as well.
So let's talk a little bit about filtering. So this is probably one of the most important things when you're building these networks out.
So number one, packet analysis resources are not infinite, right? Nothing can accept infinite bandwidth. And writing packets to disks is also not infinite, right? You don't have an infinite disk space.
In fact, if you have, let's say, 100 gigabits per second link, that will fill up one petabyte of disk per day approximately.
And it's funny, when I meet with customers, they'll tell me, hey, I want 30 days of lookback. And I go, okay, what's your average throughput?
And they might say something, I don't know, five gigabits, 10 gigabits, whatever it is, you can literally go on Google and type like 10 gigabits per second, if you type the words out, to terabytes per day.
And then you can shock them and let them know like, oh, yeah, actually, you're going to need like 10 petabytes of disk if you want to keep that much data.
And then of course, they're like, well, maybe we don't need to keep that.
Some people can't dedicate whole data centers to storing PCAP. So it's just how much money you want to throw at the problem.
Yeah, I mean, there's government agencies, right, that just have stacks and stacks and stacks of storage to do that.
But, you know, even if some of the, you know, we've worked with some of the largest companies in the world, and, you know, they don't have infinite money, like, you know, any more than anyone else.
So one of the things that can really help you, though, is filtering. Because there is a, you know, to use a non-technical term, if you're spending a bunch of traffic, there's a good chance you're going to get some junk in there that you don't really care about.
You know, things like backup traffic, data replication traffic, that stuff, it's just not typically going to be very useful to look at.
And it can consume a massive amount of bandwidth and disk space. Like, I have customers where I'd say 30-40% of their traffic in a 24-hour period is backup traffic, if I were to spend, you know, most of the traffic in their data center.
So we're going to, if we get that in, we're going to filter it out. Some other good examples are going to be your VM infrastructure traffic.
So if you use NFS, for example, to mount your disk that all your VMs use, you don't want to, you typically do not want to spend that. It's going to be a massive amount of traffic. Or if you use iSCSI, same thing.
vMotion traffic is another one. It's, you know, it's a ton of traffic, and it's just generally not going to be that interesting.
And then finally, telemetry data can also consume a ton of bandwidth. So syslog, ELF, Elastic, NetFlow, etc.
Splunk replication, one of the chattier things that I've seen on the network. There's certain network links where I've seen Splunk replication in particular consuming 30 to 40% of the total available bandwidth.
So, filter it out. Get it out. I don't care.
Alright, now, what's the other big thing besides filtering? Duplicate traffic, right? And that's, how do we handle duplicate traffic? Well, there's a couple ways. One way is deduplication.
So, when you do this, if you haven't, you know, done a lot of scanning and tapping before, you're going to get duplicate traffic. It's normal. Don't freak out. Expect to see some of it, right?
The problem is when you get a lot of it, right? There's negative impacts of having tons and tons of duplicate traffic. Number one, you know, it's like, okay, if I have a duplicate of every single packet on my network, and I don't have any way to deduplicate it, it looks like I have twice as much traffic. It's going to waste a ton of lookback space.
Let's say I bought some packet analysis solution that can handle up to 10 gig of traffic. Well, if I'm feeding it two copies of every packet, I'm literally wasting half of its capability, right? And it can also make manual analysis more difficult.
You know, if any of you have ever opened up a Wireshark trace, and there's like six copies of every packet, it can be annoying to look at. I mean, there's ways to filter it out and stuff, but it can just make it more of a pain.
So a couple of solutions here. So one is going to be deduplication, which is a feature essentially that's supported by some aggregation switches and some packet analysis solutions.
And my general rule of thumb is that it works pretty well for lower levels of duplicate traffic, right? As I said, it's going to be normal to see some amount of duplicate traffic, and deduplication can handle that pretty well.
However, when you start getting a lot of duplicate traffic, sometimes these deduplication solutions just fall over. And it's not because they're poorly designed or bad or any of that. It's because it's a hard problem to solve that requires a lot of resources.
If I'm feeding 10 gigabits of traffic, essentially the way they work is they have a buffer, they have to buffer the traffic, they have a window that they look at, where they look, you know, if they see there's various heuristics you can use to identify duplicate packets.
But within that window, they try to identify them, right? But again, when you're talking 10 gig, 20 gig, 40 gig of traffic, it just requires a ton of memory and resources to buffer that and look any more than a few milliseconds for duplicates, right?
So what do you do when you have tons of duplicate traffic? Well, usually the reason is, is because you're spanning the same traffic or tapping the same traffic in a bunch of different places. And we're going to go back to the last slide, and we're going to filter that.
So, in my experience, once you get above, you know, 5, 10, maybe 15% mean when you're up at like 50% duplicate and higher, you need to go filter. And you can filter at your where you're actually doing the scan, you can filter at the aggregation switch. And one of the things that I've found that works pretty well a lot of times is identifying, you know, let's say I have, you know, as an example, let's say I have a firewall that's passive and I span both sides.
Well, I'm going to get the same package twice, right? Every time I want to filter that traffic out, essentially, all I have to do is figure out which MAC address is on which side of the firewall, right? And just filter the same traffic on both sides, you know, other than whatever gets blocked. And then I can use like a MAC address filter like that to pretty easily get rid of that traffic.
So just as a general rule, I find that, you know, facing your filters, which by the way, most aggregation switches can do, facing your filters on MAC addresses is a pretty good approach.
As much as I wish Amazon or Google would let me into their data centers and deploy taps and spans, I don't think they're so keen on doing that. And so what we've actually seen is the top three cloud vendors, AWS, GCP, and Azure have all announced, and in the case of AWS and GCP, made generally available span as a service.
Which I think I made up. But it is the same thing as a span on prem, where you say, I have a specific set of cloud workloads, whether that's a specific instance, whether that's an ENI, whether that is an entire VPC, and you say, feed that into a specific monitoring interface, feed it into whatever analysis tool that I want to grab those packets from.
In the case of Azure, they actually announced in 2018, it is still in private preview, just as a heads up. So for all the Azure customers out there, push on them. I know we are, because AWS and GCP can do it today.
Yeah, so one thing I'll add here, that's pretty nice is, believe it or not, even though it's in the cloud, this stuff is actually quite a bit easier sometimes than on prem spans, because you literally like in AWS, you literally just say, okay, I have these, you know, 520 servers, I want to span or these 300 servers I want the traffic from, and you just can just turn on the traffic mirroring for them.
There's no figuring out which VLAN they're on, or any of the things you have to deal with in an on prem data center, right? It's just they handle the implementation details under the covers, you just say, okay, I'm going to pay some money and get my traffic.
It's all API based, right? So you go get BOTO, or pick your poison, and you have give me all of these tagged instances, give me their data, you don't have to coordinate with network teams, you just get the data as soon as you want.
Containers, you're running Kubernetes, you're running Docker, they have this similar challenge to VMs in that there is traffic that never hits a physical network, or it's encapsulated in such a way where when it hits the physical network, it's hard to tell what the actual data is.
So first thing, because it's similar challenge to VMs, you actually have similar solution, tap a physical network and validate what you are and are not seeing. Again, in our experience, you actually see a decent amount of the critical traffic, especially the inter service traffic.
What you won't necessarily see is inter pod traffic. But we as an industry are still trying to decide on what the best way to actually capture this traffic is.
You can deploy a daemon set, which is a specific pod that you run on every single node within your K8s cluster. And so you can hook into the node networking and use that as a source of data.
You can also run a sidecar, which is a container that is run in the pod and use that to get inter pod intra node traffic as well. And some of the technologies that we're considering using the namespace hooking into the Docker zero interface and actually tapping those internal localhost interfaces as a way of getting to some of that data.
So with that, some of our takeaways, both Pete and I have been working with packets for a long time. And I think they're one of the arguably best sources of data, not just because of the richness of the data that's there, but because of how accessible they are.
You're not having to go find every host that you care about on the network, and instead can target the network as a whole.
I don't know about you, Pete, but it seems like every network is kind of the same, but they're all unique in their own fun and interesting ways.
But that doesn't mean that you can't use general principles as you start designing these architectures, and then modify them, use different strategies for different problems.
And that's, there's many different strategies, depending on the network.
And I would say these last two points are probably the most important. That identify critical assets step is by far the most important thing that you can do.
We can set aside all the technology implementation, all of that. If you don't, you're identifying critical assets, that's essentially your problem statement.
And if you don't have a good problem statement, it's going to be hard to solve that problem.
And I also think that don't let the perfect be the enemy of good is a really important point, too. And I'm going to go back to that log analogy.
Again, this is very similar. No one is going to capture every single log that is ever written by every single hosting device in a large environment.
It's just not going to happen. And this is the same thing here. You are never going to capture every single packet.
That doesn't mean that this can't be incredibly valuable.
And what I've found is getting, let's say, 90% of what you want, it's not too hard.
Then it's twice as hard to get the next 5%, to go to 95%.
Then if you go to 97.5%, it's twice as hard as it was to get to 95%.
And it's diminishing returns.
So again, you want to identify those critical assets and don't let the perfect be the enemy of the good.
As long as you have most of the stuff you need, most of the time,
I think you can provide some incredibly valuable visibility and investigative capabilities to your organization.
Perfect. So with that, thank you for your time.
Thanks for coming to DEF CON Safe Mode.
We're thankful to get the chance to speak to you guys.
So we'll leave it open to any questions.