start	end	text
940	5820	Our next talk epitomizes what this village is all about.
6000	10120	To put it bluntly, packet hacking doesn't happen without packets.
10120	12740	And also, PCAPSR didn't happen.
12820	16160	Chris Arbella and Pete Anderson have spent decades working with
16160	18000	Fortune 500 knockin' socks
18000	23270	to implement advanced packet analysis solutions,
23540	25480	better packet pipelines,
25480	27680	and to get more from those packets.
27680	30000	It is my pleasure to introduce to you
30000	31520	Chris Arbella and Pete Anderson
31520	36560	with their talk, Packet Acquisition, Building the Haystack.
38240	40140	Take it away!
40340	44720	Hi everybody, and welcome to Packet Acquisition, Building the Haystack.
44720	46680	I'm Chris Arbella.
47200	48600	And I'm Pete Anderson.
48600	53580	And we've been designing large-scale enterprise packet acquisition
53580	56160	capture solutions for a long time.
56160	60920	Most of that is mostly Pete, if you can't tell.
61240	63200	So what are we going to talk about today?
63220	67560	So most of you have probably used Wireshark,
67560	69980	you know what a packet capture file is.
70080	72120	But when you look at a big environment,
72120	74960	you might wonder, where does that data come from?
74960	78260	How can I get it from all the different strategic points
78260	79400	in a large environment?
79400	80760	And that's what we're going to talk about today.
80760	83880	How do you instrument your environment for packet capture?
84020	85780	And there's various different topics.
85780	87280	We'll start with local packet capture.
87280	89540	Then we're going to talk about how do you scale it out?
89540	91540	How do you cap-span your environment?
91920	94700	And then what are some of the other considerations
94700	96380	you have to deal with in the enterprise?
96380	99540	And then at the end, we'll talk a little bit about cloud and containers,
99540	102220	because those are some of the newer topics that we're all dealing with
102680	104780	in the packet capture world today.
107480	108340	All right.
108340	115300	So before we dive in to talking about how we're going to build out
115300	119280	packet capture infrastructure, I want to talk a little bit about
119280	120720	what are packets good for?
120720	122640	Why do we want to capture them in the first place?
122660	123840	Why do we care?
124580	126440	Exactly. Why should you care about this?
126440	128120	Why do you want to build one of these out?
128140	129340	So what are they good for?
130360	132520	So some of you may have heard the expression,
132520	133860	packets or it didn't happen.
133920	136360	Basically, they are a ground truth.
136700	140220	You see something in a packet trace, that's what happened.
140320	142380	They're definitive information.
142380	145880	And most attacks, exploits, breaches, etc.,
145880	148780	are going to involve traffic moving over the network,
148780	149980	packets going over the network.
149980	154000	And something I always say is that packets can tell you something about everything.
157630	159950	So I'm not going to read through all of these,
159950	162010	but there's a lot of different use cases.
162010	165570	And I always like to use the analogy of logs.
165830	170330	So a lot of you have probably dealt with log files and log aggregations.
170330	176030	Very similar to packets in that they're good for all different types of things.
176030	178790	They can answer all different types of questions.
179210	184050	And actually, some of the challenges and trade-offs are pretty similar as well.
185890	189570	So there's a bunch of different use cases for packets,
189650	191790	a bunch of different types of packet analysis.
191790	196910	The one that most of you are probably familiar with is offline packet analysis.
196910	199570	So when someone gives you a PCAP or you've gathered a PCAP,
199570	205290	and you use a tool like Wireshark to go and parse through that PCAP file.
205570	209390	But there's also real-time analysis solutions.
209430	212130	Things that do shallow packet inspection,
212130	215490	where you're basically looking at the DCP IP5 tuple,
215490	218530	what ports are talking, what IP addresses are talking.
218690	222570	And honestly, if you're doing that, just use NetFlow.
222570	226930	But then it gets more interesting once you start getting into deep packet inspection,
226930	229870	where you're actually looking at the application payload.
229870	232630	You're looking at L5, L6, L7.
233350	237070	And especially when you start becoming session aware.
237110	240330	So if you've used Snort, Siricata, Zeek,
240330	244870	once you have something that can tell you when a connection has actually been established.
245030	248710	And even getting to the point where you can do full reassembly,
248710	255550	as in the case of doing file extraction from, whether it's HTTP or SMB, picket poisoning.
255970	261170	Kind of the flip to that offline packet analysis is continuous packet capture,
261170	264290	where you're just writing packets to disk as fast as you can.
264290	270150	You have a rolling buffer, and you just keep filling those disks with raw packets.
270150	275050	A good example of this is Google's open source stenographer,
275050	278570	which again is meant for high speed packets to disk.
281070	284830	So a little bit about the basics. Where do we start?
284890	287950	We kind of already mentioned some of these technologies,
287950	292590	but when you're just grabbing packets from a single host, from your local host,
292590	300030	most common you're going to use something like TCP dump, Tshark, InGrep, or Wireshark.
300030	301550	You're actually using a GUI.
301550	306610	There are some other ways of getting packets off of a host,
306610	312130	RPCAPD or remote PCAP, it's a part of PCAP, I believe.
312130	316570	But it actually lets you take hosts from a local interface,
316570	321030	and then direct them to some remote destination.
321610	327490	These are good for one-offs, troubleshooting, forensics, when you have a host isolated.
327490	332830	But this doesn't really scale to an actual enterprise deployment.
336770	341950	Alright, so speaking of scale, how do we scale out?
341950	347650	And the thing we're going to talk about first is identifying critical networks.
347650	353730	So before you start implementing anything, by far the most important step
353730	357830	is to identify what you care about and what you want to capture.
357830	360730	Enterprise networks are very complicated.
361310	363970	If you remember, I used the log analogy before.
363970	366670	You're never going to capture every packet. It's impossible.
366670	371570	It's just like you're never going to write every single log out on every host in your environment,
371570	375770	and then ship all that over somewhere in the store. Very similar.
375770	378750	So we have to target key network segments.
378750	382070	And the questions to think about are, where are your critical assets?
382630	386490	Where is your critical data stored? And how does that get accessed?
387070	393630	And you want to focus on those network paths and capture traffic there.
393630	398030	Because if you have a breach, if you have an attacker, if you have lateral movement,
398030	402310	that's the stuff they're trying to get to, and that's where you really want to focus your attention.
404790	411590	Some common examples of those critical assets, things in our experience,
411590	418470	Active Directory or any authentication service, both internal within your network for your users,
418470	422270	as well as for your customers. So you run the website,
422270	426950	making sure that you actually are monitoring the authentication services for that website.
426950	433510	Get repos, and actually tied to get repos, anything tied to the CICD pipeline.
433650	440590	There was some exports around publicly exposed Jenkins endpoints, as an example.
440590	447610	File shares, databases and data lakes, any sort of PCI network, even mainframes.
449450	455430	Basically, does sensitive data live there? Is it something that is critical to your business making money?
455610	460910	It is probably a critical network.
460910	464370	So after you've picked out what your critical networks are,
464670	468890	we have some prioritization on where we want to get the data from.
468890	473370	How do we actually start gathering data from those networks?
473370	476830	And there's really two common ways of doing it.
476830	482870	The first is taps. The second is spans. A tap is actually purpose-built hardware.
482870	489430	It's going to physically break the link between two different pieces of network infrastructure
489430	494350	and take a passive copy from that individual link.
494350	501510	So again, it's tied to everything. You'll need one tap for every link that you're trying to get packets from.
501810	507010	On the span side, it's a configuration on network infrastructure, typically a switch.
507010	510130	It's switched port analyzer, what span stands for.
510130	515730	You'll also hear mirroring or monitor sessions. Those are all the same phrase.
516470	526690	But it's taking a digital copy from the data plane of the network infrastructure and directing that copy to another location.
528050	535070	So taps, they're less common than spans, and that's mostly because of the cons of taps.
535070	543110	So I mentioned it's actually tied to individual links, which means that for every link you want to monitor,
543110	548010	you actually need another tap. So you end up having to pay for many, many different taps,
548010	555270	which also has an administrative burden, both from the outage required when you break that wire,
556050	561210	as well as just how many individual taps that you have and you have to cable.
561470	570190	On the other hand, because it's tied to the individual link, because it's actually a physical tap into that data feed,
570190	578370	you end up with the best possible data feed. You can actually see on the screen how a fiber tap works,
578370	587730	where you're actually slightly bending a cable and using refraction through that cable to pull out some percentage of the light passing over that fiber.
589670	598970	So additionally, because it's tied to the physical infrastructure, there's no impact on any sort of switches, any sort of routers, anything like that.
598970	605710	Yeah, so taps don't cause packet loss and they preserve packet order and timing perfectly.
608490	614690	So where should you tap? There's typically a couple of different places that we see it.
614690	621130	One is transit links, so north-south boundary into and out of a data center, for example.
621130	633350	You'll also often see it between link layers or links between layers in a traditional three-tier network architecture, whether that's cord distribution or distribution to access.
633890	642970	And in spine-leaf networks, you're actually going to most commonly see it on the link between the spines and the leaves themselves.
642970	649510	And the reason for these different points of tapping, they give you the most bang for your buck.
649510	654990	You'll have the most complete data set for critical network junctures.
659420	665820	So spans. Spans are what you're going to run into most of the time, and they're what you're going to use most of the time.
665820	669220	Taps are actually great, and in a perfect world, you would tap stuff all over the place.
669220	674860	The problem is they're really expensive, and Chris mentioned some of the other cons.
675100	680540	So we're going to use spans. Now, why are spans so great? Because guess what? You've already got them.
680540	688100	So the vast majority of managed switches support at least some type of span port veering technology.
688520	693420	They are, indeed, inexpensive and ubiquitous, but it's extremely flexible, too.
693420	706560	So when you think about a tap, you're tapping a link. With spans, the basic idea is you're spanning a port, but there are a lot of more advanced configurations around VLAN, filtering, and other things along those lines.
706560	710360	And you can target traffic very specifically with a span.
710820	716200	So, sounds like spans are great. Why don't we just always use them? So there are a couple of downsides.
716780	730700	The data can be lower fidelity. You really have to watch out for over-subscribing your destination and having what I sometimes call observation drops, which is essentially a packet loss on the span destination.
732100	740760	Occasionally, you'll see some negative impact on switch performance. It's pretty rare. It can happen. It's less common than it used to be.
741100	751960	Now that switch hardware has gotten more powerful, but every once in a while, you'll have a switch vendor make an implementation change, which I'm sure some of you are familiar with, that can cause issues.
751960	761300	But generally, that's pretty rare. And overall, this is the approach that you're going to use the majority of the time, at least in most environments.
761300	769320	I'd say 98% of the environments that I work in are all based off of spans versus caps.
773640	781220	So let's talk a little bit about the types of spans. So you may hear these different terms. Span, RSpan, ERspan. What do they all mean?
781360	787460	So a span, or a port mirror, that is going to be the most common span type, and that is going to be local to a single switch.
787460	799640	So a simple example. I want to span all the traffic going in and out of port 1, and I'm going to send it all to port 2, and I'm going to put my packet analysis, packet capture solution on port 2.
801000	811000	That's going to be your most common span type. RSpan, you're not going to run into very much, at least in my experience. I've been doing this for almost 20 years. I think I've run into RSpan like twice.
811200	816640	And both times it didn't work that well, so that's probably why I never run into it.
816640	818220	I've never had it.
820180	829780	Exactly. So basically, an RSpan is your source is on one switch, your destination can be on the other, and you use an L2 transport to get there.
830980	839760	We're going to talk about this later, but generally when you have a situation like that, an aggregation solution is much more preferable, and I think that's one of the big reasons we don't see RSpan.
841120	850380	ERspan is a little bit more common. It's somewhat similar to an RSpan, except that instead of a Layer 2 transport, it uses a Layer 3 transport.
850380	860360	So you can span stuff on one switch and send it to another, and it does that by encapsulating the spanned packets in a Layer 3 GRE tunnel.
860360	881420	In a traditional switch network, it's somewhat uncommon, but when you get into the VMware world, it is more common because the VMware distributed switch supports an ERspan, so you can essentially span virtual ports on that virtual distributed switch and then send it somewhere else.
881420	905180	It's also fairly common in Spineleaf networks, although that is a little bit different because although it's technically an ERspan, it doesn't leave the... you're not always going over the data network, it's usually using that underlay network that a lot of the Spineleaf implementations use, so it doesn't leave it that way.
906100	920700	The big caveat with ERspan is that it can put a lot of traffic on your network, right? If you're spanning everything in and out of a port and then you're encapsulating it and sending it over your IP network, you're literally doubling whatever that traffic is, now it's on your network twice.
921020	931560	The other thing you have to keep in mind is there is some encapsulation overhead, so you've got to make sure jumbo frames are supported everywhere, because otherwise packets are going to be too big.
932820	950700	I'll say I've seen, especially in VMware environments, sometimes you'll actually have dedicated monitoring interfaces, physical interfaces on the physical hosts that are dedicated for ERspan. That way you're avoiding putting monitor traffic onto the data networks.
950700	964240	Actually, Chris brings up a great point there. If I was designing an environment from scratch, I would absolutely make it a requirement to have a second 10 gig NIC in every single virtual host and have that connect to an out-of-band management network.
964240	969380	Then you could use ERspan to your heart's content without impacting your data network at all.
972430	987050	So this is a quick example, or a couple of examples, of how you might configure a SPAN session. And you can see a link to a documentation down below. This is Cisco specific. Most vendors are fairly similar though.
989070	995930	I will tell you, I've been doing this a long time, there are always slight little differences in the different software versions.
995930	1009790	So it's always a good idea to Google, hey, I've got this particular switch with this version of iOS. If we're talking Cisco, obviously, if you have another vendor, do whatever the similar search would be and make sure you have the right syntax.
1010530	1016030	Cisco likes, at least, to change their syntax quite a bit from version to version.
1016030	1028090	It's funny, as I said, I've been doing this almost 20 years, and it's always looked pretty much like this, but the way you arrange your VLANs has changed over the years and things of that nature.
1028670	1042650	By the way, so one of the things we're going to be talking about here is you'll notice that in one case we have an example of spanning a port, in another one we have an example of spanning a VLAN, and then you'll notice how you can send the traffic to a destination.
1042650	1052710	So a lot of times, by the way, you're not just limited to one destination, you can do multiple destinations, and I'll talk about this a little more, but virtual interfaces are often supported.
1053030	1056830	You'll notice there's some directionality there as well, which is something else we'll talk about.
1059590	1071250	Alright, so, as I said, we were going to talk about some of the spam sources. So, at its most basic, right, a spam is going to be just spanning a port or multiple ports.
1072290	1077090	And when you think of an interface, right, you might think, hey, yes, it's just an Ethernet interface on a Switch.
1077090	1088310	But a lot of Switch vendors also support things like virtual interfaces and aggregate interfaces, so in the Cisco world, like a port channel, you can spam a port channel, essentially.
1088450	1098650	And so what's the use case of spamming a particular port? It's usually in a large environment to target a particular network path or link.
1098650	1108670	Now, if you have a very small environment where you have, you know, like one Switch, like I have one Cisco Switch in my home environment, right, so I just spam every port on the Switch.
1109470	1118890	That is going to be less common in an enterprise environment, but if you do have a small environment, it's a reasonable approach to just spam all the ports.
1120170	1129850	However, when you get to the enterprise, it's much more common to use a VLAN spam, right, because most modern networks, things are segmented by VLAN.
1132050	1137930	So your source is going to be a VLAN or multiple VLANs, right? Typically, when I do this, it's multiple VLANs.
1137930	1143350	I'll go in and we'll say, hey, where are all the critical servers? Let's spam all those VLANs, and that's a lot of times where we start.
1143350	1150390	How often do people actually know what lives on which VLANs?
1151430	1154550	Well, that varies, of course.
1154670	1169770	It's usually, in my case, a discovery where you'll start with a wide caster net wide and capture many, many VLANs and then prune off as you validate what traffic lives where.
1170550	1179750	And I've actually done the opposite approach to, you know, we'll start and we'll say, OK, we're going to start with these VLANs and then inevitably you'll go look for traffic and it's not there.
1179950	1185190	And then, you know, you add some more. So Chris brings up a good point, right? I mean, there's always going to be some tweaking.
1185190	1189730	It's not like, hey, you have to have everything set in stone on day one.
1190870	1197130	You'll want to take your best shot at it, what you think you need, and then refine your data from there.
1198230	1205930	So a couple more notes on this. With port spans, by the way, typically you're going to span in both directions, so ingress and egress.
1206090	1211650	With VLANs, you only need to span ingress. And in fact, some newer switches only support ingress.
1211650	1214850	So you won't even have the choice anyway. So you don't even have to worry about it.
1215170	1218050	And then the last thing I'll say, a couple of last things on VLANs.
1218050	1223070	So they're a really good way to target certain assets, which is the reason they're the most common method.
1223070	1228230	And then when we get into Spineleaf, there are some similar concepts in there.
1228370	1232190	They just have different names because, of course, we want to confuse everyone.
1235760	1238300	So what are some common span issues?
1238460	1243160	So number one is probably oversubscribed monitor reports.
1243160	1248340	And I'll give you the simple explanation of this. Believe it or not, you can have a single...
1248340	1255600	if you span a single 1GIG interface in both directions, you can completely overrun a destination 1GIG interface.
1256060	1258080	And you say, Pete, how is that possible?
1258080	1263220	Well, think of it this way, an interface sends traffic in both directions, right?
1263220	1268560	So you're not... a 1GIG interface can actually have a total of 2 gigabits per second of traffic, right?
1268560	1270340	If you combine the ingress and the egress.
1270340	1278740	However, your destination interface that's sending the traffic to your analysis solution, that can only send at 1 gigabit per second, right?
1278760	1283020	Now, in reality, is spanning a 1GIG port going to overrun another 1GIG port?
1283020	1285180	No, almost never, right?
1285180	1290380	So it's very rare for switch ports to consistently run maxed out.
1290820	1293400	But it is something to be aware of, right?
1293440	1298200	And this is where we're going to get into, you know, having multiple destinations and aggregation.
1298440	1303340	In an enterprise environment, your destination port should almost always be at least 10 gigs.
1303840	1307120	Because you're going to... just trust me, you'll run into way less problems.
1307120	1315880	Trying to span out 1GIG interfaces in most modern networks, unless you're in a really small environment, is... you know, you're looking for trouble there.
1316120	1323740	I mentioned this on the previous slide, but when you have an interface span, typically, you're going to want to do both directions.
1323740	1326280	If you don't, you're going to get unidirectional traffic.
1326660	1330220	And you'll be really confused because your packet trace won't make any sense.
1331540	1332820	And we do have a note here.
1332820	1337960	If you go back to my simple, like, super small network example, if you're spanning all the interfaces on a switch,
1337960	1342580	in that case, you could do... you could just do receive.
1343540	1345820	But most of the time, you're going to do both.
1345900	1350240	And then I mentioned this before, but another one is ER-span without JumboFrame support.
1350540	1355640	So I know some implementations of ER-span actually mark the packets do not fragment.
1355640	1362260	So that means if you try to send a full-size packet that's been encapsulated over a network without JumboFrame support,
1362260	1364940	well, it's not going to go anywhere. It can't be fragmented.
1373130	1374850	So where do we span?
1375510	1381270	So in a three-tier architecture, when I say three-tier architecture, this is your traditional core distribution access.
1381630	1384190	Those are typically going to be your three options.
1385090	1386790	So we'll start with the core.
1389470	1392750	I sometimes work with people that span at the core.
1393390	1394630	Depends on the network.
1394990	1397150	I'll get to where we usually span.
1397150	1401590	But when you look at the core, your north-south traffic is going to be there a lot of times.
1401590	1402910	So you can get all your north-south traffic.
1402910	1405370	But your east-west traffic can be pretty limited.
1406190	1411630	Sometimes some networks will really only get routed traffic and things like that.
1412230	1417470	The advantage of spanning at the core is it does not require a lot of cabling or port density.
1418590	1423630	Some networks use more of a collapsed core where a lot of traffic goes to the core.
1423630	1426530	In those cases, it can be good to span at the core.
1426530	1430290	But generally, I don't recommend only spanning at the core.
1430290	1432510	You're just going to miss too much east-west traffic.
1433670	1439090	Usually, the keyword I look for is if a network architect describes their network as flat,
1439950	1445510	then you probably can span at the core and get a decent amount of the traffic.
1445510	1448870	But again, as Pete mentioned, you're still going to be missing a significant amount.
1453360	1457560	So distribution. This, as we say, is often the sweet spot.
1458160	1462100	Typically, you're still going to get... so we mentioned north-south on the last slide.
1462500	1465720	A lot of times you get north-south anyway because you move down the stack.
1465720	1469900	So you may have to do a little supplementing of certain transit networks,
1469900	1473240	or maybe you grab lots of DMV traffic that's up closer to the core,
1473240	1475120	but most of you are north-south.
1475120	1478600	It's going to stop all the way at the bottom of this diagram anyways.
1478600	1482860	So you're going to get it regardless of the layer you span at.
1482860	1486080	And what's the advantage of the distribution over the core?
1486080	1488440	More east-west traffic, period.
1488780	1494200	You're still not going to get all of it because you won't get traffic that never leaves an access switch.
1494200	1497180	So if you have two servers that talk to each other on the same access switch,
1497180	1499520	it's not going to come up in the distribution layer.
1500500	1507300	But you'll get a lot of stuff, and it doesn't have incredibly onerous tabling requirements.
1507300	1515160	So even in large networks, a lot of times we're only talking 8, 12, 16 distribution switches.
1515640	1519960	So it's not like you're going to be running dozens and dozens and dozens of tables,
1519960	1524220	or requiring massive aggregation switches.
1528340	1529980	Finally, the access layer.
1529980	1535280	Now, like I mentioned in a perfect world with PaaS, this is another perfect world situation.
1535280	1539360	If I could just wave a wand, I would just spam it the access layer.
1539360	1541460	Because you're going to get the most traffic.
1543200	1546360	Because you'll get almost all your east-west traffic there.
1546360	1548260	Even hosts on the same switch.
1548520	1553880	The problem is it can require extensive tabling.
1553880	1559260	I mean, we have customers with dozens, hundreds of access switches.
1560760	1565380	And that's a lot of labor and work to cable all that up.
1566180	1567540	You say getting crazy?
1567660	1571740	I do know some organizations who spam all their TopperRack switches though.
1571740	1573160	And they get great coverage.
1573260	1574840	So you have to look at your environment, right?
1574840	1579980	It depends how sprawling it is, how many access switches you have.
1580180	1583840	Sometimes what it might make sense to do is not do all the access switches,
1583840	1587400	but if you have a certain critical one where there's certain access for it, then you can look at those.
1587400	1595720	I think that just gets back to inter-packet acquisition strategy design with a plan.
1595760	1597460	Know what you want to get at.
1597460	1600740	And even if you don't necessarily know where it lives, if you know what you want,
1600740	1605140	you can figure it out over time as you do some discovery.
1609720	1612600	All right, so we talked about 3Gear.
1612600	1614600	Now we're going to talk a little bit about SpineLink.
1614880	1617960	This example is going to be very ACI-specific.
1618220	1620220	Just because that's what I'm most familiar with.
1621620	1624640	But other vendors are pretty similar.
1625140	1631680	The most confusing thing about SpineLink is they decided to use all different terminology to talk about everything.
1632580	1640480	But in some ways, when you start digging into it, it actually can be a little easier than traditional three-tier networks.
1640580	1643340	Number one, there's not three tiers anymore. There's only two, right?
1643340	1645560	There's Spine switches and Leaf switches.
1645980	1649680	And specifically in ACI networks, there's three span types.
1649680	1655080	There's an access span. That is very similar to a traditional span.
1655080	1657700	So an access span is on the Leaf switch.
1657760	1660020	And Leaf switches essentially have two different types of ports.
1660020	1664380	They have access ports, which is traditional switch ports. You hang a server off it.
1664380	1667320	And they have uplink ports, or fabric ports.
1667320	1669060	And that's what talks to the Spine switches.
1669060	1672640	So an access span would be spanning only access ports.
1672820	1678840	And for those, your destination can be local on that switch, or it can be a remote using ER spans.
1678840	1690780	Now, as I mentioned before, while this is technically ER span, it's not exactly the same as, you know, just a, you know, ER span in a vacuum.
1690840	1695640	Because you're doing ER span, but you're going somewhere else on the fabric.
1695640	1701720	And the fabric already has its own transport IP network as part of it.
1702900	1709920	So it's some of the down... you don't have to worry about as many of the downsides of ER span when you're dealing with Spine switches.
1710280	1712040	That's the high level summary.
1712360	1716400	It's much less likely you're going to, you know, stomp on your other traffic.
1716800	1720720	And, you know, in modern timing networks, they're all going to support jumbo-brings.
1722220	1725200	So the second type of span is a fabric span.
1725320	1731500	That's going to be spanning the uplink port between the Leaf and the Spine.
1732300	1738540	One thing to note here is when you go to look at the packets, if you wanted to, like, pull them into Wireshark, they're going to be DX LAN encapsulated.
1738540	1743260	So you're going to have to use... you're going to have to decapsulate them before you analyze them.
1743440	1756260	And actually, if you go beyond Wireshark, it's all... and you're looking at this kind of traffic, your aggregation solution or your analysis solution would also need to be able to decapsulate that traffic.
1757260	1759240	This one always uses ER span.
1759860	1767120	And then the final one, which is the one I at least personally have used the most and dealt with the most, is what they call a tenant span.
1767280	1770080	And that spans one or more endpoint groups.
1770340	1775340	And it's sort of beyond the scope of this presentation to get in to what an endpoint group is.
1775800	1780560	It's somewhat similar to a VLAN, though.
1781540	1784320	And that's why it ends up being a pretty popular approach, right?
1784320	1786920	Because it's a good way to target assets.
1786920	1790300	And kind of nice in Spineleaf, it doesn't really matter.
1790300	1794760	You can span... this endpoint group can be on all different heap switches.
1794900	1802440	And because it all uses ER span, you can pull it all back to one spot and send it over to your analysis solution or your aggregation network.
1805670	1816950	So once we get beyond just three-tier architectures, Spineleaf architectures, we actually end up having to have some additional challenges in enterprise deployments.
1816950	1820510	And one of those is going to be virtual environments.
1821230	1829390	The challenge in a virtual environment is that not all of the network traffic is going to hit a physical network.
1829390	1848010	So the image on the screen, the three virtual hosts that are on host one, if they're talking to each other, that traffic is going to stay on host one, even though they're all hooked into the same virtual distributed switch.
1848770	1856790	However, we always recommend do your physical spanning and tapping first.
1856790	1866150	This is a problem that gets blown out of proportion. You often get 80 plus percent of the traffic that you're looking for.
1866910	1879170	And it's funny we use 80% here because I would say, not only is it like 80% of the traffic you're looking for, I'd say about 80% of the time when people are worried about this, it ends up not being a problem.
1879170	1897270	Pete mentioned already ER span, so virtual distributed switches support ER span for critical workloads. You can target individual interfaces and get that traffic off of the hypervisor host.
1897270	1905450	There's also virtual taps, which are installed on a host by host or VM by VM basis.
1905450	1919850	You start running into the management overhead of installing those virtual taps, but for hypercritical pieces of infrastructure and pieces of applications, this can be a very viable solution.
1919850	1935010	Another less obvious solution is to separate key VMs and make sure that they're on different hosts, forcing their communication to actually traverse a physical network that you already have monitoring in place for.
1936190	1952950	And that last solution, by the way, is it sounds kind of funny. Hey, yeah, we'll just put them on separate hosts. But turns out that that easiest choose part is pretty important, right? Because some people are like, wait, that's all I have to do. And I'm like, yep, that's all you have to do. And they're like, oh, well, that's easy.
1953910	1961570	So, you know, don't, you know, I think we might have mentioned earlier, don't let the perfect be the enemy of the good. And that's a good example of that.
1962390	1979670	Alright, so let's talk a little bit about span aggregation, right? So what happens when you get these large environments is you don't have to just tap and span all over the place, you got to find a way to bring all that traffic back somewhere. So you can feed it one or more analysis and capture solutions.
1979670	1999670	Right? So one thing to note about most of these analysis and capture solutions is they are not designed for port density. So a lot of them use expensive high end network cards, you know, that can cost, you know, over $10,000 each. So they're not going to have 16 of them sitting, you know, in the solution, right?
1999670	2022950	So we need to, we need some way, you know, if I'm spanning 8 or 16 different switches, how do I get that fed into my analysis solution that has, say, you know, 2 or 4 or 10 gig ports on it? So what's the solution? An aggregation switch. And if you're not familiar with aggregation switches, there are some big vendors that do it. So if you've heard of Gigamon or AppCon or HCM, they do it.
2023610	2044870	Some switch vendors like Arista, you can also just configure their switches to be an aggregation switch. And really, when we say it's an aggregation switch, I mean, there are some that are purpose built, but it is a switch. Like the hardware, that's what it is. It's just a switch, except instead of sending traffic over it, you're going to use it to aggregate all your span and tap feeds, right?
2044870	2071470	So essentially, what it does is, let's say I have that example where I have 8 switches span, I need to get it into 2 ports, it's just going to multiplex all that traffic from those 8 ingest ports and send it out, 2 of them, to my analysis solution, right? And in fact, most of them can feed multiple analysis solutions. So maybe you have high speed packet capture, you've got an IDS, right? You've got some, you know, network detection and response solution, you want to feed all those.
2071470	2083590	So nice thing is, you know, you just have one set of spans and taps, feed it into your aggregation switch, and then it can feed multiple solutions that are connected to it.
2083630	2099330	And a couple things to add as well. So if you are thinking about taps at any sort of enterprise scale, you have to be thinking about span aggregation as well. Like, it is not an option to not consider that, not factor that into your deployment.
2099330	2111270	Yeah, one thing we didn't mention about taps that maybe we should have, if you have a single 10 gig fiber tap, you might say, okay, I'm going to tap one link. So how many links do you actually have that feed your analysis solution?
2111270	2121590	It's not one, it's actually two, because the way taps work is they feed you the ingress and the egress on that link, and they separate it out into two different links.
2121590	2131910	And they do that because, as I mentioned before, it's a full duplex link, and taps want to do perfect preservation. They give you two outputs so that there is no packet loss at all possible.
2132330	2139850	So Chris is exactly right. I mean, when you're talking taps, you can be talking a huge amount of links if you do them at any scale at all.
2139850	2148310	One additional thing, we've mentioned ER-SPAN a couple of times. We've also mentioned VXLAN in the case of a Spineleaf deployment.
2148990	2160450	SPAN aggregation solutions are capable a lot of times of doing decapsulation to get all of your traffic into whatever the base level traffic pattern that you want to analyze.
2160450	2168850	So you want just the raw traffic that a host would see, for example. Great, fantastic. A lot of SPAN aggregation solutions can get it to that specific...
2168850	2182490	And that's really important because if you feed, let's say Spineleaf encapsulated VXLAN traffic to an analysis solution and it's not decapsulated and that solution can't decapsulate it, it's not going to do anything for you.
2183450	2192450	You're going to be like, what are these IP addresses? Because they're all going to be the underlay network and it's not going to be your actual host IP addresses.
2193510	2204910	So some other things aggregation switches do, filtering traffic, they do deduplication, which we're going to talk about as well.
2204910	2212750	So let's talk a little bit about filtering. So this is probably one of the most important things when you're building these networks out.
2212750	2224390	So number one, packet analysis resources are not infinite, right? Nothing can accept infinite bandwidth. And writing packets to disks is also not infinite, right? You don't have an infinite disk space.
2224390	2230890	In fact, if you have, let's say, 100 gigabits per second link, that will fill up one petabyte of disk per day approximately.
2231590	2238050	And it's funny, when I meet with customers, they'll tell me, hey, I want 30 days of lookback. And I go, okay, what's your average throughput?
2238050	2249150	And they might say something, I don't know, five gigabits, 10 gigabits, whatever it is, you can literally go on Google and type like 10 gigabits per second, if you type the words out, to terabytes per day.
2249190	2256830	And then you can shock them and let them know like, oh, yeah, actually, you're going to need like 10 petabytes of disk if you want to keep that much data.
2257150	2261830	And then of course, they're like, well, maybe we don't need to keep that.
2261830	2268950	Some people can't dedicate whole data centers to storing PCAP. So it's just how much money you want to throw at the problem.
2269450	2275910	Yeah, I mean, there's government agencies, right, that just have stacks and stacks and stacks of storage to do that.
2275910	2284290	But, you know, even if some of the, you know, we've worked with some of the largest companies in the world, and, you know, they don't have infinite money, like, you know, any more than anyone else.
2285070	2296670	So one of the things that can really help you, though, is filtering. Because there is a, you know, to use a non-technical term, if you're spending a bunch of traffic, there's a good chance you're going to get some junk in there that you don't really care about.
2297830	2306510	You know, things like backup traffic, data replication traffic, that stuff, it's just not typically going to be very useful to look at.
2306510	2320130	And it can consume a massive amount of bandwidth and disk space. Like, I have customers where I'd say 30-40% of their traffic in a 24-hour period is backup traffic, if I were to spend, you know, most of the traffic in their data center.
2320230	2326130	So we're going to, if we get that in, we're going to filter it out. Some other good examples are going to be your VM infrastructure traffic.
2326130	2337610	So if you use NFS, for example, to mount your disk that all your VMs use, you don't want to, you typically do not want to spend that. It's going to be a massive amount of traffic. Or if you use iSCSI, same thing.
2337910	2343710	vMotion traffic is another one. It's, you know, it's a ton of traffic, and it's just generally not going to be that interesting.
2343970	2354290	And then finally, telemetry data can also consume a ton of bandwidth. So syslog, ELF, Elastic, NetFlow, etc.
2355990	2371470	Splunk replication, one of the chattier things that I've seen on the network. There's certain network links where I've seen Splunk replication in particular consuming 30 to 40% of the total available bandwidth.
2371470	2375310	So, filter it out. Get it out. I don't care.
2379940	2390420	Alright, now, what's the other big thing besides filtering? Duplicate traffic, right? And that's, how do we handle duplicate traffic? Well, there's a couple ways. One way is deduplication.
2390800	2403260	So, when you do this, if you haven't, you know, done a lot of scanning and tapping before, you're going to get duplicate traffic. It's normal. Don't freak out. Expect to see some of it, right?
2403260	2420220	The problem is when you get a lot of it, right? There's negative impacts of having tons and tons of duplicate traffic. Number one, you know, it's like, okay, if I have a duplicate of every single packet on my network, and I don't have any way to deduplicate it, it looks like I have twice as much traffic. It's going to waste a ton of lookback space.
2422040	2435540	Let's say I bought some packet analysis solution that can handle up to 10 gig of traffic. Well, if I'm feeding it two copies of every packet, I'm literally wasting half of its capability, right? And it can also make manual analysis more difficult.
2435540	2446120	You know, if any of you have ever opened up a Wireshark trace, and there's like six copies of every packet, it can be annoying to look at. I mean, there's ways to filter it out and stuff, but it can just make it more of a pain.
2447360	2458720	So a couple of solutions here. So one is going to be deduplication, which is a feature essentially that's supported by some aggregation switches and some packet analysis solutions.
2459560	2471740	And my general rule of thumb is that it works pretty well for lower levels of duplicate traffic, right? As I said, it's going to be normal to see some amount of duplicate traffic, and deduplication can handle that pretty well.
2473140	2487120	However, when you start getting a lot of duplicate traffic, sometimes these deduplication solutions just fall over. And it's not because they're poorly designed or bad or any of that. It's because it's a hard problem to solve that requires a lot of resources.
2487120	2498860	If I'm feeding 10 gigabits of traffic, essentially the way they work is they have a buffer, they have to buffer the traffic, they have a window that they look at, where they look, you know, if they see there's various heuristics you can use to identify duplicate packets.
2498900	2515020	But within that window, they try to identify them, right? But again, when you're talking 10 gig, 20 gig, 40 gig of traffic, it just requires a ton of memory and resources to buffer that and look any more than a few milliseconds for duplicates, right?
2515020	2528960	So what do you do when you have tons of duplicate traffic? Well, usually the reason is, is because you're spanning the same traffic or tapping the same traffic in a bunch of different places. And we're going to go back to the last slide, and we're going to filter that.
2528960	2555120	So, in my experience, once you get above, you know, 5, 10, maybe 15% mean when you're up at like 50% duplicate and higher, you need to go filter. And you can filter at your where you're actually doing the scan, you can filter at the aggregation switch. And one of the things that I've found that works pretty well a lot of times is identifying, you know, let's say I have, you know, as an example, let's say I have a firewall that's passive and I span both sides.
2555120	2577060	Well, I'm going to get the same package twice, right? Every time I want to filter that traffic out, essentially, all I have to do is figure out which MAC address is on which side of the firewall, right? And just filter the same traffic on both sides, you know, other than whatever gets blocked. And then I can use like a MAC address filter like that to pretty easily get rid of that traffic.
2577060	2586680	So just as a general rule, I find that, you know, facing your filters, which by the way, most aggregation switches can do, facing your filters on MAC addresses is a pretty good approach.
2611020	2639680	As much as I wish Amazon or Google would let me into their data centers and deploy taps and spans, I don't think they're so keen on doing that. And so what we've actually seen is the top three cloud vendors, AWS, GCP, and Azure have all announced, and in the case of AWS and GCP, made generally available span as a service.
2639960	2667060	Which I think I made up. But it is the same thing as a span on prem, where you say, I have a specific set of cloud workloads, whether that's a specific instance, whether that's an ENI, whether that is an entire VPC, and you say, feed that into a specific monitoring interface, feed it into whatever analysis tool that I want to grab those packets from.
2667060	2683920	In the case of Azure, they actually announced in 2018, it is still in private preview, just as a heads up. So for all the Azure customers out there, push on them. I know we are, because AWS and GCP can do it today.
2683920	2706700	Yeah, so one thing I'll add here, that's pretty nice is, believe it or not, even though it's in the cloud, this stuff is actually quite a bit easier sometimes than on prem spans, because you literally like in AWS, you literally just say, okay, I have these, you know, 520 servers, I want to span or these 300 servers I want the traffic from, and you just can just turn on the traffic mirroring for them.
2706700	2720340	There's no figuring out which VLAN they're on, or any of the things you have to deal with in an on prem data center, right? It's just they handle the implementation details under the covers, you just say, okay, I'm going to pay some money and get my traffic.
2720340	2737660	It's all API based, right? So you go get BOTO, or pick your poison, and you have give me all of these tagged instances, give me their data, you don't have to coordinate with network teams, you just get the data as soon as you want.
2740080	2762020	Containers, you're running Kubernetes, you're running Docker, they have this similar challenge to VMs in that there is traffic that never hits a physical network, or it's encapsulated in such a way where when it hits the physical network, it's hard to tell what the actual data is.
2762700	2783160	So first thing, because it's similar challenge to VMs, you actually have similar solution, tap a physical network and validate what you are and are not seeing. Again, in our experience, you actually see a decent amount of the critical traffic, especially the inter service traffic.
2783160	2794360	What you won't necessarily see is inter pod traffic. But we as an industry are still trying to decide on what the best way to actually capture this traffic is.
2794360	2807820	You can deploy a daemon set, which is a specific pod that you run on every single node within your K8s cluster. And so you can hook into the node networking and use that as a source of data.
2807820	2836180	You can also run a sidecar, which is a container that is run in the pod and use that to get inter pod intra node traffic as well. And some of the technologies that we're considering using the namespace hooking into the Docker zero interface and actually tapping those internal localhost interfaces as a way of getting to some of that data.
2838520	2857440	So with that, some of our takeaways, both Pete and I have been working with packets for a long time. And I think they're one of the arguably best sources of data, not just because of the richness of the data that's there, but because of how accessible they are.
2857440	2865740	You're not having to go find every host that you care about on the network, and instead can target the network as a whole.
2867140	2878940	I don't know about you, Pete, but it seems like every network is kind of the same, but they're all unique in their own fun and interesting ways.
2882740	2895240	But that doesn't mean that you can't use general principles as you start designing these architectures, and then modify them, use different strategies for different problems.
2895900	2900360	And that's, there's many different strategies, depending on the network.
2901240	2912380	And I would say these last two points are probably the most important. That identify critical assets step is by far the most important thing that you can do.
2912540	2921220	We can set aside all the technology implementation, all of that. If you don't, you're identifying critical assets, that's essentially your problem statement.
2921220	2925920	And if you don't have a good problem statement, it's going to be hard to solve that problem.
2926760	2933640	And I also think that don't let the perfect be the enemy of good is a really important point, too. And I'm going to go back to that log analogy.
2933660	2943880	Again, this is very similar. No one is going to capture every single log that is ever written by every single hosting device in a large environment.
2943880	2949240	It's just not going to happen. And this is the same thing here. You are never going to capture every single packet.
2949240	2952260	That doesn't mean that this can't be incredibly valuable.
2952260	2960220	And what I've found is getting, let's say, 90% of what you want, it's not too hard.
2960220	2964580	Then it's twice as hard to get the next 5%, to go to 95%.
2964580	2970280	Then if you go to 97.5%, it's twice as hard as it was to get to 95%.
2970280	2973040	And it's diminishing returns.
2973560	2978600	So again, you want to identify those critical assets and don't let the perfect be the enemy of the good.
2978600	2983240	As long as you have most of the stuff you need, most of the time,
2983240	2990120	I think you can provide some incredibly valuable visibility and investigative capabilities to your organization.
2990980	2995500	Perfect. So with that, thank you for your time.
2995500	2998520	Thanks for coming to DEF CON Safe Mode.
2999040	3001820	We're thankful to get the chance to speak to you guys.
3001820	3005900	So we'll leave it open to any questions.