1
00:00:00,940 --> 00:00:05,820
Our next talk epitomizes what this village is all about.

2
00:00:06,000 --> 00:00:10,120
To put it bluntly, packet hacking doesn't happen without packets.

3
00:00:10,120 --> 00:00:12,740
And also, PCAPSR didn't happen.

4
00:00:12,820 --> 00:00:27,680
Chris Arbella and Pete Anderson have spent decades working with Fortune 500 knockin' socks to implement advanced packet analysis solutions, better packet pipelines, and to get more from those packets.

5
00:00:27,680 --> 00:00:36,560
It is my pleasure to introduce to you Chris Arbella and Pete Anderson with their talk, Packet Acquisition, Building the Haystack.

6
00:00:38,240 --> 00:00:40,140
Take it away!

7
00:00:40,340 --> 00:00:44,720
Hi everybody, and welcome to Packet Acquisition, Building the Haystack.

8
00:00:44,720 --> 00:00:46,680
I'm Chris Arbella.

9
00:00:47,200 --> 00:00:48,600
And I'm Pete Anderson.

10
00:00:48,600 --> 00:00:56,160
And we've been designing large-scale enterprise packet acquisition capture solutions for a long time.

11
00:00:56,160 --> 00:01:00,920
Most of that is mostly Pete, if you can't tell.

12
00:01:01,240 --> 00:01:03,200
So what are we going to talk about today?

13
00:01:03,220 --> 00:01:09,980
So most of you have probably used Wireshark, you know what a packet capture file is.

14
00:01:10,080 --> 00:01:14,960
But when you look at a big environment, you might wonder, where does that data come from?

15
00:01:14,960 --> 00:01:19,400
How can I get it from all the different strategic points in a large environment?

16
00:01:19,400 --> 00:01:20,760
And that's what we're going to talk about today.

17
00:01:20,760 --> 00:01:23,880
How do you instrument your environment for packet capture?

18
00:01:24,020 --> 00:01:25,780
And there's various different topics.

19
00:01:25,780 --> 00:01:27,280
We'll start with local packet capture.

20
00:01:27,280 --> 00:01:29,540
Then we're going to talk about how do you scale it out?

21
00:01:29,540 --> 00:01:31,540
How do you cap-span your environment?

22
00:01:31,920 --> 00:01:36,380
And then what are some of the other considerations you have to deal with in the enterprise?

23
00:01:36,380 --> 00:01:44,780
And then at the end, we'll talk a little bit about cloud and containers, because those are some of the newer topics that we're all dealing with in the packet capture world today.

24
00:01:47,480 --> 00:01:48,340
All right.

25
00:01:48,340 --> 00:02:00,720
So before we dive in to talking about how we're going to build out packet capture infrastructure, I want to talk a little bit about what are packets good for?

26
00:02:00,720 --> 00:02:02,640
Why do we want to capture them in the first place?

27
00:02:02,660 --> 00:02:03,840
Why do we care?

28
00:02:04,580 --> 00:02:05,080
Exactly.

29
00:02:05,080 --> 00:02:06,440
Why should you care about this?

30
00:02:06,440 --> 00:02:08,120
Why do you want to build one of these out?

31
00:02:08,140 --> 00:02:09,340
So what are they good for?

32
00:02:10,360 --> 00:02:13,860
So some of you may have heard the expression, packets or it didn't happen.

33
00:02:13,920 --> 00:02:16,360
Basically, they are a ground truth.

34
00:02:16,700 --> 00:02:20,220
You see something in a packet trace, that's what happened.

35
00:02:20,320 --> 00:02:22,380
They're definitive information.

36
00:02:22,380 --> 00:02:29,980
And most attacks, exploits, breaches, etc., are going to involve traffic moving over the network, packets going over the network.

37
00:02:29,980 --> 00:02:34,000
And something I always say is that packets can tell you something about everything.

38
00:02:37,630 --> 00:02:42,010
So I'm not going to read through all of these, but there's a lot of different use cases.

39
00:02:42,010 --> 00:02:45,570
And I always like to use the analogy of logs.

40
00:02:45,830 --> 00:02:50,330
So a lot of you have probably dealt with log files and log aggregations.

41
00:02:50,330 --> 00:02:56,030
Very similar to packets in that they're good for all different types of things.

42
00:02:56,030 --> 00:02:58,790
They can answer all different types of questions.

43
00:02:59,210 --> 00:03:04,050
And actually, some of the challenges and trade-offs are pretty similar as well.

44
00:03:05,890 --> 00:03:11,790
So there's a bunch of different use cases for packets, a bunch of different types of packet analysis.

45
00:03:11,790 --> 00:03:16,910
The one that most of you are probably familiar with is offline packet analysis.

46
00:03:16,910 --> 00:03:25,290
So when someone gives you a PCAP or you've gathered a PCAP, and you use a tool like Wireshark to go and parse through that PCAP file.

47
00:03:25,570 --> 00:03:29,390
But there's also real-time analysis solutions.

48
00:03:29,430 --> 00:03:38,530
Things that do shallow packet inspection, where you're basically looking at the DCP IP5 tuple, what ports are talking, what IP addresses are talking.

49
00:03:38,690 --> 00:03:42,570
And honestly, if you're doing that, just use NetFlow.

50
00:03:42,570 --> 00:03:49,870
But then it gets more interesting once you start getting into deep packet inspection, where you're actually looking at the application payload.

51
00:03:49,870 --> 00:03:52,630
You're looking at L5, L6, L7.

52
00:03:53,350 --> 00:03:57,070
And especially when you start becoming session aware.

53
00:03:57,110 --> 00:04:04,870
So if you've used Snort, Siricata, Zeek, once you have something that can tell you when a connection has actually been established.

54
00:04:05,030 --> 00:04:15,550
And even getting to the point where you can do full reassembly, as in the case of doing file extraction from, whether it's HTTP or SMB, picket poisoning.

55
00:04:15,970 --> 00:04:24,290
Kind of the flip to that offline packet analysis is continuous packet capture, where you're just writing packets to disk as fast as you can.

56
00:04:24,290 --> 00:04:30,150
You have a rolling buffer, and you just keep filling those disks with raw packets.

57
00:04:30,150 --> 00:04:38,570
A good example of this is Google's open source stenographer, which again is meant for high speed packets to disk.

58
00:04:41,070 --> 00:04:43,550
So a little bit about the basics.

59
00:04:43,550 --> 00:04:44,830
Where do we start?

60
00:04:44,890 --> 00:05:00,030
We kind of already mentioned some of these technologies, but when you're just grabbing packets from a single host, from your local host, most common you're going to use something like TCP dump, Tshark, InGrep, or Wireshark.

61
00:05:00,030 --> 00:05:01,550
You're actually using a GUI.

62
00:05:01,550 --> 00:05:12,130
There are some other ways of getting packets off of a host, RPCAPD or remote PCAP, it's a part of PCAP, I believe.

63
00:05:12,130 --> 00:05:21,030
But it actually lets you take hosts from a local interface, and then direct them to some remote destination.

64
00:05:21,610 --> 00:05:27,490
These are good for one-offs, troubleshooting, forensics, when you have a host isolated.

65
00:05:27,490 --> 00:05:32,830
But this doesn't really scale to an actual enterprise deployment.

66
00:05:36,770 --> 00:05:41,950
Alright, so speaking of scale, how do we scale out?

67
00:05:41,950 --> 00:05:47,650
And the thing we're going to talk about first is identifying critical networks.

68
00:05:47,650 --> 00:05:57,830
So before you start implementing anything, by far the most important step is to identify what you care about and what you want to capture.

69
00:05:57,830 --> 00:06:00,730
Enterprise networks are very complicated.

70
00:06:01,310 --> 00:06:03,970
If you remember, I used the log analogy before.

71
00:06:03,970 --> 00:06:05,570
You're never going to capture every packet.

72
00:06:05,570 --> 00:06:06,670
It's impossible.

73
00:06:06,670 --> 00:06:14,770
It's just like you're never going to write every single log out on every host in your environment, and then ship all that over somewhere in the store.

74
00:06:14,770 --> 00:06:15,770
Very similar.

75
00:06:15,770 --> 00:06:18,750
So we have to target key network segments.

76
00:06:18,750 --> 00:06:22,070
And the questions to think about are, where are your critical assets?

77
00:06:22,630 --> 00:06:24,630
Where is your critical data stored?

78
00:06:24,630 --> 00:06:26,490
And how does that get accessed?

79
00:06:27,070 --> 00:06:33,630
And you want to focus on those network paths and capture traffic there.

80
00:06:33,630 --> 00:06:42,310
Because if you have a breach, if you have an attacker, if you have lateral movement, that's the stuff they're trying to get to, and that's where you really want to focus your attention.

81
00:06:44,790 --> 00:07:00,530
Some common examples of those critical assets, things in our experience, Active Directory or any authentication service, both internal within your network for your users, as well as for your customers.

82
00:07:00,530 --> 00:07:06,950
So you run the website, making sure that you actually are monitoring the authentication services for that website.

83
00:07:06,950 --> 00:07:13,510
Get repos, and actually tied to get repos, anything tied to the CICD pipeline.

84
00:07:13,650 --> 00:07:20,590
There was some exports around publicly exposed Jenkins endpoints, as an example.

85
00:07:20,590 --> 00:07:27,610
File shares, databases and data lakes, any sort of PCI network, even mainframes.

86
00:07:29,450 --> 00:07:32,070
Basically, does sensitive data live there?

87
00:07:32,070 --> 00:07:35,430
Is it something that is critical to your business making money?

88
00:07:35,610 --> 00:07:40,910
It is probably a critical network.

89
00:07:40,910 --> 00:07:48,890
So after you've picked out what your critical networks are, we have some prioritization on where we want to get the data from.

90
00:07:48,890 --> 00:07:53,370
How do we actually start gathering data from those networks?

91
00:07:53,370 --> 00:07:56,830
And there's really two common ways of doing it.

92
00:07:56,830 --> 00:07:58,270
The first is taps.

93
00:07:58,270 --> 00:08:00,030
The second is spans.

94
00:08:00,030 --> 00:08:02,870
A tap is actually purpose-built hardware.

95
00:08:02,870 --> 00:08:14,350
It's going to physically break the link between two different pieces of network infrastructure and take a passive copy from that individual link.

96
00:08:14,350 --> 00:08:16,330
So again, it's tied to everything.

97
00:08:16,330 --> 00:08:21,510
You'll need one tap for every link that you're trying to get packets from.

98
00:08:21,810 --> 00:08:27,010
On the span side, it's a configuration on network infrastructure, typically a switch.

99
00:08:27,010 --> 00:08:30,130
It's switched port analyzer, what span stands for.

100
00:08:30,130 --> 00:08:33,470
You'll also hear mirroring or monitor sessions.

101
00:08:33,470 --> 00:08:35,730
Those are all the same phrase.

102
00:08:36,470 --> 00:08:46,690
But it's taking a digital copy from the data plane of the network infrastructure and directing that copy to another location.

103
00:08:48,050 --> 00:08:55,070
So taps, they're less common than spans, and that's mostly because of the cons of taps.

104
00:08:55,070 --> 00:09:05,050
So I mentioned it's actually tied to individual links, which means that for every link you want to monitor, you actually need another tap.

105
00:09:05,050 --> 00:09:21,210
So you end up having to pay for many, many different taps, which also has an administrative burden, both from the outage required when you break that wire, as well as just how many individual taps that you have and you have to cable.

106
00:09:21,470 --> 00:09:34,050
On the other hand, because it's tied to the individual link, because it's actually a physical tap into that data feed, you end up with the best possible data feed.

107
00:09:34,050 --> 00:09:47,730
You can actually see on the screen how a fiber tap works, where you're actually slightly bending a cable and using refraction through that cable to pull out some percentage of the light passing over that fiber.

108
00:09:49,670 --> 00:09:58,970
So additionally, because it's tied to the physical infrastructure, there's no impact on any sort of switches, any sort of routers, anything like that.

109
00:09:58,970 --> 00:10:05,710
Yeah, so taps don't cause packet loss and they preserve packet order and timing perfectly.

110
00:10:08,490 --> 00:10:11,210
So where should you tap?

111
00:10:11,330 --> 00:10:14,690
There's typically a couple of different places that we see it.

112
00:10:14,690 --> 00:10:21,130
One is transit links, so north-south boundary into and out of a data center, for example.

113
00:10:21,130 --> 00:10:33,350
You'll also often see it between link layers or links between layers in a traditional three-tier network architecture, whether that's cord distribution or distribution to access.

114
00:10:33,890 --> 00:10:42,970
And in spine-leaf networks, you're actually going to most commonly see it on the link between the spines and the leaves themselves.

115
00:10:42,970 --> 00:10:49,510
And the reason for these different points of tapping, they give you the most bang for your buck.

116
00:10:49,510 --> 00:10:54,990
You'll have the most complete data set for critical network junctures.

117
00:10:59,420 --> 00:11:00,620
So spans.

118
00:11:01,360 --> 00:11:05,820
Spans are what you're going to run into most of the time, and they're what you're going to use most of the time.

119
00:11:05,820 --> 00:11:09,220
Taps are actually great, and in a perfect world, you would tap stuff all over the place.

120
00:11:09,220 --> 00:11:14,860
The problem is they're really expensive, and Chris mentioned some of the other cons.

121
00:11:15,100 --> 00:11:16,660
So we're going to use spans.

122
00:11:16,660 --> 00:11:18,060
Now, why are spans so great?

123
00:11:18,120 --> 00:11:19,380
Because guess what?

124
00:11:19,380 --> 00:11:20,540
You've already got them.

125
00:11:20,540 --> 00:11:28,100
So the vast majority of managed switches support at least some type of span port veering technology.

126
00:11:28,520 --> 00:11:33,420
They are, indeed, inexpensive and ubiquitous, but it's extremely flexible, too.

127
00:11:33,420 --> 00:11:36,200
So when you think about a tap, you're tapping a link.

128
00:11:36,300 --> 00:11:46,560
With spans, the basic idea is you're spanning a port, but there are a lot of more advanced configurations around VLAN, filtering, and other things along those lines.

129
00:11:46,560 --> 00:11:50,360
And you can target traffic very specifically with a span.

130
00:11:50,820 --> 00:11:53,000
So, sounds like spans are great.

131
00:11:53,000 --> 00:11:54,240
Why don't we just always use them?

132
00:11:54,240 --> 00:11:56,200
So there are a couple of downsides.

133
00:11:56,780 --> 00:11:58,800
The data can be lower fidelity.

134
00:11:58,800 --> 00:12:10,700
You really have to watch out for over-subscribing your destination and having what I sometimes call observation drops, which is essentially a packet loss on the span destination.

135
00:12:12,100 --> 00:12:15,400
Occasionally, you'll see some negative impact on switch performance.

136
00:12:15,400 --> 00:12:16,540
It's pretty rare.

137
00:12:16,540 --> 00:12:17,920
It can happen.

138
00:12:17,920 --> 00:12:20,760
It's less common than it used to be.

139
00:12:21,100 --> 00:12:31,960
Now that switch hardware has gotten more powerful, but every once in a while, you'll have a switch vendor make an implementation change, which I'm sure some of you are familiar with, that can cause issues.

140
00:12:31,960 --> 00:12:33,840
But generally, that's pretty rare.

141
00:12:34,560 --> 00:12:41,300
And overall, this is the approach that you're going to use the majority of the time, at least in most environments.

142
00:12:41,300 --> 00:12:49,320
I'd say 98% of the environments that I work in are all based off of spans versus caps.

143
00:12:53,640 --> 00:12:55,780
So let's talk a little bit about the types of spans.

144
00:12:55,780 --> 00:12:58,400
So you may hear these different terms.

145
00:12:58,500 --> 00:13:00,140
Span, RSpan, ERspan.

146
00:13:00,140 --> 00:13:01,220
What do they all mean?

147
00:13:01,360 --> 00:13:07,460
So a span, or a port mirror, that is going to be the most common span type, and that is going to be local to a single switch.

148
00:13:07,460 --> 00:13:10,520
So a simple example.

149
00:13:10,520 --> 00:13:19,640
I want to span all the traffic going in and out of port 1, and I'm going to send it all to port 2, and I'm going to put my packet analysis, packet capture solution on port 2.

150
00:13:21,000 --> 00:13:22,980
That's going to be your most common span type.

151
00:13:23,360 --> 00:13:26,520
RSpan, you're not going to run into very much, at least in my experience.

152
00:13:26,520 --> 00:13:28,340
I've been doing this for almost 20 years.

153
00:13:28,340 --> 00:13:31,000
I think I've run into RSpan like twice.

154
00:13:31,200 --> 00:13:36,640
And both times it didn't work that well, so that's probably why I never run into it.

155
00:13:36,640 --> 00:13:38,220
I've never had it.

156
00:13:40,180 --> 00:13:40,900
Exactly.

157
00:13:41,180 --> 00:13:49,780
So basically, an RSpan is your source is on one switch, your destination can be on the other, and you use an L2 transport to get there.

158
00:13:50,980 --> 00:13:59,760
We're going to talk about this later, but generally when you have a situation like that, an aggregation solution is much more preferable, and I think that's one of the big reasons we don't see RSpan.

159
00:14:01,120 --> 00:14:03,620
ERspan is a little bit more common.

160
00:14:03,720 --> 00:14:10,380
It's somewhat similar to an RSpan, except that instead of a Layer 2 transport, it uses a Layer 3 transport.

161
00:14:10,380 --> 00:14:20,360
So you can span stuff on one switch and send it to another, and it does that by encapsulating the spanned packets in a Layer 3 GRE tunnel.

162
00:14:20,360 --> 00:14:41,420
In a traditional switch network, it's somewhat uncommon, but when you get into the VMware world, it is more common because the VMware distributed switch supports an ERspan, so you can essentially span virtual ports on that virtual distributed switch and then send it somewhere else.

163
00:14:41,420 --> 00:15:03,380
It's also fairly common in Spineleaf networks, although that is a little bit different because although it's technically an ERspan, it doesn't leave the... you're not always going over the data network, it's usually using that underlay network that a lot of the Spineleaf implementations use,

164
00:15:03,380 --> 00:15:05,180
so it doesn't leave it that way.

165
00:15:06,100 --> 00:15:11,740
The big caveat with ERspan is that it can put a lot of traffic on your network, right?

166
00:15:11,740 --> 00:15:20,700
If you're spanning everything in and out of a port and then you're encapsulating it and sending it over your IP network, you're literally doubling whatever that traffic is, now it's on your network twice.

167
00:15:21,020 --> 00:15:31,560
The other thing you have to keep in mind is there is some encapsulation overhead, so you've got to make sure jumbo frames are supported everywhere, because otherwise packets are going to be too big.

168
00:15:32,820 --> 00:15:45,740
I'll say I've seen, especially in VMware environments, sometimes you'll actually have dedicated monitoring interfaces, physical interfaces on the physical hosts that are dedicated for ERspan.

169
00:15:45,740 --> 00:15:50,700
That way you're avoiding putting monitor traffic onto the data networks.

170
00:15:50,700 --> 00:15:52,400
Actually, Chris brings up a great point there.

171
00:15:52,400 --> 00:16:04,240
If I was designing an environment from scratch, I would absolutely make it a requirement to have a second 10 gig NIC in every single virtual host and have that connect to an out-of-band management network.

172
00:16:04,240 --> 00:16:09,380
Then you could use ERspan to your heart's content without impacting your data network at all.

173
00:16:12,430 --> 00:16:19,930
So this is a quick example, or a couple of examples, of how you might configure a SPAN session.

174
00:16:19,930 --> 00:16:22,390
And you can see a link to a documentation down below.

175
00:16:22,390 --> 00:16:23,930
This is Cisco specific.

176
00:16:23,950 --> 00:16:27,050
Most vendors are fairly similar though.

177
00:16:29,070 --> 00:16:35,930
I will tell you, I've been doing this a long time, there are always slight little differences in the different software versions.

178
00:16:35,930 --> 00:16:41,530
So it's always a good idea to Google, hey, I've got this particular switch with this version of iOS.

179
00:16:41,530 --> 00:16:49,790
If we're talking Cisco, obviously, if you have another vendor, do whatever the similar search would be and make sure you have the right syntax.

180
00:16:50,530 --> 00:16:56,030
Cisco likes, at least, to change their syntax quite a bit from version to version.

181
00:16:56,030 --> 00:17:08,090
It's funny, as I said, I've been doing this almost 20 years, and it's always looked pretty much like this, but the way you arrange your VLANs has changed over the years and things of that nature.

182
00:17:08,670 --> 00:17:22,650
By the way, so one of the things we're going to be talking about here is you'll notice that in one case we have an example of spanning a port, in another one we have an example of spanning a VLAN, and then you'll notice how you can send the traffic to a destination.

183
00:17:22,650 --> 00:17:32,710
So a lot of times, by the way, you're not just limited to one destination, you can do multiple destinations, and I'll talk about this a little more, but virtual interfaces are often supported.

184
00:17:33,030 --> 00:17:36,830
You'll notice there's some directionality there as well, which is something else we'll talk about.

185
00:17:39,590 --> 00:17:43,590
Alright, so, as I said, we were going to talk about some of the spam sources.

186
00:17:43,590 --> 00:17:51,250
So, at its most basic, right, a spam is going to be just spanning a port or multiple ports.

187
00:17:52,290 --> 00:17:57,090
And when you think of an interface, right, you might think, hey, yes, it's just an Ethernet interface on a Switch.

188
00:17:57,090 --> 00:18:08,310
But a lot of Switch vendors also support things like virtual interfaces and aggregate interfaces, so in the Cisco world, like a port channel, you can spam a port channel, essentially.

189
00:18:08,450 --> 00:18:12,590
And so what's the use case of spamming a particular port?

190
00:18:13,510 --> 00:18:18,650
It's usually in a large environment to target a particular network path or link.

191
00:18:18,650 --> 00:18:28,670
Now, if you have a very small environment where you have, you know, like one Switch, like I have one Cisco Switch in my home environment, right, so I just spam every port on the Switch.

192
00:18:29,470 --> 00:18:38,890
That is going to be less common in an enterprise environment, but if you do have a small environment, it's a reasonable approach to just spam all the ports.

193
00:18:40,170 --> 00:18:49,850
However, when you get to the enterprise, it's much more common to use a VLAN spam, right, because most modern networks, things are segmented by VLAN.

194
00:18:52,050 --> 00:18:55,670
So your source is going to be a VLAN or multiple VLANs, right?

195
00:18:55,670 --> 00:18:57,930
Typically, when I do this, it's multiple VLANs.

196
00:18:57,930 --> 00:19:00,650
I'll go in and we'll say, hey, where are all the critical servers?

197
00:19:00,650 --> 00:19:03,350
Let's spam all those VLANs, and that's a lot of times where we start.

198
00:19:03,350 --> 00:19:10,390
How often do people actually know what lives on which VLANs?

199
00:19:11,430 --> 00:19:14,550
Well, that varies, of course.

200
00:19:14,670 --> 00:19:29,770
It's usually, in my case, a discovery where you'll start with a wide caster net wide and capture many, many VLANs and then prune off as you validate what traffic lives where.

201
00:19:30,550 --> 00:19:39,750
And I've actually done the opposite approach to, you know, we'll start and we'll say, OK, we're going to start with these VLANs and then inevitably you'll go look for traffic and it's not there.

202
00:19:39,950 --> 00:19:41,730
And then, you know, you add some more.

203
00:19:41,990 --> 00:19:43,550
So Chris brings up a good point, right?

204
00:19:43,550 --> 00:19:45,190
I mean, there's always going to be some tweaking.

205
00:19:45,190 --> 00:19:49,730
It's not like, hey, you have to have everything set in stone on day one.

206
00:19:50,870 --> 00:19:57,130
You'll want to take your best shot at it, what you think you need, and then refine your data from there.

207
00:19:58,230 --> 00:20:00,550
So a couple more notes on this.

208
00:20:00,570 --> 00:20:05,930
With port spans, by the way, typically you're going to span in both directions, so ingress and egress.

209
00:20:06,090 --> 00:20:08,610
With VLANs, you only need to span ingress.

210
00:20:08,610 --> 00:20:11,650
And in fact, some newer switches only support ingress.

211
00:20:11,650 --> 00:20:13,410
So you won't even have the choice anyway.

212
00:20:13,410 --> 00:20:14,850
So you don't even have to worry about it.

213
00:20:15,170 --> 00:20:18,050
And then the last thing I'll say, a couple of last things on VLANs.

214
00:20:18,050 --> 00:20:23,070
So they're a really good way to target certain assets, which is the reason they're the most common method.

215
00:20:23,070 --> 00:20:28,230
And then when we get into Spineleaf, there are some similar concepts in there.

216
00:20:28,370 --> 00:20:32,190
They just have different names because, of course, we want to confuse everyone.

217
00:20:35,760 --> 00:20:38,300
So what are some common span issues?

218
00:20:38,460 --> 00:20:43,160
So number one is probably oversubscribed monitor reports.

219
00:20:43,160 --> 00:20:45,640
And I'll give you the simple explanation of this.

220
00:20:45,640 --> 00:20:48,340
Believe it or not, you can have a single...

221
00:20:48,340 --> 00:20:55,600
if you span a single 1GIG interface in both directions, you can completely overrun a destination 1GIG interface.

222
00:20:56,060 --> 00:20:58,080
And you say, Pete, how is that possible?

223
00:20:58,080 --> 00:21:03,220
Well, think of it this way, an interface sends traffic in both directions, right?

224
00:21:03,220 --> 00:21:08,560
So you're not... a 1GIG interface can actually have a total of 2 gigabits per second of traffic, right?

225
00:21:08,560 --> 00:21:10,340
If you combine the ingress and the egress.

226
00:21:10,340 --> 00:21:18,740
However, your destination interface that's sending the traffic to your analysis solution, that can only send at 1 gigabit per second, right?

227
00:21:18,760 --> 00:21:23,020
Now, in reality, is spanning a 1GIG port going to overrun another 1GIG port?

228
00:21:23,020 --> 00:21:25,180
No, almost never, right?

229
00:21:25,180 --> 00:21:30,380
So it's very rare for switch ports to consistently run maxed out.

230
00:21:30,820 --> 00:21:33,400
But it is something to be aware of, right?

231
00:21:33,440 --> 00:21:38,200
And this is where we're going to get into, you know, having multiple destinations and aggregation.

232
00:21:38,440 --> 00:21:43,340
In an enterprise environment, your destination port should almost always be at least 10 gigs.

233
00:21:43,840 --> 00:21:47,120
Because you're going to... just trust me, you'll run into way less problems.

234
00:21:47,120 --> 00:21:55,880
Trying to span out 1GIG interfaces in most modern networks, unless you're in a really small environment, is... you know, you're looking for trouble there.

235
00:21:56,120 --> 00:22:03,740
I mentioned this on the previous slide, but when you have an interface span, typically, you're going to want to do both directions.

236
00:22:03,740 --> 00:22:06,280
If you don't, you're going to get unidirectional traffic.

237
00:22:06,660 --> 00:22:10,220
And you'll be really confused because your packet trace won't make any sense.

238
00:22:11,540 --> 00:22:12,820
And we do have a note here.

239
00:22:12,820 --> 00:22:22,580
If you go back to my simple, like, super small network example, if you're spanning all the interfaces on a switch, in that case, you could do... you could just do receive.

240
00:22:23,540 --> 00:22:25,820
But most of the time, you're going to do both.

241
00:22:25,900 --> 00:22:30,240
And then I mentioned this before, but another one is ER-span without JumboFrame support.

242
00:22:30,540 --> 00:22:35,640
So I know some implementations of ER-span actually mark the packets do not fragment.

243
00:22:35,640 --> 00:22:43,580
So that means if you try to send a full-size packet that's been encapsulated over a network without JumboFrame support, well, it's not going to go anywhere.

244
00:22:43,580 --> 00:22:44,940
It can't be fragmented.

245
00:22:53,130 --> 00:22:54,850
So where do we span?

246
00:22:55,510 --> 00:23:01,270
So in a three-tier architecture, when I say three-tier architecture, this is your traditional core distribution access.

247
00:23:01,630 --> 00:23:04,190
Those are typically going to be your three options.

248
00:23:05,090 --> 00:23:06,790
So we'll start with the core.

249
00:23:09,470 --> 00:23:12,750
I sometimes work with people that span at the core.

250
00:23:13,390 --> 00:23:14,630
Depends on the network.

251
00:23:14,990 --> 00:23:17,150
I'll get to where we usually span.

252
00:23:17,150 --> 00:23:21,590
But when you look at the core, your north-south traffic is going to be there a lot of times.

253
00:23:21,590 --> 00:23:22,910
So you can get all your north-south traffic.

254
00:23:22,910 --> 00:23:25,370
But your east-west traffic can be pretty limited.

255
00:23:26,190 --> 00:23:31,630
Sometimes some networks will really only get routed traffic and things like that.

256
00:23:32,230 --> 00:23:37,470
The advantage of spanning at the core is it does not require a lot of cabling or port density.

257
00:23:38,590 --> 00:23:43,630
Some networks use more of a collapsed core where a lot of traffic goes to the core.

258
00:23:43,630 --> 00:23:46,530
In those cases, it can be good to span at the core.

259
00:23:46,530 --> 00:23:50,290
But generally, I don't recommend only spanning at the core.

260
00:23:50,290 --> 00:23:52,510
You're just going to miss too much east-west traffic.

261
00:23:53,670 --> 00:24:05,510
Usually, the keyword I look for is if a network architect describes their network as flat, then you probably can span at the core and get a decent amount of the traffic.

262
00:24:05,510 --> 00:24:08,870
But again, as Pete mentioned, you're still going to be missing a significant amount.

263
00:24:13,360 --> 00:24:14,440
So distribution.

264
00:24:14,460 --> 00:24:17,560
This, as we say, is often the sweet spot.

265
00:24:18,160 --> 00:24:22,100
Typically, you're still going to get... so we mentioned north-south on the last slide.

266
00:24:22,500 --> 00:24:25,720
A lot of times you get north-south anyway because you move down the stack.

267
00:24:25,720 --> 00:24:35,120
So you may have to do a little supplementing of certain transit networks, or maybe you grab lots of DMV traffic that's up closer to the core, but most of you are north-south.

268
00:24:35,120 --> 00:24:38,600
It's going to stop all the way at the bottom of this diagram anyways.

269
00:24:38,600 --> 00:24:42,860
So you're going to get it regardless of the layer you span at.

270
00:24:42,860 --> 00:24:46,080
And what's the advantage of the distribution over the core?

271
00:24:46,080 --> 00:24:48,440
More east-west traffic, period.

272
00:24:48,780 --> 00:24:54,200
You're still not going to get all of it because you won't get traffic that never leaves an access switch.

273
00:24:54,200 --> 00:24:59,520
So if you have two servers that talk to each other on the same access switch, it's not going to come up in the distribution layer.

274
00:25:00,500 --> 00:25:07,300
But you'll get a lot of stuff, and it doesn't have incredibly onerous tabling requirements.

275
00:25:07,300 --> 00:25:15,160
So even in large networks, a lot of times we're only talking 8, 12, 16 distribution switches.

276
00:25:15,640 --> 00:25:24,220
So it's not like you're going to be running dozens and dozens and dozens of tables, or requiring massive aggregation switches.

277
00:25:28,340 --> 00:25:29,980
Finally, the access layer.

278
00:25:29,980 --> 00:25:35,280
Now, like I mentioned in a perfect world with PaaS, this is another perfect world situation.

279
00:25:35,280 --> 00:25:39,360
If I could just wave a wand, I would just spam it the access layer.

280
00:25:39,360 --> 00:25:41,460
Because you're going to get the most traffic.

281
00:25:43,200 --> 00:25:46,360
Because you'll get almost all your east-west traffic there.

282
00:25:46,360 --> 00:25:48,260
Even hosts on the same switch.

283
00:25:48,520 --> 00:25:53,880
The problem is it can require extensive tabling.

284
00:25:53,880 --> 00:25:59,260
I mean, we have customers with dozens, hundreds of access switches.

285
00:26:00,760 --> 00:26:05,380
And that's a lot of labor and work to cable all that up.

286
00:26:06,180 --> 00:26:07,540
You say getting crazy?

287
00:26:07,660 --> 00:26:11,740
I do know some organizations who spam all their TopperRack switches though.

288
00:26:11,740 --> 00:26:13,160
And they get great coverage.

289
00:26:13,260 --> 00:26:14,840
So you have to look at your environment, right?

290
00:26:14,840 --> 00:26:19,980
It depends how sprawling it is, how many access switches you have.

291
00:26:20,180 --> 00:26:27,400
Sometimes what it might make sense to do is not do all the access switches, but if you have a certain critical one where there's certain access for it, then you can look at those.

292
00:26:27,400 --> 00:26:35,720
I think that just gets back to inter-packet acquisition strategy design with a plan.

293
00:26:35,760 --> 00:26:37,460
Know what you want to get at.

294
00:26:37,460 --> 00:26:45,140
And even if you don't necessarily know where it lives, if you know what you want, you can figure it out over time as you do some discovery.

295
00:26:49,720 --> 00:26:52,600
All right, so we talked about 3Gear.

296
00:26:52,600 --> 00:26:54,600
Now we're going to talk a little bit about SpineLink.

297
00:26:54,880 --> 00:26:57,960
This example is going to be very ACI-specific.

298
00:26:58,220 --> 00:27:00,220
Just because that's what I'm most familiar with.

299
00:27:01,620 --> 00:27:04,640
But other vendors are pretty similar.

300
00:27:05,140 --> 00:27:11,680
The most confusing thing about SpineLink is they decided to use all different terminology to talk about everything.

301
00:27:12,580 --> 00:27:20,480
But in some ways, when you start digging into it, it actually can be a little easier than traditional three-tier networks.

302
00:27:20,580 --> 00:27:22,420
Number one, there's not three tiers anymore.

303
00:27:22,420 --> 00:27:23,340
There's only two, right?

304
00:27:23,340 --> 00:27:25,560
There's Spine switches and Leaf switches.

305
00:27:25,980 --> 00:27:29,680
And specifically in ACI networks, there's three span types.

306
00:27:29,680 --> 00:27:31,360
There's an access span.

307
00:27:31,360 --> 00:27:35,080
That is very similar to a traditional span.

308
00:27:35,080 --> 00:27:37,700
So an access span is on the Leaf switch.

309
00:27:37,760 --> 00:27:40,020
And Leaf switches essentially have two different types of ports.

310
00:27:40,020 --> 00:27:43,020
They have access ports, which is traditional switch ports.

311
00:27:43,020 --> 00:27:44,380
You hang a server off it.

312
00:27:44,380 --> 00:27:47,320
And they have uplink ports, or fabric ports.

313
00:27:47,320 --> 00:27:49,060
And that's what talks to the Spine switches.

314
00:27:49,060 --> 00:27:52,640
So an access span would be spanning only access ports.

315
00:27:52,820 --> 00:27:58,840
And for those, your destination can be local on that switch, or it can be a remote using ER spans.

316
00:27:58,840 --> 00:28:10,780
Now, as I mentioned before, while this is technically ER span, it's not exactly the same as, you know, just a, you know, ER span in a vacuum.

317
00:28:10,840 --> 00:28:15,640
Because you're doing ER span, but you're going somewhere else on the fabric.

318
00:28:15,640 --> 00:28:21,720
And the fabric already has its own transport IP network as part of it.

319
00:28:22,900 --> 00:28:29,920
So it's some of the down... you don't have to worry about as many of the downsides of ER span when you're dealing with Spine switches.

320
00:28:30,280 --> 00:28:32,040
That's the high level summary.

321
00:28:32,360 --> 00:28:36,400
It's much less likely you're going to, you know, stomp on your other traffic.

322
00:28:36,800 --> 00:28:40,720
And, you know, in modern timing networks, they're all going to support jumbo-brings.

323
00:28:42,220 --> 00:28:45,200
So the second type of span is a fabric span.

324
00:28:45,320 --> 00:28:51,500
That's going to be spanning the uplink port between the Leaf and the Spine.

325
00:28:52,300 --> 00:28:58,540
One thing to note here is when you go to look at the packets, if you wanted to, like, pull them into Wireshark, they're going to be DX LAN encapsulated.

326
00:28:58,540 --> 00:29:03,260
So you're going to have to use... you're going to have to decapsulate them before you analyze them.

327
00:29:03,440 --> 00:29:16,260
And actually, if you go beyond Wireshark, it's all... and you're looking at this kind of traffic, your aggregation solution or your analysis solution would also need to be able to decapsulate that traffic.

328
00:29:17,260 --> 00:29:19,240
This one always uses ER span.

329
00:29:19,860 --> 00:29:27,120
And then the final one, which is the one I at least personally have used the most and dealt with the most, is what they call a tenant span.

330
00:29:27,280 --> 00:29:30,080
And that spans one or more endpoint groups.

331
00:29:30,340 --> 00:29:35,340
And it's sort of beyond the scope of this presentation to get in to what an endpoint group is.

332
00:29:35,800 --> 00:29:40,560
It's somewhat similar to a VLAN, though.

333
00:29:41,540 --> 00:29:44,320
And that's why it ends up being a pretty popular approach, right?

334
00:29:44,320 --> 00:29:46,920
Because it's a good way to target assets.

335
00:29:46,920 --> 00:29:50,300
And kind of nice in Spineleaf, it doesn't really matter.

336
00:29:50,300 --> 00:29:54,760
You can span... this endpoint group can be on all different heap switches.

337
00:29:54,900 --> 00:30:02,440
And because it all uses ER span, you can pull it all back to one spot and send it over to your analysis solution or your aggregation network.

338
00:30:05,670 --> 00:30:16,950
So once we get beyond just three-tier architectures, Spineleaf architectures, we actually end up having to have some additional challenges in enterprise deployments.

339
00:30:16,950 --> 00:30:20,510
And one of those is going to be virtual environments.

340
00:30:21,230 --> 00:30:29,390
The challenge in a virtual environment is that not all of the network traffic is going to hit a physical network.

341
00:30:29,390 --> 00:30:48,010
So the image on the screen, the three virtual hosts that are on host one, if they're talking to each other, that traffic is going to stay on host one, even though they're all hooked into the same virtual distributed switch.

342
00:30:48,770 --> 00:30:56,790
However, we always recommend do your physical spanning and tapping first.

343
00:30:56,790 --> 00:31:00,350
This is a problem that gets blown out of proportion.

344
00:31:01,110 --> 00:31:06,150
You often get 80 plus percent of the traffic that you're looking for.

345
00:31:06,910 --> 00:31:19,170
And it's funny we use 80% here because I would say, not only is it like 80% of the traffic you're looking for, I'd say about 80% of the time when people are worried about this, it ends up not being a problem.

346
00:31:19,170 --> 00:31:29,770
Pete mentioned already ER span, so virtual distributed switches support ER span for critical workloads.

347
00:31:29,770 --> 00:31:37,270
You can target individual interfaces and get that traffic off of the hypervisor host.

348
00:31:37,270 --> 00:31:45,450
There's also virtual taps, which are installed on a host by host or VM by VM basis.

349
00:31:45,450 --> 00:31:59,850
You start running into the management overhead of installing those virtual taps, but for hypercritical pieces of infrastructure and pieces of applications, this can be a very viable solution.

350
00:31:59,850 --> 00:32:15,010
Another less obvious solution is to separate key VMs and make sure that they're on different hosts, forcing their communication to actually traverse a physical network that you already have monitoring in place for.

351
00:32:16,190 --> 00:32:20,850
And that last solution, by the way, is it sounds kind of funny.

352
00:32:20,850 --> 00:32:22,870
Hey, yeah, we'll just put them on separate hosts.

353
00:32:23,230 --> 00:32:27,670
But turns out that that easiest choose part is pretty important, right?

354
00:32:27,670 --> 00:32:30,010
Because some people are like, wait, that's all I have to do.

355
00:32:30,010 --> 00:32:31,490
And I'm like, yep, that's all you have to do.

356
00:32:31,490 --> 00:32:32,950
And they're like, oh, well, that's easy.

357
00:32:33,910 --> 00:32:40,030
So, you know, don't, you know, I think we might have mentioned earlier, don't let the perfect be the enemy of the good.

358
00:32:40,030 --> 00:32:41,570
And that's a good example of that.

359
00:32:42,390 --> 00:32:45,310
Alright, so let's talk a little bit about span aggregation, right?

360
00:32:45,310 --> 00:32:54,650
So what happens when you get these large environments is you don't have to just tap and span all over the place, you got to find a way to bring all that traffic back somewhere.

361
00:32:55,050 --> 00:32:59,670
So you can feed it one or more analysis and capture solutions.

362
00:32:59,670 --> 00:33:00,470
Right?

363
00:33:00,470 --> 00:33:06,630
So one thing to note about most of these analysis and capture solutions is they are not designed for port density.

364
00:33:06,650 --> 00:33:14,010
So a lot of them use expensive high end network cards, you know, that can cost, you know, over $10,000 each.

365
00:33:14,010 --> 00:33:19,670
So they're not going to have 16 of them sitting, you know, in the solution, right?

366
00:33:19,670 --> 00:33:31,430
So we need to, we need some way, you know, if I'm spanning 8 or 16 different switches, how do I get that fed into my analysis solution that has, say, you know, 2 or 4 or 10 gig ports on it?

367
00:33:31,750 --> 00:33:32,830
So what's the solution?

368
00:33:32,830 --> 00:33:34,190
An aggregation switch.

369
00:33:34,470 --> 00:33:39,570
And if you're not familiar with aggregation switches, there are some big vendors that do it.

370
00:33:39,570 --> 00:33:42,950
So if you've heard of Gigamon or AppCon or HCM, they do it.

371
00:33:43,610 --> 00:33:49,330
Some switch vendors like Arista, you can also just configure their switches to be an aggregation switch.

372
00:33:49,330 --> 00:33:55,090
And really, when we say it's an aggregation switch, I mean, there are some that are purpose built, but it is a switch.

373
00:33:55,090 --> 00:33:56,830
Like the hardware, that's what it is.

374
00:33:56,830 --> 00:34:04,870
It's just a switch, except instead of sending traffic over it, you're going to use it to aggregate all your span and tap feeds, right?

375
00:34:04,870 --> 00:34:18,650
So essentially, what it does is, let's say I have that example where I have 8 switches span, I need to get it into 2 ports, it's just going to multiplex all that traffic from those 8 ingest ports and send it out, 2 of them, to my analysis solution, right?

376
00:34:18,790 --> 00:34:23,110
And in fact, most of them can feed multiple analysis solutions.

377
00:34:23,110 --> 00:34:26,650
So maybe you have high speed packet capture, you've got an IDS, right?

378
00:34:26,650 --> 00:34:31,470
You've got some, you know, network detection and response solution, you want to feed all those.

379
00:34:31,470 --> 00:34:43,590
So nice thing is, you know, you just have one set of spans and taps, feed it into your aggregation switch, and then it can feed multiple solutions that are connected to it.

380
00:34:43,630 --> 00:34:46,330
And a couple things to add as well.

381
00:34:46,330 --> 00:34:54,890
So if you are thinking about taps at any sort of enterprise scale, you have to be thinking about span aggregation as well.

382
00:34:54,890 --> 00:34:59,330
Like, it is not an option to not consider that, not factor that into your deployment.

383
00:34:59,330 --> 00:35:07,850
Yeah, one thing we didn't mention about taps that maybe we should have, if you have a single 10 gig fiber tap, you might say, okay, I'm going to tap one link.

384
00:35:07,850 --> 00:35:11,270
So how many links do you actually have that feed your analysis solution?

385
00:35:11,270 --> 00:35:21,590
It's not one, it's actually two, because the way taps work is they feed you the ingress and the egress on that link, and they separate it out into two different links.

386
00:35:21,590 --> 00:35:27,070
And they do that because, as I mentioned before, it's a full duplex link, and taps want to do perfect preservation.

387
00:35:27,070 --> 00:35:31,910
They give you two outputs so that there is no packet loss at all possible.

388
00:35:32,330 --> 00:35:33,510
So Chris is exactly right.

389
00:35:33,510 --> 00:35:39,850
I mean, when you're talking taps, you can be talking a huge amount of links if you do them at any scale at all.

390
00:35:39,850 --> 00:35:44,170
One additional thing, we've mentioned ER-SPAN a couple of times.

391
00:35:44,170 --> 00:35:48,310
We've also mentioned VXLAN in the case of a Spineleaf deployment.

392
00:35:48,990 --> 00:36:00,450
SPAN aggregation solutions are capable a lot of times of doing decapsulation to get all of your traffic into whatever the base level traffic pattern that you want to analyze.

393
00:36:00,450 --> 00:36:04,930
So you want just the raw traffic that a host would see, for example.

394
00:36:04,930 --> 00:36:06,030
Great, fantastic.

395
00:36:06,030 --> 00:36:08,850
A lot of SPAN aggregation solutions can get it to that specific...

396
00:36:08,850 --> 00:36:22,490
And that's really important because if you feed, let's say Spineleaf encapsulated VXLAN traffic to an analysis solution and it's not decapsulated and that solution can't decapsulate it, it's not going to do anything for you.

397
00:36:23,450 --> 00:36:25,550
You're going to be like, what are these IP addresses?

398
00:36:25,750 --> 00:36:32,450
Because they're all going to be the underlay network and it's not going to be your actual host IP addresses.

399
00:36:33,510 --> 00:36:44,910
So some other things aggregation switches do, filtering traffic, they do deduplication, which we're going to talk about as well.

400
00:36:44,910 --> 00:36:46,970
So let's talk a little bit about filtering.

401
00:36:46,970 --> 00:36:52,750
So this is probably one of the most important things when you're building these networks out.

402
00:36:52,750 --> 00:36:56,850
So number one, packet analysis resources are not infinite, right?

403
00:36:56,850 --> 00:36:58,690
Nothing can accept infinite bandwidth.

404
00:36:59,010 --> 00:37:03,010
And writing packets to disks is also not infinite, right?

405
00:37:03,010 --> 00:37:04,390
You don't have an infinite disk space.

406
00:37:04,390 --> 00:37:10,890
In fact, if you have, let's say, 100 gigabits per second link, that will fill up one petabyte of disk per day approximately.

407
00:37:11,590 --> 00:37:16,130
And it's funny, when I meet with customers, they'll tell me, hey, I want 30 days of lookback.

408
00:37:16,130 --> 00:37:18,050
And I go, okay, what's your average throughput?

409
00:37:18,050 --> 00:37:29,150
And they might say something, I don't know, five gigabits, 10 gigabits, whatever it is, you can literally go on Google and type like 10 gigabits per second, if you type the words out, to terabytes per day.

410
00:37:29,190 --> 00:37:36,830
And then you can shock them and let them know like, oh, yeah, actually, you're going to need like 10 petabytes of disk if you want to keep that much data.

411
00:37:37,150 --> 00:37:41,830
And then of course, they're like, well, maybe we don't need to keep that.

412
00:37:41,830 --> 00:37:46,090
Some people can't dedicate whole data centers to storing PCAP.

413
00:37:46,090 --> 00:37:48,950
So it's just how much money you want to throw at the problem.

414
00:37:49,450 --> 00:37:55,910
Yeah, I mean, there's government agencies, right, that just have stacks and stacks and stacks of storage to do that.

415
00:37:55,910 --> 00:38:04,290
But , you know, even if some of the, you know, we've worked with some of the largest companies in the world, and, you know, they don't have infinite money, like, you know, any more than anyone else.

416
00:38:05,070 --> 00:38:07,650
So one of the things that can really help you, though, is filtering.

417
00:38:08,250 --> 00:38:16,670
Because there is a, you know, to use a non-technical term, if you're spending a bunch of traffic, there's a good chance you're going to get some junk in there that you don't really care about.

418
00:38:17,830 --> 00:38:26,510
You know, things like backup traffic, data replication traffic, that stuff, it's just not typically going to be very useful to look at.

419
00:38:26,510 --> 00:38:30,890
And it can consume a massive amount of bandwidth and disk space.

420
00:38:30,890 --> 00:38:40,130
Like, I have customers where I'd say 30-40% of their traffic in a 24-hour period is backup traffic, if I were to spend, you know, most of the traffic in their data center.

421
00:38:40,230 --> 00:38:42,770
So we're going to, if we get that in, we're going to filter it out.

422
00:38:42,910 --> 00:38:46,130
Some other good examples are going to be your VM infrastructure traffic.

423
00:38:46,130 --> 00:38:54,050
So if you use NFS, for example, to mount your disk that all your VMs use, you don't want to, you typically do not want to spend that.

424
00:38:54,050 --> 00:38:55,350
It's going to be a massive amount of traffic.

425
00:38:55,350 --> 00:38:57,610
Or if you use iSCSI, same thing.

426
00:38:57,910 --> 00:38:59,610
vMotion traffic is another one.

427
00:38:59,610 --> 00:39:03,710
It's, you know, it's a ton of traffic, and it's just generally not going to be that interesting.

428
00:39:03,970 --> 00:39:09,190
And then finally, telemetry data can also consume a ton of bandwidth.

429
00:39:09,190 --> 00:39:14,290
So syslog, ELF, Elastic, NetFlow, etc.

430
00:39:15,990 --> 00:39:21,670
Splunk replication, one of the chattier things that I've seen on the network.

431
00:39:21,690 --> 00:39:31,470
There's certain network links where I've seen Splunk replication in particular consuming 30 to 40% of the total available bandwidth.

432
00:39:31,470 --> 00:39:33,790
So, filter it out.

433
00:39:33,790 --> 00:39:34,230
Get it out.

434
00:39:34,230 --> 00:39:35,310
I don't care.

435
00:39:39,940 --> 00:39:43,040
Alright, now, what's the other big thing besides filtering?

436
00:39:43,840 --> 00:39:45,440
Duplicate traffic, right?

437
00:39:45,440 --> 00:39:47,780
And that's, how do we handle duplicate traffic?

438
00:39:47,780 --> 00:39:48,540
Well, there's a couple ways.

439
00:39:48,540 --> 00:39:50,420
One way is deduplication.

440
00:39:50,800 --> 00:39:57,980
So, when you do this, if you haven't, you know, done a lot of scanning and tapping before, you're going to get duplicate traffic.

441
00:39:57,980 --> 00:39:59,060
It's normal.

442
00:39:59,080 --> 00:40:00,380
Don't freak out.

443
00:40:01,220 --> 00:40:03,260
Expect to see some of it, right?

444
00:40:03,260 --> 00:40:06,880
The problem is when you get a lot of it, right?

445
00:40:06,880 --> 00:40:10,160
There's negative impacts of having tons and tons of duplicate traffic.

446
00:40:10,160 --> 00:40:18,080
Number one, you know, it's like, okay, if I have a duplicate of every single packet on my network, and I don't have any way to deduplicate it, it looks like I have twice as much traffic.

447
00:40:18,080 --> 00:40:20,220
It's going to waste a ton of lookback space.

448
00:40:22,040 --> 00:40:26,500
Let's say I bought some packet analysis solution that can handle up to 10 gig of traffic.

449
00:40:26,500 --> 00:40:32,880
Well, if I'm feeding it two copies of every packet, I'm literally wasting half of its capability, right?

450
00:40:32,920 --> 00:40:35,540
And it can also make manual analysis more difficult.

451
00:40:35,540 --> 00:40:41,460
You know, if any of you have ever opened up a Wireshark trace, and there's like six copies of every packet, it can be annoying to look at.

452
00:40:41,460 --> 00:40:46,120
I mean, there's ways to filter it out and stuff, but it can just make it more of a pain.

453
00:40:47,360 --> 00:40:49,280
So a couple of solutions here.

454
00:40:49,280 --> 00:40:58,720
So one is going to be deduplication, which is a feature essentially that's supported by some aggregation switches and some packet analysis solutions.

455
00:40:59,560 --> 00:41:05,540
And my general rule of thumb is that it works pretty well for lower levels of duplicate traffic, right?

456
00:41:05,540 --> 00:41:11,740
As I said, it's going to be normal to see some amount of duplicate traffic, and deduplication can handle that pretty well.

457
00:41:13,140 --> 00:41:19,700
However, when you start getting a lot of duplicate traffic, sometimes these deduplication solutions just fall over.

458
00:41:19,700 --> 00:41:23,120
And it's not because they're poorly designed or bad or any of that.

459
00:41:23,120 --> 00:41:27,120
It's because it's a hard problem to solve that requires a lot of resources.

460
00:41:27,120 --> 00:41:38,860
If I'm feeding 10 gigabits of traffic, essentially the way they work is they have a buffer, they have to buffer the traffic, they have a window that they look at, where they look, you know, if they see there's various heuristics you can use to identify duplicate packets.

461
00:41:38,900 --> 00:41:41,280
But within that window, they try to identify them, right?

462
00:41:41,280 --> 00:41:55,020
But again, when you're talking 10 gig, 20 gig, 40 gig of traffic, it just requires a ton of memory and resources to buffer that and look any more than a few milliseconds for duplicates, right?

463
00:41:55,020 --> 00:41:58,140
So what do you do when you have tons of duplicate traffic?

464
00:41:58,180 --> 00:42:05,760
Well, usually the reason is, is because you're spanning the same traffic or tapping the same traffic in a bunch of different places.

465
00:42:05,820 --> 00:42:08,960
And we're going to go back to the last slide, and we're going to filter that.

466
00:42:08,960 --> 00:42:18,080
So, in my experience, once you get above, you know, 5, 10, maybe 15% mean when you're up at like 50% duplicate and higher, you need to go filter.

467
00:42:18,760 --> 00:42:23,840
And you can filter at your where you're actually doing the scan, you can filter at the aggregation switch.

468
00:42:24,140 --> 00:42:35,120
And one of the things that I've found that works pretty well a lot of times is identifying, you know, let's say I have, you know, as an example, let's say I have a firewall that's passive and I span both sides.

469
00:42:35,120 --> 00:42:38,220
Well, I'm going to get the same package twice, right?

470
00:42:38,220 --> 00:42:46,100
Every time I want to filter that traffic out, essentially, all I have to do is figure out which MAC address is on which side of the firewall, right?

471
00:42:46,100 --> 00:42:50,780
And just filter the same traffic on both sides, you know, other than whatever gets blocked.

472
00:42:51,740 --> 00:42:57,060
And then I can use like a MAC address filter like that to pretty easily get rid of that traffic.

473
00:42:57,060 --> 00:43:06,680
So just as a general rule, I find that, you know, facing your filters, which by the way, most aggregation switches can do, facing your filters on MAC addresses is a pretty good approach.

474
00:43:31,020 --> 00:43:42,300
As much as I wish Amazon or Google would let me into their data centers and deploy taps and spans, I don't think they're so keen on doing that.

475
00:43:42,480 --> 00:43:59,680
And so what we've actually seen is the top three cloud vendors, AWS, GCP, and Azure have all announced, and in the case of AWS and GCP, made generally available span as a service.

476
00:43:59,960 --> 00:44:02,360
Which I think I made up.

477
00:44:02,620 --> 00:44:22,600
But it is the same thing as a span on prem, where you say, I have a specific set of cloud workloads, whether that's a specific instance, whether that's an ENI, whether that is an entire VPC, and you say, feed that into a specific monitoring interface,

478
00:44:22,600 --> 00:44:27,060
feed it into whatever analysis tool that I want to grab those packets from.

479
00:44:27,060 --> 00:44:35,020
In the case of Azure, they actually announced in 2018, it is still in private preview, just as a heads up.

480
00:44:35,020 --> 00:44:39,100
So for all the Azure customers out there, push on them.

481
00:44:39,100 --> 00:44:43,920
I know we are, because AWS and GCP can do it today.

482
00:44:43,920 --> 00:44:59,200
Yeah, so one thing I'll add here, that's pretty nice is, believe it or not, even though it's in the cloud, this stuff is actually quite a bit easier sometimes than on prem spans, because you literally like in AWS, you literally just say, okay, I have these,

483
00:44:59,200 --> 00:45:06,700
you know, 520 servers, I want to span or these 300 servers I want the traffic from, and you just can just turn on the traffic mirroring for them.

484
00:45:06,700 --> 00:45:14,480
There's no figuring out which VLAN they're on, or any of the things you have to deal with in an on prem data center, right?

485
00:45:14,480 --> 00:45:20,340
It's just they handle the implementation details under the covers, you just say, okay, I'm going to pay some money and get my traffic.

486
00:45:20,340 --> 00:45:22,220
It's all API based, right?

487
00:45:22,220 --> 00:45:37,660
So you go get BOTO, or pick your poison, and you have give me all of these tagged instances, give me their data, you don't have to coordinate with network teams, you just get the data as soon as you want.

488
00:45:40,080 --> 00:46:02,020
Containers, you're running Kubernetes, you're running Docker, they have this similar challenge to VMs in that there is traffic that never hits a physical network, or it's encapsulated in such a way where when it hits the physical network, it's hard to tell what the actual data is.

489
00:46:02,700 --> 00:46:14,460
So first thing, because it's similar challenge to VMs, you actually have similar solution, tap a physical network and validate what you are and are not seeing.

490
00:46:14,560 --> 00:46:23,160
Again, in our experience, you actually see a decent amount of the critical traffic, especially the inter service traffic.

491
00:46:23,160 --> 00:46:28,020
What you won't necessarily see is inter pod traffic.

492
00:46:28,480 --> 00:46:34,360
But we as an industry are still trying to decide on what the best way to actually capture this traffic is.

493
00:46:34,360 --> 00:46:40,920
You can deploy a daemon set, which is a specific pod that you run on every single node within your K8s cluster.

494
00:46:41,380 --> 00:46:47,820
And so you can hook into the node networking and use that as a source of data.

495
00:46:47,820 --> 00:47:02,600
You can also run a sidecar, which is a container that is run in the pod and use that to get inter pod intra node traffic as well.

496
00:47:02,760 --> 00:47:16,180
And some of the technologies that we're considering using the namespace hooking into the Docker zero interface and actually tapping those internal localhost interfaces as a way of getting to some of that data.

497
00:47:18,520 --> 00:47:26,160
So with that, some of our takeaways, both Pete and I have been working with packets for a long time.

498
00:47:26,160 --> 00:47:37,440
And I think they're one of the arguably best sources of data, not just because of the richness of the data that's there, but because of how accessible they are.

499
00:47:37,440 --> 00:47:45,740
You're not having to go find every host that you care about on the network, and instead can target the network as a whole.

500
00:47:47,140 --> 00:47:58,940
I don't know about you, Pete, but it seems like every network is kind of the same, but they're all unique in their own fun and interesting ways.

501
00:48:02,740 --> 00:48:15,240
But that doesn't mean that you can't use general principles as you start designing these architectures, and then modify them, use different strategies for different problems.

502
00:48:15,900 --> 00:48:20,360
And that's, there's many different strategies, depending on the network.

503
00:48:21,240 --> 00:48:24,760
And I would say these last two points are probably the most important.

504
00:48:24,760 --> 00:48:32,380
That identify critical assets step is by far the most important thing that you can do.

505
00:48:32,540 --> 00:48:36,180
We can set aside all the technology implementation, all of that.

506
00:48:37,340 --> 00:48:41,220
If you don't, you're identifying critical assets, that's essentially your problem statement.

507
00:48:41,220 --> 00:48:45,920
And if you don't have a good problem statement, it's going to be hard to solve that problem.

508
00:48:46,760 --> 00:48:51,400
And I also think that don't let the perfect be the enemy of good is a really important point, too.

509
00:48:51,400 --> 00:48:53,640
And I'm going to go back to that log analogy.

510
00:48:53,660 --> 00:48:55,760
Again, this is very similar.

511
00:48:55,760 --> 00:49:03,880
No one is going to capture every single log that is ever written by every single hosting device in a large environment.

512
00:49:03,880 --> 00:49:05,400
It's just not going to happen.

513
00:49:05,400 --> 00:49:06,420
And this is the same thing here.

514
00:49:06,420 --> 00:49:09,240
You are never going to capture every single packet.

515
00:49:09,240 --> 00:49:12,260
That doesn't mean that this can't be incredibly valuable.

516
00:49:12,260 --> 00:49:20,220
And what I've found is getting, let's say, 90% of what you want, it's not too hard.

517
00:49:20,220 --> 00:49:24,580
Then it's twice as hard to get the next 5%, to go to 95%.

518
00:49:24,580 --> 00:49:30,280
Then if you go to 97.5%, it's twice as hard as it was to get to 95%.

519
00:49:30,280 --> 00:49:33,040
And it's diminishing returns.

520
00:49:33,560 --> 00:49:38,600
So again, you want to identify those critical assets and don't let the perfect be the enemy of the good.

521
00:49:38,600 --> 00:49:50,120
As long as you have most of the stuff you need, most of the time, I think you can provide some incredibly valuable visibility and investigative capabilities to your organization.

522
00:49:50,980 --> 00:49:52,020
Perfect.

523
00:49:52,500 --> 00:49:55,500
So with that, thank you for your time.

524
00:49:55,500 --> 00:49:58,520
Thanks for coming to DEF CON Safe Mode.

525
00:49:59,040 --> 00:50:01,820
We're thankful to get the chance to speak to you guys.

526
00:50:01,820 --> 00:50:05,900
So we'll leave it open to any questions.