1
00:00:12,740 --> 00:00:14,360
Hello and welcome back.

2
00:00:14,520 --> 00:00:21,780
I hope you're looking forward to another fantastic talk here at the AppSec Village for DEF CON 29 in safe mode.

3
00:00:23,320 --> 00:00:26,240
It's important to remember you're 3-2-1.

4
00:00:26,240 --> 00:00:35,380
Even though we are separate from each other and socially distanced, your cohabitants will really appreciate that one in there.

5
00:00:35,580 --> 00:00:37,320
And just take care of yourselves.

6
00:00:37,320 --> 00:00:41,180
Get some good sleep, get some good food, make sure you take a shower.

7
00:00:41,180 --> 00:00:48,000
For our next talk, we've got David Waldrop talking about the DevOps and Agile Security Toolkit.

8
00:00:48,000 --> 00:00:54,860
After 25 years in the application development, David moved to information security just over seven years ago.

9
00:00:54,860 --> 00:00:58,140
His passion is helping developers write secure code.

10
00:00:58,140 --> 00:01:05,000
He's currently an information security advisor supporting the application development community at his employer.

11
00:01:05,000 --> 00:01:08,700
Let's put our hands together and welcome David Waldrop.

12
00:01:10,000 --> 00:01:12,760
Hi and welcome to the AppSec Village.

13
00:01:12,760 --> 00:01:17,100
I hope you're having a great DEF CON 2020, albeit in safe mode.

14
00:01:17,240 --> 00:01:24,220
My name is Dave and for the next 30-45 minutes, we're going to talk about the DevOps and Agile Security Toolkit.

15
00:01:24,220 --> 00:01:27,980
Now before I get started, I have to make the standard disclaimers.

16
00:01:27,980 --> 00:01:34,800
First, this presentation and any comments that I make are my own and do not necessarily represent those of my employer.

17
00:01:35,280 --> 00:01:37,800
Secondly, I will mention a few example products.

18
00:01:37,800 --> 00:01:40,120
Some are commercial, some are open source.

19
00:01:40,400 --> 00:01:43,100
These are not necessarily meant as recommendations.

20
00:01:43,260 --> 00:01:47,500
They are mentioned to serve as a starting point for your own investigation and review.

21
00:01:47,500 --> 00:01:51,620
These are things that I've run across in my journey, so take that as it is.

22
00:01:52,200 --> 00:02:00,180
Let's start with the two terms that are on everybody's buzzword bingo sheet these days, DevOps and Agile.

23
00:02:00,180 --> 00:02:07,780
We need to get definitions in place for the duration of this talk, so we have an idea that we're all dealing with the same playing field.

24
00:02:07,960 --> 00:02:09,040
Agile.

25
00:02:09,040 --> 00:02:13,540
Agile focuses on continuous iterative development and testing.

26
00:02:13,540 --> 00:02:19,820
Software is developed in smaller increments and then is integrated for final testing and delivery.

27
00:02:19,820 --> 00:02:25,540
Scrum, Kanban, and XP Extreme Programming are examples of Agile approaches.

28
00:02:25,700 --> 00:02:31,000
On the other hand, DevOps focuses on the collaboration between app dev and operations.

29
00:02:31,000 --> 00:02:37,340
The focus is on deploying code to production in a fast manner, usually automated.

30
00:02:38,920 --> 00:02:43,720
So you can see from the definitions that these are two separate concepts.

31
00:02:43,720 --> 00:02:47,760
However, many organizations tend to combine them into one initiative.

32
00:02:47,820 --> 00:02:54,080
Very, very rarely do you find a shop doing DevOps and not doing Agile, or vice versa.

33
00:02:54,860 --> 00:03:02,380
So no matter whether they start with DevOps or they start with Agile, the one piece that's usually late to the party is security.

34
00:03:02,520 --> 00:03:15,740
But no matter how late security is invited to the party, or whether we have to crash the party ourselves, we usually can still bring valuable resources without slowing things down, or at least without slowing them down a lot.

35
00:03:15,740 --> 00:03:19,620
And that's what we're going to talk about through the rest of this discussion.

36
00:03:19,620 --> 00:03:25,740
So for the past few years, I've worked with several different Agile and DevOps teams.

37
00:03:25,980 --> 00:03:38,400
My integration with this started as a pilot program where I was the information security representative that was assigned to help introduce Agile to our organization.

38
00:03:38,400 --> 00:03:46,100
More recently, I've implemented the information security integration with the Agile and DevOps teams across our organization.

39
00:03:46,500 --> 00:03:52,400
During this session, I'm going to share with you some things that really worked well and some things that just didn't seem to work at all.

40
00:03:52,460 --> 00:03:57,860
So hopefully you can take something away from this and apply it to your own organization.

41
00:03:59,760 --> 00:04:04,540
This particular slide will give you the map of what we're going to talk about for the rest of this discussion.

42
00:04:04,600 --> 00:04:06,840
We're going to talk about Agile staffing.

43
00:04:06,840 --> 00:04:09,600
We're going to talk about the eight simple questions.

44
00:04:09,600 --> 00:04:16,720
We're going to talk about the security champions program, and then go into developer training, which is my favorite part of my job.

45
00:04:17,460 --> 00:04:20,200
Security in the software development life cycle.

46
00:04:20,340 --> 00:04:23,240
And then we're going to hit the hat trick of securing your code.

47
00:04:23,240 --> 00:04:27,560
Static code analysis, dynamic code analysis, and open source analysis.

48
00:04:27,620 --> 00:04:31,660
And then finally, we're going to wrap things up by automating the heck out of it.

49
00:04:32,240 --> 00:04:34,680
So let's start with Agile staffing.

50
00:04:35,660 --> 00:04:38,900
Sherman set the way back machine from when I started with Agile.

51
00:04:39,260 --> 00:04:44,820
One information security person was assigned to one Agile team.

52
00:04:44,820 --> 00:04:55,360
That security person would be integrated into that team and attend every ceremony, every meeting, everything, every day.

53
00:04:56,000 --> 00:04:57,520
That didn't work.

54
00:04:57,520 --> 00:04:59,740
There are a couple of reasons I think that that didn't work.

55
00:04:59,740 --> 00:05:02,700
First of all, Agile teams tend to multiply.

56
00:05:02,880 --> 00:05:10,800
You can go from a proof of concept with one team to having close to 30 teams in less than two years.

57
00:05:10,820 --> 00:05:19,340
Most organizations do not have the information security bandwidth to support that type of staffing model.

58
00:05:19,660 --> 00:05:26,500
In addition, the scrum masters tend to treat every member on the team as though they can perform all tasks.

59
00:05:26,500 --> 00:05:29,700
And when you're dealing with a team of all developers, that makes perfect sense.

60
00:05:29,700 --> 00:05:43,540
This model tends to fall apart though when you put information security, QA, middleware, any other infrastructure or shared services team as a part of the Agile team.

61
00:05:43,560 --> 00:05:52,300
What happens is in the Agile process, their hours count against your available development hours.

62
00:05:52,540 --> 00:06:02,240
But the infrastructure people, the information security people do not have the skills or the capabilities to do some of the development.

63
00:06:02,240 --> 00:06:05,360
So you kind of see that there's a mismatch.

64
00:06:05,360 --> 00:06:07,660
You're given hours, but you're not allowed to use them.

65
00:06:07,660 --> 00:06:11,300
So it caused a lot of problems for some of our scrum masters early on.

66
00:06:11,300 --> 00:06:13,680
Again, we realized this was not the way to do it.

67
00:06:13,680 --> 00:06:20,620
So I ended up coming up with a two-tiered approach for trying to integrate information security into the Agile team.

68
00:06:21,180 --> 00:06:35,520
First, at a high level, at the enterprise level, we will have an architect, an information security architect, and they will inject into those high-level projects and basically set up guardrails and guidelines.

69
00:06:35,520 --> 00:06:48,500
So as projects are being devised at the visionary level, they can jump in and try to help guide them into making more secure decisions at that high level.

70
00:06:48,580 --> 00:06:56,180
Now when you get down into the weeds though, each Agile team does need to have somebody they can rely upon for information security.

71
00:06:56,700 --> 00:07:06,740
And so what we ended up doing is we created named resources within the information security team that are assigned to Agile teams.

72
00:07:06,740 --> 00:07:08,920
Now it's not like the previous model.

73
00:07:08,920 --> 00:07:12,960
You're not assigned, you're not sitting with them, you're not assigned to go to every meeting.

74
00:07:12,960 --> 00:07:27,120
We're going to talk about how we scaled that back, but we ended up finding a way where based on where information security can add the most value, we attend those ceremonies and only those ceremonies.

75
00:07:27,320 --> 00:07:35,040
So at the end of the day, each information security representative can support between six and eight different Agile teams.

76
00:07:35,040 --> 00:07:37,680
That is a model that is certainly sustainable.

77
00:07:38,460 --> 00:07:40,200
So what is a little different?

78
00:07:40,200 --> 00:07:48,880
Well first of all, we make sure that every Agile team has a named resource that is responsible for information security for them.

79
00:07:48,880 --> 00:07:50,840
So this gives them a named contact.

80
00:07:50,840 --> 00:07:57,720
If they ever need something, they know exactly who to call and where to start that quest.

81
00:07:58,320 --> 00:08:04,420
Secondly, we attend, or the Agile reps attend, the backlog refinement sessions.

82
00:08:04,420 --> 00:08:12,540
These are meetings that happen once every two weeks before the next sprint, and it defines what they're going to work on in that next sprint.

83
00:08:12,540 --> 00:08:23,100
It's a perfect opportunity to jump in and say, you're planning this work, we need to think about blah blah blah from an information security standpoint.

84
00:08:23,100 --> 00:08:27,480
We can get in just ahead of when development starts.

85
00:08:27,480 --> 00:08:33,180
It's a perfect time to bring up security issues, make recommendations, and answer questions.

86
00:08:33,180 --> 00:08:46,320
And then finally, we attend the quarterly planning increment meetings where the vision is cast for the next quarter, and we can make sure that we're on their radar for things that we think we need to be a part of.

87
00:08:47,540 --> 00:08:50,400
So this is kind of a high-level diagram.

88
00:08:50,400 --> 00:08:54,400
The purple arrow at the top is the information security architect.

89
00:08:54,560 --> 00:09:01,600
That particular person, as you can see from the diagram, sets guardrails for enterprise planning projects.

90
00:09:01,640 --> 00:09:10,420
All of the red arrows are the different places where our individual security representatives can or may inject into the Agile teams.

91
00:09:10,420 --> 00:09:16,360
We don't inject in all those places, but there are places where we can, and we give them that option to call us if they're needed.

92
00:09:18,280 --> 00:09:22,480
So let's move on to what I like to call the eight simple questions.

93
00:09:22,600 --> 00:09:32,200
This particular concept came up as a result of assigning our Agile reps to their teams.

94
00:09:32,800 --> 00:09:38,620
We found that the leadership within the Agile teams really wasn't sure when to call us.

95
00:09:38,840 --> 00:09:52,360
And since we weren't sitting there in every meeting, there was a possibility that something could come up that might fall off the radar, that we might not get contacted for, that they may not even think is important.

96
00:09:52,360 --> 00:10:01,200
So we decided that we need to give clear guidelines to the scrum masters and product owners as to when they should include information security.

97
00:10:01,520 --> 00:10:05,220
That's how I created the eight simple questions.

98
00:10:05,560 --> 00:10:18,540
So basically these eight questions are a guideline for that leadership team, and if they answer yes to any of them, they need to make sure to include their information security representative in those discussions.

99
00:10:19,680 --> 00:10:23,000
These are the questions as I currently have them.

100
00:10:23,000 --> 00:10:26,680
We're going to talk about that in just a bit because they change over time.

101
00:10:26,680 --> 00:10:28,920
These are not one and done kind of things.

102
00:10:29,200 --> 00:10:31,260
Am I dealing with any sensitive information?

103
00:10:31,260 --> 00:10:35,800
Now information that is sensitive, it's going to mean something different for each one of our organizations.

104
00:10:35,800 --> 00:10:47,340
If you're working in healthcare, if you're working with banking information, if you're working with retail with PCI information, you will be able to define what sensitive information means for your organization.

105
00:10:47,560 --> 00:10:50,900
Am I sending or receiving data outside of the company?

106
00:10:51,080 --> 00:10:53,200
Is there a new vendor involved?

107
00:10:53,200 --> 00:10:58,060
Am I going to introduce new software systems or libraries, even open source?

108
00:10:58,060 --> 00:11:00,400
And remember open source because we're going to talk about that.

109
00:11:00,400 --> 00:11:01,980
It's a unique challenge.

110
00:11:02,000 --> 00:11:07,480
Do I need access to secrets such as passwords, keys, or certificates?

111
00:11:07,620 --> 00:11:09,900
Will I need the ability to use single sign-on?

112
00:11:09,900 --> 00:11:11,960
Am I going to leverage the cloud?

113
00:11:11,960 --> 00:11:14,480
Will I need updated access roles?

114
00:11:14,800 --> 00:11:25,100
So these eight questions are what we are currently using as a guideline for the interaction between the Agile team and their information security representative.

115
00:11:25,480 --> 00:11:29,560
As I mentioned, the exact questions are going to differ on your organization.

116
00:11:29,640 --> 00:11:33,380
The questions themselves will actually change over time.

117
00:11:33,380 --> 00:11:40,400
I started I think with seven questions and then we had several issues come up with access roles and who to ask and blah blah blah.

118
00:11:40,400 --> 00:11:42,420
So we added a question for that too.

119
00:11:42,840 --> 00:11:48,320
So don't be afraid to take a stab at it and massage it and change it.

120
00:11:49,000 --> 00:11:56,660
That's the thing about Agile and DevOps, it's changing all the time and we as information security people need to be able to change our stuff as well.

121
00:11:56,860 --> 00:11:58,240
So the basic idea is this.

122
00:11:58,240 --> 00:12:09,200
Create a list of questions, usually 10 or less, something that's going to fit easily on a PowerPoint slide or a single piece of paper so they can take it to meetings with them.

123
00:12:09,480 --> 00:12:13,400
Give that to the Agile team leadership and they have a guideline.

124
00:12:13,400 --> 00:12:18,680
They now have a concrete way of knowing when they need to work with information security.

125
00:12:19,240 --> 00:12:29,980
So we found another advantage to this approach as we onboarded additional information security people to become representatives for the Agile teams.

126
00:12:29,980 --> 00:12:36,400
A lot of these folks were pretty new to the organization and they didn't really know when they needed to inject themselves into their teams.

127
00:12:36,400 --> 00:12:40,400
They didn't know what they needed to listen for in the backlog refinement sessions.

128
00:12:40,400 --> 00:12:44,460
So we used these 8 simple questions on our own people.

129
00:12:44,460 --> 00:12:51,240
It gave them guidelines and hints of what to listen for when they're working with their Agile teams.

130
00:12:51,240 --> 00:12:56,420
So this kind of became a win-win on both sides of the coin.

131
00:12:57,480 --> 00:13:01,420
So I think this was a great tool that we've been able to use.

132
00:13:02,840 --> 00:13:07,040
Another tool that I honestly have to thank Jim Manico for.

133
00:13:07,040 --> 00:13:08,580
Jim, if you're out there, thank you for this.

134
00:13:08,580 --> 00:13:13,700
This was a recommendation he made to me after one of our training sessions.

135
00:13:13,760 --> 00:13:16,040
And that is the Security Champions Program.

136
00:13:17,080 --> 00:13:20,860
A Security Champions Program is a team of application developers.

137
00:13:20,860 --> 00:13:23,600
We get one from each Agile team.

138
00:13:23,780 --> 00:13:31,460
And these developers have expressed an interest in information security, or secure coding, or just security in general.

139
00:13:31,460 --> 00:13:35,460
This is a voluntary program.

140
00:13:35,700 --> 00:13:41,940
The members of the team volunteer to come to the Security Champions Program.

141
00:13:42,040 --> 00:13:44,060
And we typically meet quarterly.

142
00:13:44,060 --> 00:13:47,760
I provide them lunch when we're in the office, thanks 2020.

143
00:13:47,760 --> 00:13:51,860
I look forward to getting back to the office and having lunch with these guys and gals.

144
00:13:51,860 --> 00:13:53,200
They're a great team.

145
00:13:53,440 --> 00:13:55,680
So the team has this following charter.

146
00:13:55,680 --> 00:13:59,980
We assist the Agile teams in terms of application security.

147
00:14:00,000 --> 00:14:06,920
We serve as a communication conduit between the Agile teams and information security.

148
00:14:06,920 --> 00:14:12,140
And then we meet quarterly to share experiences, we ask questions, we share knowledge.

149
00:14:12,180 --> 00:14:14,720
It has had some incredible benefits.

150
00:14:14,720 --> 00:14:20,800
First of all, security has a relationship with at least one person on every Agile team.

151
00:14:20,840 --> 00:14:22,480
I'm not saying we have them all.

152
00:14:22,480 --> 00:14:29,460
What I am saying is, sometimes we hear about things long before it percolates through the management chain.

153
00:14:29,460 --> 00:14:34,020
I might get a phone call about, hey, we're thinking about using this open source library.

154
00:14:34,020 --> 00:14:35,020
Is that cool?

155
00:14:35,020 --> 00:14:36,440
We take a quick look at it.

156
00:14:36,440 --> 00:14:37,280
We approve it.

157
00:14:37,280 --> 00:14:42,260
By the time the formal approver gets into my hands, we've already done it and they're coding.

158
00:14:42,260 --> 00:14:45,940
So it gives us a great communication conduit.

159
00:14:46,380 --> 00:14:49,580
And that helps me make things faster.

160
00:14:50,380 --> 00:14:52,500
I may have not mentioned this before.

161
00:14:52,500 --> 00:14:53,700
Hopefully, I have.

162
00:14:53,780 --> 00:14:55,480
Relationship is key.

163
00:14:55,480 --> 00:15:00,120
You're going to hear about relationship throughout this entire discussion.

164
00:15:01,160 --> 00:15:04,020
We build relationships with those Agile teams.

165
00:15:04,020 --> 00:15:06,940
It starts with having an assigned rep.

166
00:15:07,660 --> 00:15:14,160
It moves into a different relationship level when we have a security security champions team.

167
00:15:14,380 --> 00:15:18,660
Secure coding training takes it to yet another level.

168
00:15:18,660 --> 00:15:23,040
And you'll see why in just a few minutes as we talk about the developer training.

169
00:15:23,040 --> 00:15:24,820
But it's about building relationships.

170
00:15:24,820 --> 00:15:29,500
It's about having fun while we're doing the right thing for our organizations.

171
00:15:30,000 --> 00:15:33,160
So, so far, the team has done a lot of really cool things.

172
00:15:33,160 --> 00:15:38,640
We've reviewed vendors and products, both on the development side and on the information security side.

173
00:15:38,640 --> 00:15:48,400
We've selected our developer training programs for the last two years based on the input from the security champions team members.

174
00:15:48,660 --> 00:15:51,160
We've done a bunch of POCs together.

175
00:15:51,160 --> 00:15:52,820
We've cross consulted.

176
00:15:52,820 --> 00:16:09,280
Now, this was kind of an unexpected benefit is where if I had a team that was very well versed in one technology and they mentioned that in a security champions lunch, another team might reach out and say, hey, we're about to do that.

177
00:16:09,280 --> 00:16:10,840
Can we sit down and talk?

178
00:16:10,840 --> 00:16:19,040
So, it actually starts to cross pollinate some security knowledge and some technical knowledge between different application teams.

179
00:16:19,160 --> 00:16:25,140
And then finally, it served as a candidate group for application security openings within the InfoSec team.

180
00:16:26,160 --> 00:16:28,560
The software development life cycle.

181
00:16:28,560 --> 00:16:37,920
So, as we started to roll out security, a lot of people have asked us, how does this fit into our SDLC?

182
00:16:38,040 --> 00:16:46,060
So, I ended up creating a plan that maps information security phases to the software development life cycle as it currently exists.

183
00:16:46,060 --> 00:16:56,780
It is not a perfect fit, but I think it is a good representation of how information security can bring value to each step of the SDLC.

184
00:16:57,320 --> 00:17:01,920
So, this is what I like to call my dark side of the moon diagram.

185
00:17:01,920 --> 00:17:05,860
Those of you old enough to remember, probably recognize the album cover.

186
00:17:05,880 --> 00:17:19,360
So, if you take a look on the right, we have the software development life cycle from concept into design, up into coding testing and QA, QA and deployment, and then post-deployment at the very top.

187
00:17:19,360 --> 00:17:33,040
Now, if you look on the left side, you'll see that it matches up pretty closely with security knowledge, security design, secure coding techniques, security testing, and then security response.

188
00:17:33,560 --> 00:17:48,840
The each individual layer of the pyramid, I think, are deliverables that security can bring that help take that concept on the left and apply it to the SDLC phase on the right.

189
00:17:48,840 --> 00:17:50,820
And we're going to walk through those real quick.

190
00:17:50,820 --> 00:17:53,480
Training is the base of the pyramid.

191
00:17:53,480 --> 00:17:57,500
I think training is foundational to everything above it.

192
00:17:57,500 --> 00:18:04,620
If you don't have a solid foundation of training for your application developers, everything else is at risk.

193
00:18:04,680 --> 00:18:12,920
So, this particular approach, we're going to talk about in depth, again, because it's my favorite part of what I do.

194
00:18:12,920 --> 00:18:21,520
It includes things like lunch and learns, outside speakers, conferences like you're attending right now, video training, and things like that.

195
00:18:21,680 --> 00:18:24,260
The next level up is threat modeling.

196
00:18:24,260 --> 00:18:36,920
Threat modeling, the application team, and InfoSec, we sit together, we work together to examine the application design, the system flows, whatever they can bring to us that they have available in that design phase.

197
00:18:37,040 --> 00:18:45,440
We look, we try to work together and identify potential vulnerable vectors and try to figure out a way that we can mitigate them.

198
00:18:46,600 --> 00:18:51,620
Now, threat modeling, there are books upon books written about threat modeling.

199
00:18:51,680 --> 00:18:56,480
And there are people in my organization on my InfoSec team that are much better at it than I am.

200
00:18:56,480 --> 00:19:03,180
And they can actually create incredible documents and spend hours and days and days and days doing this.

201
00:19:03,180 --> 00:19:04,900
I think you need to start out small.

202
00:19:04,900 --> 00:19:06,900
I think it's about baby steps at first.

203
00:19:06,900 --> 00:19:08,620
Start with a discussion.

204
00:19:08,620 --> 00:19:14,860
This can be a one or two hour discussion where you have those deliverables from the design phase, and you whiteboard them.

205
00:19:14,860 --> 00:19:19,200
And you come out of there with to-dos, or questions, or things to research.

206
00:19:19,200 --> 00:19:27,820
It doesn't have to be this big, heavyweight, overbearing approach to threat modeling at first.

207
00:19:27,820 --> 00:19:29,520
It needs to be a conversation.

208
00:19:29,520 --> 00:19:31,240
That gets back to relationships.

209
00:19:31,240 --> 00:19:36,020
You need to have those relationships in place so you can have those conversations.

210
00:19:36,560 --> 00:19:39,440
The third level up is static code analysis.

211
00:19:39,440 --> 00:19:43,780
We're going to talk about this in much more detail later.

212
00:19:43,780 --> 00:19:49,260
But most tools are designed to identify potential vulnerabilities within the code.

213
00:19:49,260 --> 00:19:55,840
Now, static code analysis actually looks at the code and looks for patterns within the code without running it.

214
00:19:55,840 --> 00:19:58,960
Now, we're going to see a little bit later why that's important.

215
00:19:59,540 --> 00:20:01,340
I think I went the wrong way.

216
00:20:02,260 --> 00:20:03,520
There we go.

217
00:20:03,620 --> 00:20:06,460
Dynamic code analysis is the next layer up.

218
00:20:06,460 --> 00:20:10,300
And that is very similar to the previous layer, which was static code analysis.

219
00:20:10,300 --> 00:20:16,400
But instead of just looking at the code, the code is actually executed and tested for vulnerabilities.

220
00:20:16,640 --> 00:20:23,660
So, like I said, these two pieces are very, very similar.

221
00:20:23,660 --> 00:20:25,700
One is looking at patterns with the code.

222
00:20:25,700 --> 00:20:30,000
The other is actually exercising the code and looking for issues.

223
00:20:30,000 --> 00:20:31,640
The next is pin testing.

224
00:20:31,640 --> 00:20:38,040
Pin testing can be performed by your internal team if you have the skills and resources, or you can use an external firm.

225
00:20:38,440 --> 00:20:43,960
This is usually a greater scope than dynamic testing.

226
00:20:44,520 --> 00:20:53,520
It will actually look at things not just in your application, but system-wide for vulnerabilities and attack vectors.

227
00:20:53,520 --> 00:20:55,320
And usually there's a person involved.

228
00:20:55,320 --> 00:20:57,340
It's not just an automated exercise.

229
00:20:57,340 --> 00:21:02,860
We're going to see a little bit more when we talk about dynamic testing, the difference between that and pin testing.

230
00:21:02,860 --> 00:21:13,880
And then finally, when the code is in deployment, you need to have a vulnerability management process that you can classify, assign, and track identified issues.

231
00:21:15,700 --> 00:21:26,680
Right now, InfoSec meets regularly with a representative from each application team to review these and to make sure that they're working and they're on track.

232
00:21:26,680 --> 00:21:31,500
Again, those relationships are absolutely key to making this work.

233
00:21:31,980 --> 00:21:39,960
So now we are getting to what I consider my absolute favorite part of my job, and that is developer training.

234
00:21:40,140 --> 00:21:50,540
If you remember back to that pyramid we just looked at, this is the foundation for everything that InfoSec does with AppDev.

235
00:21:50,540 --> 00:22:06,780
When I came in to Information Security about seven years ago, I started a developer training program because I realized that the developers really are your last best hope for defense when it comes to application security.

236
00:22:06,780 --> 00:22:16,340
If your application is written correctly, then everything you throw in front of it, application firewalls, everything like that, that's gravy.

237
00:22:16,340 --> 00:22:23,460
That's nice, but you've got a really tight application, and that's where real security takes place.

238
00:22:23,460 --> 00:22:32,700
The problem is, I was a developer for roughly 25 years before coming to Information Security, and nobody taught this stuff.

239
00:22:32,700 --> 00:22:39,920
This is stuff you learned on your own if you had an interest in it, and it wasn't universally available to all developers.

240
00:22:40,120 --> 00:22:43,020
So I started small.

241
00:22:43,020 --> 00:22:45,240
I started with Lunch and Learns.

242
00:22:45,600 --> 00:22:53,060
It was really something that, hey, I think this is important, let's try it, let's see if it works.

243
00:22:53,060 --> 00:22:54,240
And it really did.

244
00:22:54,240 --> 00:22:57,220
We would end up having Lunch and Learns about every other month.

245
00:22:58,060 --> 00:23:00,240
I would create a class on a specific topic.

246
00:23:00,240 --> 00:23:03,060
My first one was cross-site scripting, a lot of fun.

247
00:23:03,320 --> 00:23:10,500
It's neat when you can show people cross-site scripting examples, even if you just run WebGoat or something like that.

248
00:23:10,580 --> 00:23:11,760
They get excited.

249
00:23:11,760 --> 00:23:15,400
They kind of dig it, and they start trying it on their own stuff.

250
00:23:16,000 --> 00:23:22,840
The content usually lasts about 40 or 45 minutes, a lot like this session, and then we leave 10 to 15 minutes for open discussion and questions.

251
00:23:22,880 --> 00:23:25,440
Now the key here is, make it fun.

252
00:23:25,440 --> 00:23:28,160
It has to be fun.

253
00:23:28,360 --> 00:23:35,340
I do things like, I'll go out and I'll buy cookies or some kind of treat, so we can all sit around, eat something together.

254
00:23:35,580 --> 00:23:39,500
There's usually some kind of door prize or door prizes, and they don't have to be expensive.

255
00:23:39,500 --> 00:23:43,720
You can just go down to five below and get a couple of wireless Bluetooth speakers.

256
00:23:44,200 --> 00:23:45,080
And people dig it.

257
00:23:45,080 --> 00:23:46,340
It's a lot of fun.

258
00:23:46,700 --> 00:23:52,600
Again, making it fun is absolutely key to a successful Lunch and Learn program.

259
00:23:53,180 --> 00:23:58,440
So once we got that up and running, I decided we needed to take this up a notch.

260
00:23:58,920 --> 00:24:11,860
There is a level of training that I couldn't provide that I think our developers really, and in order to do that, I knew that I had to reach out to industry experts.

261
00:24:12,940 --> 00:24:26,660
And what we ended up doing is, we were able to get commitment from management, both on the IT development side and on InfoSec side, that we could do one or two days where most, if not all, of the developers will attend a training event.

262
00:24:26,780 --> 00:24:27,980
Now here's a hint.

263
00:24:27,980 --> 00:24:34,280
If you are going to try this and you're doing Agile, put it in your Innovation Sprint.

264
00:24:34,280 --> 00:24:52,380
If you put it in the Innovation Sprint, which is usually the last sprint of the quarter, and the developers have a little more leeway of what they work on during the Innovation Sprint, you'll very likely get more participation than if you step on a middle of a sprint where there's actual deliverables due.

265
00:24:52,380 --> 00:24:53,700
So keep that in mind.

266
00:24:53,800 --> 00:24:56,060
I was really, really lucky.

267
00:24:56,060 --> 00:25:06,140
My very first interaction with this program, I was able to secure Jim Manico, and he came in and made an incredible impression on our development teams.

268
00:25:06,140 --> 00:25:08,260
Jim's been back, I think, three times now.

269
00:25:08,260 --> 00:25:09,600
Always a pleasure.

270
00:25:10,420 --> 00:25:12,020
Rich Mogul's been by twice.

271
00:25:12,020 --> 00:25:14,580
We've had a recent one with Ron Paris.

272
00:25:14,580 --> 00:25:14,920
Dr.

273
00:25:15,140 --> 00:25:19,880
Philippe Des Riques, phenomenal instructor.

274
00:25:20,120 --> 00:25:30,240
I just took his OAuth class, so if you're looking for OAuth, looking forward to that.

275
00:25:30,240 --> 00:25:35,840
The reason I actually want to call these gentlemen out by name is I think it's important to be grateful.

276
00:25:35,840 --> 00:25:41,140
I am very grateful for everything they've done for me and my development teams.

277
00:25:41,340 --> 00:25:47,560
And it's important to be grateful to those who help you along the way on this journey, because none of us make it alone.

278
00:25:47,560 --> 00:25:49,520
So gentlemen, thank you very much.

279
00:25:49,520 --> 00:25:58,860
If you see this or hear it later, your training, your mentorship has meant a lot to me and my development teams.

280
00:25:59,280 --> 00:26:08,920
So one cool aside is, I mentioned this earlier, the Security Champions team has been involved in the selection of the instructor for the last two years.

281
00:26:08,920 --> 00:26:14,140
This gives your application development teams a voice in the training.

282
00:26:14,160 --> 00:26:18,140
And we've seen that participation has actually increased.

283
00:26:18,140 --> 00:26:35,100
We had great participation before then, but when the developers actually had a voice in picking who's coming in and what they're going to talk about, we're finding that it's much more applicable to what's on their roadmap.

284
00:26:35,140 --> 00:26:38,900
It's much more applicable to the skills they're hoping to get.

285
00:26:39,140 --> 00:26:43,840
And participation goes beyond what I could have ever hoped for.

286
00:26:43,840 --> 00:26:52,940
So again, we're starting to see some of these initiatives are starting to weave together and starting to build kind of a program.

287
00:26:52,940 --> 00:26:53,940
It's kind of cool.

288
00:26:54,420 --> 00:26:59,240
So this next one, I really wish I was doing this right now.

289
00:26:59,420 --> 00:27:01,360
Take a developer to DEF CON.

290
00:27:01,360 --> 00:27:08,480
We had management agree to provide scholarships to developers that were selected by the information security team.

291
00:27:08,700 --> 00:27:15,020
So we would give them the money to buy their badges so they wouldn't have to go and get money out of their own bank account.

292
00:27:16,400 --> 00:27:22,380
So each year I've been able to bring up to three developers to DEF CON.

293
00:27:22,400 --> 00:27:24,360
That's been awesome.

294
00:27:24,360 --> 00:27:28,700
It has been just a phenomenal experience.

295
00:27:28,780 --> 00:27:34,700
They've gained a ton of knowledge and a ton of awareness by DEF CON.

296
00:27:35,320 --> 00:27:43,760
One of the things that I found really interesting was I really advised them not to restrict their sessions to things that they think will apply to their job.

297
00:27:43,780 --> 00:27:50,240
I did that my very first DEF CON and I wasn't really sure I was going to come back, to be honest with you.

298
00:27:50,600 --> 00:27:55,000
The next year, one of my co-workers strongly encouraged me to go.

299
00:27:55,000 --> 00:27:56,020
Thank you, Barry.

300
00:27:57,740 --> 00:28:04,400
And encouraged me to go and find a few fun things that just had nothing to do with my immediate job.

301
00:28:04,400 --> 00:28:14,780
The interesting thing is they ended up having more to do with the things that were coming down the road in my job than I ever could have imagined.

302
00:28:14,780 --> 00:28:21,960
So I fully encourage each one of these developers that I bring to DEF CON to experience the full DEF CON experience.

303
00:28:21,960 --> 00:28:23,480
Go to all the sessions.

304
00:28:23,480 --> 00:28:26,880
Try some of the capture the flags.

305
00:28:26,880 --> 00:28:28,720
Try whatever you want.

306
00:28:28,720 --> 00:28:37,640
Have a good time because it's really important for them to be immersed in the DEF CON experience.

307
00:28:37,720 --> 00:28:41,380
So like I said before, they come back with a changed perspective.

308
00:28:41,380 --> 00:28:48,800
It's a lot like that scene in The Matrix where that thing crawls out of Neo and he goes, my god, that thing is real?

309
00:28:49,100 --> 00:29:00,360
They understand now that the security is not just something that we're teaching and preaching about within InfoSec, that this stuff has real world application.

310
00:29:00,480 --> 00:29:07,700
And it's that realization that makes them a better developer, a more secure developer going forward.

311
00:29:09,400 --> 00:29:21,660
We've seen a trend where a lot of the folks that we've taken, actually, I think all but one of the developers that we've taken to DEF CON have eventually become security champions members.

312
00:29:21,660 --> 00:29:36,160
And then more recently, in the last couple of months, one of those developers, I think he was the first one I took to DEF CON with us, ended up joining the Information Security team on a full-time manner.

313
00:29:36,200 --> 00:29:38,940
And Shane, if you're listening, welcome.

314
00:29:39,020 --> 00:29:41,540
I'm really glad that you're part of the team.

315
00:29:41,960 --> 00:29:45,800
So there's a couple of other fun activities that we do in developer training.

316
00:29:45,800 --> 00:29:47,720
Again, got to keep it fun.

317
00:29:47,780 --> 00:29:49,240
Everything has to be fun.

318
00:29:49,240 --> 00:29:51,900
You learn better when you're having a good time.

319
00:29:51,900 --> 00:29:53,440
I look at DEF CON.

320
00:29:53,440 --> 00:29:57,000
We're all having a great time and we're learning stuff.

321
00:29:57,000 --> 00:30:02,260
So a couple of DEF CONs ago, for reasons of work, I couldn't make it.

322
00:30:02,260 --> 00:30:07,520
It was the only DEF CON since coming over to Information Security that I was not able to go to.

323
00:30:07,520 --> 00:30:10,160
And I must admit I was bummed.

324
00:30:10,160 --> 00:30:15,300
So what I decided to do was I was going to have my own little DEF CON.

325
00:30:15,340 --> 00:30:21,720
And we decided that, okay, let's just take people along for the ride.

326
00:30:21,720 --> 00:30:29,820
So we identified five videos from previous DEF CONs and we had a lunch and learn for five days that week where we watched it.

327
00:30:29,820 --> 00:30:31,220
We talked about it.

328
00:30:31,220 --> 00:30:35,880
We had drawings for DEF CON related swag or items or giveaways.

329
00:30:35,880 --> 00:30:38,200
We just kind of had a good time.

330
00:30:38,200 --> 00:30:41,980
It was kind of built up as, you know, hey, take a break.

331
00:30:41,980 --> 00:30:44,020
Come bring your lunch.

332
00:30:44,020 --> 00:30:44,960
Let's hang out.

333
00:30:44,960 --> 00:30:46,760
Watch something from DEF CON.

334
00:30:46,840 --> 00:30:47,720
Talk about it.

335
00:30:47,720 --> 00:30:49,340
You might win a t-shirt.

336
00:30:49,340 --> 00:30:51,840
You might win some stickers, whatever.

337
00:30:52,000 --> 00:30:53,740
And people really enjoyed it.

338
00:30:53,740 --> 00:30:55,180
It was a lot of fun.

339
00:30:55,180 --> 00:31:03,140
As a matter of fact, I'm probably going to do one this fall for the folks that didn't get to enjoy what you're doing right now, what you're enjoying right now.

340
00:31:03,400 --> 00:31:06,300
And then finally, I have an annual thank you lunch and learn.

341
00:31:06,480 --> 00:31:10,820
This was a tradition I started the first year we did lunch and learns.

342
00:31:10,940 --> 00:31:13,660
It's always the last lunch and learn of the year.

343
00:31:13,660 --> 00:31:17,740
And I typically will go out and I'll purchase quite a few giveaways.

344
00:31:17,740 --> 00:31:19,100
They don't have to be expensive.

345
00:31:19,420 --> 00:31:24,560
We do things for, you know, like say the Bluetooth speakers, boxes of candy.

346
00:31:24,560 --> 00:31:33,420
We've done some Christmas mug sets, you know, just weird things, but things that are kind of fun.

347
00:31:33,560 --> 00:31:43,300
And we do a shorter lunch and learn followed by a session where we just eat some good food and we have some drawings.

348
00:31:43,300 --> 00:31:52,120
And I have an opportunity to thank my development community for the support of the lunch and learn program and for the support they've given to information security.

349
00:31:52,260 --> 00:31:55,480
I think it's important to say thank you.

350
00:31:55,480 --> 00:31:59,720
And this has been just an incredibly popular lunch and learn every year.

351
00:31:59,720 --> 00:32:03,260
It's a lot of fun and I highly recommend it.

352
00:32:03,260 --> 00:32:08,840
So here are the keys to a successful training program.

353
00:32:08,840 --> 00:32:14,220
Make it applicable and give the developers a voice in the training.

354
00:32:14,300 --> 00:32:21,000
And usually if you give them a voice in the training, it will be applicable just because they're going to tell you what they need.

355
00:32:21,680 --> 00:32:26,260
I think I've mentioned this 1427 times, but make it fun.

356
00:32:26,260 --> 00:32:28,000
It's got to be fun.

357
00:32:29,520 --> 00:32:32,100
Share how much you care about this stuff.

358
00:32:32,100 --> 00:32:39,660
If you're engaged, you're passionate about it, people are going to feel that and they're going to gravitate to you and gravitate to that topic.

359
00:32:39,660 --> 00:32:41,720
And then finally build relationships.

360
00:32:41,720 --> 00:32:47,300
Listen, reward, and always express gratitude along the way in this journey.

361
00:32:47,540 --> 00:32:56,120
So we're getting into the next phase of the toolkit and this is what I like to call the analysis hat trick.

362
00:32:56,120 --> 00:33:03,980
Three types of analysis that if we implement all three, we're going to have a much more secure code base in production.

363
00:33:03,980 --> 00:33:06,080
And the first is static code analysis.

364
00:33:06,080 --> 00:33:09,400
You might remember that from our SDLC pyramid.

365
00:33:10,220 --> 00:33:15,200
So static code analysis scans the code without running it.

366
00:33:15,200 --> 00:33:18,580
It looks for known patterns of vulnerabilities.

367
00:33:19,240 --> 00:33:22,760
You can usually select what you scan for.

368
00:33:22,760 --> 00:33:28,980
Some will allow you to scan for PCI, I almost always scan for OWASP top 10.

369
00:33:29,120 --> 00:33:34,940
If you're not familiar with OWASP or OWASP top 10, I strongly recommend that you check it out.

370
00:33:35,220 --> 00:33:38,820
Some of the deliverables from that organization are absolutely stellar.

371
00:33:39,040 --> 00:33:45,020
I've been a member for I think seven years now, just after joining the InfoSec team.

372
00:33:45,820 --> 00:33:49,360
And I've leveraged a ton of their stuff.

373
00:33:49,360 --> 00:33:52,540
So great organization, hats off to you if you're out there.

374
00:33:52,700 --> 00:33:58,300
In an ideal world, the static code analysis tools would only report the confirmed vulnerabilities.

375
00:33:58,820 --> 00:34:04,140
But even if you tune the daylights out of this thing, you're still going to get false positives.

376
00:34:04,140 --> 00:34:05,560
And that becomes a problem.

377
00:34:05,560 --> 00:34:15,120
Because originally, when we first set up this process, I was the only one that could review the false positives and whitelist them or waive them, as it were.

378
00:34:15,840 --> 00:34:19,620
That slowed things down immensely, because that was not my full-time job.

379
00:34:19,620 --> 00:34:27,080
That was something else on top of my full-time job, and possibly another full-time job some days, it felt like.

380
00:34:27,080 --> 00:34:32,300
So that being said, I decided to leverage the security champions.

381
00:34:32,300 --> 00:34:35,820
These were folks that had some basic knowledge of security.

382
00:34:35,820 --> 00:34:39,280
They knew their applications better than I ever would.

383
00:34:39,280 --> 00:34:44,640
And they had a passion for making secure code.

384
00:34:44,860 --> 00:34:46,180
Seemed like a perfect fit.

385
00:34:46,180 --> 00:34:55,960
So what I ended up giving several members of the security champions team the ability to review and waive false positives.

386
00:34:55,960 --> 00:34:57,560
And that has worked out great.

387
00:34:57,560 --> 00:35:03,040
The end result was we've had a lot more throughput.

388
00:35:03,040 --> 00:35:06,540
We've been able to address false positives much more quickly.

389
00:35:06,540 --> 00:35:13,720
And the adoption of the static code analysis tools have actually increased as a result of not having that delay.

390
00:35:14,260 --> 00:35:19,520
So I also see static code analysis as an extension of developer training.

391
00:35:19,520 --> 00:35:23,180
A lot of these tools have an IDE component.

392
00:35:23,300 --> 00:35:28,140
So for example, I use the SpringToolSweep Eclipse.

393
00:35:28,140 --> 00:35:37,420
And you can use the plugin and execute your scan from within your developer tool.

394
00:35:37,420 --> 00:35:41,000
And it will take you straight to the line that's got the problem.

395
00:35:41,000 --> 00:35:53,400
And in some of these tools, you can even have recommendations pop up in the bottom of one of the windows that says, hey, you might try blank, blank, and blank to address this vulnerability.

396
00:35:53,660 --> 00:36:02,100
If developers are doing this while they're developing, not waiting for the build process, it becomes a teaching tool.

397
00:36:02,100 --> 00:36:12,780
Because as a developer uses this, he or she is going to identify and learn that, oh, I need to check my input fields to make sure they're the right type and right size.

398
00:36:12,780 --> 00:36:13,200
Okay.

399
00:36:13,200 --> 00:36:15,720
You do that enough times, it becomes second nature.

400
00:36:15,720 --> 00:36:23,800
So this becomes not only a way to secure your code, but a teaching opportunity as well.

401
00:36:24,280 --> 00:36:31,800
So studies have shown that static code analysis can identify up to 80% of vulnerabilities.

402
00:36:31,800 --> 00:36:32,880
That's pretty awesome.

403
00:36:32,880 --> 00:36:38,880
If I could tell you that you can do one thing and remove 80% of vulnerabilities, I think that'd be worth doing.

404
00:36:38,880 --> 00:36:41,940
So this might be something you might want to take a look at.

405
00:36:42,180 --> 00:36:44,140
Here's some example products.

406
00:36:44,140 --> 00:36:47,640
There's three products along the top that are commercial.

407
00:36:47,820 --> 00:36:53,860
OWASP has an entire list of code analysis tools at that link right there.

408
00:36:53,860 --> 00:37:06,140
Again, not meant as recommendations, but these were tools that I've come across in my journeys, and they might be worth at least acting as a starting point for your own investigation.

409
00:37:07,200 --> 00:37:13,760
So the second goal in our hat trick, the second piece to the tripod, is dynamic code analysis.

410
00:37:13,760 --> 00:37:16,620
This was also in the SDLC pyramid.

411
00:37:16,820 --> 00:37:23,340
So unlike static code analysis, dynamic code analysis actually executes the code.

412
00:37:23,440 --> 00:37:32,540
Now that has a prerequisite that the code has to be compiled and free of errors and can run before you can actually perform the dynamic code analysis.

413
00:37:32,720 --> 00:37:41,180
Dynamic code analysis, I mentioned this before, is different than pen testing, whereas dynamic code analysis looks for the flaws in the code that can be exploited.

414
00:37:41,920 --> 00:37:51,080
Pen testing looks for ways to exploit the system in general, often at a greater detail, and almost always involving a human attacker.

415
00:37:51,080 --> 00:37:54,320
So again, two different things.

416
00:37:54,320 --> 00:38:04,140
They're kind of similar, but I really think that dynamic code analysis is really, at best, a subset of a good penetration test.

417
00:38:05,260 --> 00:38:23,200
So dynamic code analysis tools can identify up to 85%, but if you combine them with the static code analysis tools on the market, up to 95% of your application's vulnerabilities can be identified early on.

418
00:38:23,920 --> 00:38:25,880
95%, that's pretty darn impressive.

419
00:38:26,160 --> 00:38:29,820
So again, I'm going to throw a few products up here for you to take a look at.

420
00:38:29,960 --> 00:38:31,900
I've used several of these.

421
00:38:33,020 --> 00:38:36,340
Again, not recommendations , but a place for you to start.

422
00:38:37,360 --> 00:38:38,760
Give you a word of warning.

423
00:38:38,760 --> 00:38:45,540
I think I told you earlier on, I would tell you some things that work and some things that just didn't seem to work quite so well.

424
00:38:46,300 --> 00:38:49,320
Some of these tools can be weapons.

425
00:38:49,320 --> 00:38:51,220
I'm looking at you, Burp Suite.

426
00:38:52,340 --> 00:39:01,300
You need to do things like make sure you set the scope of your testing tool so it doesn't see that, oh, I've got a link to Facebook here.

427
00:39:01,300 --> 00:39:05,920
I will jump and follow that link in my crawl and I will start attacking Facebook.

428
00:39:05,920 --> 00:39:11,320
Set your scope absolutely to your application only.

429
00:39:11,560 --> 00:39:13,260
Run this stuff in test.

430
00:39:13,260 --> 00:39:14,780
Don't run it in production.

431
00:39:15,300 --> 00:39:28,600
If you find a SQL injection exploit and you start to execute it, it's very, very likely that you're going to corrupt or lose data on the database on the backside of that injection vulnerability.

432
00:39:29,340 --> 00:39:32,200
Also, watch out for forms.

433
00:39:32,540 --> 00:39:40,860
So let's say you're doing some dynamic testing and it's automated and you're getting ready to essentially attack a web form.

434
00:39:41,320 --> 00:39:46,500
Make sure you find out where that web form is going before you start your test.

435
00:39:46,500 --> 00:39:55,000
You can easily overrun a service desk by submitting thousands upon thousands of emails from a web mail form.

436
00:39:55,000 --> 00:39:58,900
Not saying that I've ever done that, but I know that it has happened.

437
00:39:58,900 --> 00:40:03,780
So the third goal in our hat trick is open source analysis.

438
00:40:04,000 --> 00:40:05,720
Open source is awesome.

439
00:40:05,720 --> 00:40:07,840
We all use open source.

440
00:40:08,060 --> 00:40:16,100
Some estimates put the percentage of open source in internally developed applications at 70%.

441
00:40:16,100 --> 00:40:17,940
That's 70%.

442
00:40:18,600 --> 00:40:33,300
Now the problem with that is, as a result, about 50% of our applications contain one or more open source pieces that has a high security vulnerability.

443
00:40:33,380 --> 00:40:41,700
And keeping that out of production is a huge, There are two different approaches I've taken when I've tried to address this problem.

444
00:40:41,700 --> 00:40:45,080
The first was a firewall slash proxy solution.

445
00:40:45,080 --> 00:40:51,220
There are products out there that you can actually set up that will proxy the main repos out in the wild.

446
00:40:51,220 --> 00:40:58,920
And they'll check as you pull down code libraries, they'll check them for license compliance.

447
00:40:58,920 --> 00:41:04,180
So you can say, hey, I'm okay with this license, this license, but I don't want that license.

448
00:41:04,180 --> 00:41:12,660
So you can set that up ahead of time and it'll check each time a module is pulled down that it is in compliance with your licensing.

449
00:41:12,660 --> 00:41:17,400
But more importantly, you can also set a vulnerability threshold.

450
00:41:17,460 --> 00:41:22,040
So you could, for example, say, hey, I don't want any critical or high translation.

451
00:41:22,040 --> 00:41:28,120
I don't want any CVE 7 or higher coming into my organization's local repository.

452
00:41:28,460 --> 00:41:44,340
So the problem with that is a lot of the tools that developers use, particularly in DevOps for automating building of systems, automating testing, have vulnerabilities because they really weren't meant to go live anywhere.

453
00:41:44,340 --> 00:41:55,940
They were basically developer aids, developer tools, and your policy is going to quarantine or prevent them from coming into your local repository.

454
00:41:55,940 --> 00:41:58,980
The result is you get a lot of phone calls.

455
00:41:58,980 --> 00:42:03,520
Hey, this module, I need it, I need it now, and it's quarantined.

456
00:42:03,520 --> 00:42:19,820
Well, that requires getting out on the firewall or proxy and looking at what's quarantined, determining how it's going to be used, where it's going to be used, and does that CVE apply in this use case, and then deciding whether you can waive it or not.

457
00:42:19,820 --> 00:42:23,220
It is a labor-intensive process.

458
00:42:24,300 --> 00:42:34,380
I didn't really like doing it this way at first, so I ended up moving to a build-based solution.

459
00:42:34,380 --> 00:42:43,060
Build-based solutions are very similar to what we're going to see in just a few minutes, where you basically add an open source review step to your automated build process.

460
00:42:43,060 --> 00:42:55,380
If you find that same vulnerability as it's going to production, let's say it's a CVE of 8, and it's going into production, you fail the build.

461
00:42:55,380 --> 00:43:12,820
At that point, the developers have the opportunity to go, okay, I need to get a security waiver, or I need to find a more recent or more updated library that doesn't have that CVE as an outstanding issue.

462
00:43:12,820 --> 00:43:16,680
So this ended up reducing the manual intervention.

463
00:43:16,980 --> 00:43:26,700
The only problem is you could end up having something on your internal network with a CVE of 7 or higher that's a developer tool.

464
00:43:26,700 --> 00:43:38,800
You've got to be very careful when you're whitelisting or removing items from quarantine to understand how it's going to be used, and if the CVE itself can be exploited in that manner.

465
00:43:38,800 --> 00:43:41,180
So here's some products once again.

466
00:43:42,560 --> 00:43:45,160
OWASP, I've mentioned them several times.

467
00:43:45,160 --> 00:43:46,480
Great organization.

468
00:43:46,480 --> 00:43:48,160
Again, these are not recommendations.

469
00:43:48,160 --> 00:43:52,440
These are simply places for you to start if you haven't looked at these already.

470
00:43:53,440 --> 00:43:55,740
And finally, let's talk about automation.

471
00:43:55,740 --> 00:43:59,300
So those last three steps, they seem like a lot.

472
00:43:59,300 --> 00:44:06,480
And originally we were doing them manually, and it became unwieldy very quickly.

473
00:44:07,020 --> 00:44:09,180
It just took a lot of time.

474
00:44:09,180 --> 00:44:12,100
It took a lot of effort.

475
00:44:12,520 --> 00:44:17,860
So what we were able to do was automate most of this into the build pipeline.

476
00:44:17,860 --> 00:44:30,000
So if you have a continuous integration tool, something like Jenkins, you can add steps that execute static code analysis, dynamic code analysis, and open source analysis with every build.

477
00:44:30,000 --> 00:44:38,480
So this gives you all of that protection over 95% from the studies with every single build that you do.

478
00:44:38,800 --> 00:44:41,080
Now, there's some things that are just too stinking big.

479
00:44:41,080 --> 00:45:02,380
So if you're still dealing with legacy applications, big old monolithic jars and ears, if you've got something that's somewhere in the hundreds of thousands of lines of code, sometimes close to a million lines of code, this is not going to go well for you because the dynamic scan,

480
00:45:02,380 --> 00:45:07,200
the static scan, those pieces are going to take a long, long time.

481
00:45:07,200 --> 00:45:13,860
I tested one application for a friend of mine just to try it.

482
00:45:13,860 --> 00:45:21,600
It was a legacy application, and the scan for just the static code analysis piece took over 12 hours.

483
00:45:21,600 --> 00:45:27,580
You can't stop a build for 12 hours and expect the developers not to be concerned.

484
00:45:27,580 --> 00:45:31,920
So I don't recommend this for large applications.

485
00:45:31,920 --> 00:45:38,080
If you do, you might do it as an asynchronous build step.

486
00:45:38,080 --> 00:45:41,700
So basically, it says, hey, I've got to go do static code analysis.

487
00:45:41,700 --> 00:45:44,580
I'm going to launch it and I'm going to continue with my build.

488
00:45:44,580 --> 00:45:53,160
You lose the advantage of being able to block at that point, but you still have an automated scan going along.

489
00:45:53,160 --> 00:46:02,840
You'll need to have a process in place to alert the development team as well as information security of the results of that scan, and then decide how you want to react.

490
00:46:02,840 --> 00:46:04,980
But you lose the ability to block.

491
00:46:04,980 --> 00:46:12,400
But fortunately, the industry is really moving towards smaller builds, APIs, microservices, things like that.

492
00:46:12,400 --> 00:46:16,260
And inline scans are very, very possible.

493
00:46:16,260 --> 00:46:17,440
They're very fast.

494
00:46:17,440 --> 00:46:24,000
These tools are tuned for this kind of work, and they run incredibly, incredibly fast.

495
00:46:25,720 --> 00:46:33,260
Just aside, some of the tools are still working on their automated pipeline plugins.

496
00:46:33,260 --> 00:46:46,700
So if you are looking at any of the tools, whether it's the ones I've mentioned or something else, make sure that you ask about the maturity of their continuous integration pipeline plugins.

497
00:46:48,340 --> 00:46:54,440
Otherwise, you might find yourself stuck manually doing this for a while until you can find a way to do that.

498
00:46:56,080 --> 00:46:58,560
I want to end with a couple of final thoughts.

499
00:46:58,860 --> 00:47:05,600
So the keys to success, hopefully you can read this, they are start small.

500
00:47:05,600 --> 00:47:06,700
Do one thing.

501
00:47:06,700 --> 00:47:14,500
For me, it was starting with a training program that literally led to everything else that you saw in this presentation.

502
00:47:14,500 --> 00:47:19,760
And as you saw in the presentation, this stuff tended to feed upon itself.

503
00:47:19,960 --> 00:47:26,380
The developer training led to more developer training, which led to a security program.

504
00:47:26,380 --> 00:47:30,160
So start small, and it will all come together.

505
00:47:30,660 --> 00:47:35,800
Get management buy-in by both Information Security and by AppSec Management.

506
00:47:36,460 --> 00:47:40,020
The centerpiece right there, that has to be the key for me.

507
00:47:40,020 --> 00:47:41,220
It has to be fun.

508
00:47:41,220 --> 00:47:44,580
When this stuff is no longer fun, I'm not going to do it anymore.

509
00:47:44,580 --> 00:47:48,060
We've got to keep making it fun, particularly for our developers.

510
00:47:48,060 --> 00:47:50,500
Developers are doing a lot of stuff.

511
00:47:50,500 --> 00:47:55,600
Adding security on top of everything else they're doing is a pain in the butt.

512
00:47:55,600 --> 00:48:00,680
And for the most part, it's not welcome, unless you can make it fun.

513
00:48:00,680 --> 00:48:03,640
If you can make it fun, it's going to happen.

514
00:48:03,820 --> 00:48:08,300
So just like we just mentioned, automate everything you possibly can.

515
00:48:08,340 --> 00:48:10,300
And remember, it's all about those relationships.

516
00:48:10,300 --> 00:48:13,820
The relationships with your scrum teams.

517
00:48:13,820 --> 00:48:17,660
It's the relationships with your developers.

518
00:48:17,660 --> 00:48:23,120
It's the relationships with your vendors and the trainers that you can get to come in.

519
00:48:23,440 --> 00:48:27,700
So building a good relationship is key.

520
00:48:28,500 --> 00:48:30,600
Don't try to reinvent the wheel.

521
00:48:31,200 --> 00:48:35,500
The best innovation is usually stolen from somewhere, or given from somewhere.

522
00:48:35,500 --> 00:48:44,600
So if you can take one piece from this discussion and implement it in your organization, I will feel blessed, and I'd love to hear about it.

523
00:48:45,560 --> 00:48:46,880
Take it slow.

524
00:48:47,140 --> 00:48:51,300
This stuff is new for all of us, and it's always changing.

525
00:48:52,600 --> 00:48:56,280
And finally, learn what works best in your environment.

526
00:48:56,300 --> 00:48:58,840
So I want to wrap up with thank yous.

527
00:48:58,840 --> 00:49:07,020
I want to thank you for spending the last 40, 45 minutes, whatever it ended up being, with me and with AppSec Village.

528
00:49:07,020 --> 00:49:08,560
I want to thank AppSec Village.

529
00:49:08,560 --> 00:49:10,780
I got to volunteer there last year.

530
00:49:10,780 --> 00:49:13,260
I ended up spending the entire DEF CON there.

531
00:49:13,880 --> 00:49:20,320
After, I think that was my sixth year in information security, I finally found my home, and that is the AppSec Village.

532
00:49:20,320 --> 00:49:26,220
So thank you to the leadership of AppSec Village for doing this, and for giving me this opportunity.

533
00:49:26,760 --> 00:49:37,120
I want to reiterate my thanks to Jim Manico, and he's kind of been my spirit leader on this journey, and I appreciate all you've done.

534
00:49:37,400 --> 00:49:41,620
My colleagues Barry and Shane, my partners in crime, you guys are awesome.

535
00:49:41,620 --> 00:49:50,320
And finally, my best friend, my wife, who's put up with me doing this and several other things in information security.

536
00:49:50,700 --> 00:49:54,400
She's absolutely the best, and I can't thank her enough.

537
00:49:54,460 --> 00:49:56,280
So thank you guys very much.

538
00:49:56,280 --> 00:50:04,680
I appreciate you spending this time with me, and I hope you have a really, really excellent rest of your DEF CON.

539
00:50:04,680 --> 00:50:05,580
Thanks.