start	end	text
12740	18140	Hello and welcome back. I hope you're looking forward to another fantastic talk here at the
18140	26240	AppSec Village for DEF CON 29 in safe mode. It's important to remember you're 3-2-1.
26240	32260	Even though we are separate from each other and socially distanced, your cohabitants will really
32260	38680	appreciate that one in there. And just take care of yourselves. Get some good sleep, get some good
38680	45180	food, make sure you take a shower. For our next talk, we've got David Waldrop talking about the
45180	51980	DevOps and Agile Security Toolkit. After 25 years in the application development, David moved to
51980	57300	information security just over seven years ago. His passion is helping developers write secure
57300	63060	code. He's currently an information security advisor supporting the application development
63060	68700	community at his employer. Let's put our hands together and welcome David Waldrop.
70000	75400	Hi and welcome to the AppSec Village. I hope you're having a great DEF CON 2020,
75400	81780	albeit in safe mode. My name is Dave and for the next 30-45 minutes, we're going to talk about the
81780	86420	DevOps and Agile Security Toolkit. Now before I get started, I have to make the standard
87100	92100	disclaimers. First, this presentation and any comments that I make are my own and do not
92100	97800	necessarily represent those of my employer. Secondly, I will mention a few example products.
97800	103100	Some are commercial, some are open source. These are not necessarily meant as recommendations.
103260	107500	They are mentioned to serve as a starting point for your own investigation and review.
107500	112980	These are things that I've run across in my journey, so take that as it is. Let's start
112980	120180	with the two terms that are on everybody's buzzword bingo sheet these days, DevOps and Agile.
120180	125220	We need to get definitions in place for the duration of this talk, so we have an idea that
125220	131780	we're all dealing with the same playing field. Agile. Agile focuses on continuous iterative
131780	137180	development and testing. Software is developed in smaller increments and then is integrated
137640	144320	for final testing and delivery. Scrum, Kanban, and XP Extreme Programming are examples of Agile
144320	151000	approaches. On the other hand, DevOps focuses on the collaboration between app dev and operations.
151000	157340	The focus is on deploying code to production in a fast manner, usually automated.
158920	164100	So you can see from the definitions that these are two separate concepts. However,
164100	169860	many organizations tend to combine them into one initiative. Very, very rarely do you find
170600	176680	a shop doing DevOps and not doing Agile, or vice versa. So no matter whether they start with DevOps
176680	183140	or they start with Agile, the one piece that's usually late to the party is security. But no
183140	188060	matter how late security is invited to the party, or whether we have to crash the party ourselves,
188060	194360	we usually can still bring valuable resources without slowing things down, or at least without
194360	198420	slowing them down a lot. And that's what we're going to talk about through the rest of this
198420	205740	discussion. So for the past few years, I've worked with several different Agile and DevOps teams.
205980	212920	My integration with this started as a pilot program where I was the information security
212920	219400	representative that was assigned to help introduce Agile to our organization. More recently, I've
219940	225120	implemented the information security integration with the Agile and DevOps teams across our
225120	229520	organization. During this session, I'm going to share with you some things that really worked well
229940	233940	and some things that just didn't seem to work at all. So hopefully you can take something
234340	237860	away from this and apply it to your own organization.
239760	243720	This particular slide will give you the map of what we're going to talk about for the rest of
243720	248100	this discussion. We're going to talk about Agile staffing. We're going to talk about the eight
248100	253440	simple questions. We're going to talk about the security champions program, and then go into
253440	259320	developer training, which is my favorite part of my job. Security in the software development life
259320	264540	cycle. And then we're going to hit the hat trick of securing your code. Static code analysis,
264540	269480	dynamic code analysis, and open source analysis. And then finally, we're going to wrap things up
269480	274680	by automating the heck out of it. So let's start with Agile staffing.
275660	280900	Sherman set the way back machine from when I started with Agile. One information security
280900	288620	person was assigned to one Agile team. That security person would be integrated into that team
288620	297520	and attend every ceremony, every meeting, everything, every day. That didn't work.
297520	301600	There are a couple of reasons I think that that didn't work. First of all, Agile teams tend to
301600	309560	multiply. You can go from a proof of concept with one team to having close to 30 teams in less than
309560	317540	two years. Most organizations do not have the information security bandwidth to support that
317540	324160	type of staffing model. In addition, the scrum masters tend to treat every member on the team
324160	328320	as though they can perform all tasks. And when you're dealing with a team of all developers,
328320	334420	that makes perfect sense. This model tends to fall apart though when you put information security,
334420	343540	QA, middleware, any other infrastructure or shared services team as a part of the Agile team.
343560	352300	What happens is in the Agile process, their hours count against your available development hours.
352540	360160	But the infrastructure people, the information security people do not have the skills or the
360160	365620	capabilities to do some of the development. So you kind of see that there's a mismatch. You're
365620	369880	given hours, but you're not allowed to use them. So it caused a lot of problems for some of our
369880	375260	scrum masters early on. Again, we realized this was not the way to do it. So I ended up coming up
375260	380620	with a two-tiered approach for trying to integrate information security into the Agile team.
381180	387740	First, at a high level, at the enterprise level, we will have an architect, an information security
387740	394140	architect, and they will inject into those high-level projects and basically set up guardrails
394140	402840	and guidelines. So as projects are being devised at the visionary level, they can jump in and try
402840	409080	to help guide them into making more secure decisions at that high level. Now when you get
409080	413860	down into the weeds though, each Agile team does need to have somebody they can rely upon
413860	421680	for information security. And so what we ended up doing is we created named resources within
421680	428260	the information security team that are assigned to Agile teams. Now it's not like the previous
428260	432960	model. You're not assigned, you're not sitting with them, you're not assigned to go to every meeting.
432960	439720	We're going to talk about how we scaled that back, but we ended up finding a way where based on
439720	446020	where information security can add the most value, we attend those ceremonies and only those
446020	452340	ceremonies. So at the end of the day, each information security representative can support
452340	457680	between six and eight different Agile teams. That is a model that is certainly sustainable.
458460	465120	So what is a little different? Well first of all, we make sure that every Agile team has a named
465120	470200	resource that is responsible for information security for them. So this gives them a named
470200	475360	contact. If they ever need something, they know exactly who to call and where to start that
476600	484760	quest. Secondly, we attend, or the Agile reps attend, the backlog refinement sessions. These
484760	490080	are meetings that happen once every two weeks before the next sprint, and it defines what
490080	495100	they're going to work on in that next sprint. It's a perfect opportunity to jump in and say,
495760	502300	you're planning this work, we need to think about blah blah blah from an information security
502300	509460	standpoint. We can get in just ahead of when development starts. It's a perfect time to bring
509460	514920	up security issues, make recommendations, and answer questions. And then finally, we attend
514920	520740	the quarterly planning increment meetings where the vision is cast for the next quarter, and we
520740	526320	can make sure that we're on their radar for things that we think we need to be a part of.
527540	533340	So this is kind of a high-level diagram. The purple arrow at the top is the information security
533340	540040	architect. That particular person, as you can see from the diagram, sets guardrails for enterprise
540040	545880	planning projects. All of the red arrows are the different places where our individual security
545880	552220	representatives can or may inject into the Agile teams. We don't inject in all those places,
552220	556360	but there are places where we can, and we give them that option to call us if they're needed.
558280	562480	So let's move on to what I like to call the eight simple questions.
562600	572200	This particular concept came up as a result of assigning our Agile reps to their teams.
572800	578920	We found that the leadership within the Agile teams really wasn't sure when to call us. And
578920	584740	since we weren't sitting there in every meeting, there was a possibility that something could come
584740	591560	up that might fall off the radar, that we might not get contacted for, that they may not even think
591560	597420	is important. So we decided that we need to give clear guidelines to the scrum masters and product
597420	604200	owners as to when they should include information security. That's how I created the eight simple
604200	610920	questions. So basically these eight questions are a guideline for that leadership team, and if they
610920	616620	answer yes to any of them, they need to make sure to include their information security representative
616620	623720	in those discussions. These are the questions as I currently have them. We're going to talk about
623720	628920	that in just a bit because they change over time. These are not one and done kind of things.
629200	634060	Am I dealing with any sensitive information? Now information that is sensitive, it's going to mean
634060	637860	something different for each one of our organizations. If you're working in healthcare,
637860	642740	if you're working with banking information, if you're working with retail with PCI information,
643080	647340	you will be able to define what sensitive information means for your organization.
647560	653200	Am I sending or receiving data outside of the company? Is there a new vendor involved?
653200	658820	Am I going to introduce new software systems or libraries, even open source? And remember open
658820	664040	source because we're going to talk about that. It's a unique challenge. Do I need access to
664040	669900	secrets such as passwords, keys, or certificates? Will I need the ability to use single sign-on?
669900	676540	Am I going to leverage the cloud? Will I need updated access roles? So these eight questions
676540	682380	are what we are currently using as a guideline for the interaction between the Agile team
682380	687740	and their information security representative. As I mentioned, the exact questions are going to
687740	693380	differ on your organization. The questions themselves will actually change over time.
693380	698520	I started I think with seven questions and then we had several issues come up with access roles
698520	704540	and who to ask and blah blah blah. So we added a question for that too. So don't be afraid to
704540	711260	take a stab at it and massage it and change it. That's the thing about Agile and DevOps,
711260	715500	it's changing all the time and we as information security people need to be able to change our
715500	723600	stuff as well. So the basic idea is this. Create a list of questions, usually 10 or less,
723600	727100	something that's going to fit easily on a PowerPoint slide or a single piece of paper
727100	732580	so they can take it to meetings with them. Give that to the Agile team leadership and they have
732720	737660	a guideline. They now have a concrete way of knowing when they need to work with information
737660	746180	security. So we found another advantage to this approach as we onboarded additional information
746180	751440	security people to become representatives for the Agile teams. A lot of these folks were pretty new
751440	755740	to the organization and they didn't really know when they needed to inject themselves into their
755740	760400	teams. They didn't know what they needed to listen for in the backlog refinement sessions.
760400	767960	So we used these 8 simple questions on our own people. It gave them guidelines and hints of what
767960	773500	to listen for when they're working with their Agile teams. So this kind of became a win-win
773500	781420	on both sides of the coin. So I think this was a great tool that we've been able to use.
782840	787920	Another tool that I honestly have to thank Jim Manico for. Jim, if you're out there,
787920	793700	thank you for this. This was a recommendation he made to me after one of our training sessions.
793760	799220	And that is the Security Champions Program. A Security Champions Program is a team of
799220	806440	application developers. We get one from each Agile team. And these developers have expressed
806440	811460	an interest in information security, or secure coding, or just security in general.
811460	820560	This is a voluntary program. The members of the team volunteer to come to the Security
820560	826740	Champions Program. And we typically meet quarterly. I provide them lunch when we're in the office,
826740	831400	thanks 2020. I look forward to getting back to the office and having lunch with these guys and
831400	837560	gals. They're a great team. So the team has this following charter. We assist the Agile teams in
837560	844660	terms of application security. We serve as a communication conduit between the Agile teams
844660	850440	and information security. And then we meet quarterly to share experiences, we ask questions,
850440	857040	we share knowledge. It has had some incredible benefits. First of all, security has a
857040	862480	relationship with at least one person on every Agile team. I'm not saying we have them all.
862480	868460	What I am saying is, sometimes we hear about things long before it percolates through the
868460	873320	management chain. I might get a phone call about, hey, we're thinking about using this open source
873320	878740	library. Is that cool? We take a quick look at it. We approve it. By the time the formal approver
878740	884700	gets into my hands, we've already done it and they're coding. So it gives us a great communication
884700	892980	conduit. And that helps me make things faster. I may have not mentioned this before. Hopefully,
892980	898220	I have. Relationship is key. You're going to hear about relationship throughout this
898220	905560	entire discussion. We build relationships with those Agile teams. It starts with having an
905560	911640	assigned rep. It moves into a different relationship level when we have a security
912320	919440	security champions team. Secure coding training takes it to yet another level. And you'll see why
919920	924060	in just a few minutes as we talk about the developer training. But it's about building
924060	929500	relationships. It's about having fun while we're doing the right thing for our organizations.
930000	934940	So, so far, the team has done a lot of really cool things. We've reviewed vendors and products,
934940	939380	both on the development side and on the information security side. We've selected
939880	946340	our developer training programs for the last two years based on the input from the security
946340	952980	champions team members. We've done a bunch of POCs together. We've cross consulted. Now,
952980	959960	this was kind of an unexpected benefit is where if I had a team that was very well versed in one
959960	966620	technology and they mentioned that in a security champions lunch, another team might reach out
966620	972900	and say, hey, we're about to do that. Can we sit down and talk? So, it actually starts to
972900	978080	cross pollinate some security knowledge and some technical knowledge between different application
978080	983880	teams. And then finally, it served as a candidate group for application security openings within the
983880	992060	InfoSec team. The software development life cycle. So, as we started to roll out security, a lot of
992060	1000620	people have asked us, how does this fit into our SDLC? So, I ended up creating a plan that maps
1000620	1006060	information security phases to the software development life cycle as it currently exists.
1006060	1012640	It is not a perfect fit, but I think it is a good representation of how information security can
1012640	1020620	bring value to each step of the SDLC. So, this is what I like to call my dark side of the moon
1020620	1026700	diagram. Those of you old enough to remember, probably recognize the album cover. So, if you
1026700	1032200	take a look on the right, we have the software development life cycle from concept into design,
1032200	1039360	up into coding testing and QA, QA and deployment, and then post-deployment at the very top.
1039360	1045240	Now, if you look on the left side, you'll see that it matches up pretty closely with security
1045240	1051660	knowledge, security design, secure coding techniques, security testing, and then security
1051660	1059000	response. The each individual layer of the pyramid, I think, are deliverables that security
1059000	1068960	can bring that help take that concept on the left and apply it to the SDLC phase on the right. And
1068960	1073920	we're going to walk through those real quick. Training is the base of the pyramid. I think
1073920	1079860	training is foundational to everything above it. If you don't have a solid foundation of
1079860	1087860	training for your application developers, everything else is at risk. So, this particular
1087860	1092920	approach, we're going to talk about in depth, again, because it's my favorite part of what I do.
1092920	1098020	It includes things like lunch and learns, outside speakers, conferences like you're attending right
1098020	1105100	now, video training, and things like that. The next level up is threat modeling. Threat modeling,
1105100	1109980	the application team, and InfoSec, we sit together, we work together to examine the application
1109980	1115900	design, the system flows, whatever they can bring to us that they have available in that design
1115900	1123360	phase. We look, we try to work together and identify potential vulnerable vectors and try
1123360	1129980	to figure out a way that we can mitigate them. Now, threat modeling, there are books upon books
1129980	1134580	written about threat modeling. And there are people in my organization on my InfoSec team
1134580	1140720	that are much better at it than I am. And they can actually create incredible documents and spend
1140720	1145300	hours and days and days and days doing this. I think you need to start out small. I think it's
1145300	1150800	about baby steps at first. Start with a discussion. This can be a one or two hour discussion where you
1150800	1155740	have those deliverables from the design phase, and you whiteboard them. And you come out of there
1155740	1163600	with to-dos, or questions, or things to research. It doesn't have to be this big, heavyweight,
1164640	1170400	overbearing approach to threat modeling at first. It needs to be a conversation. That gets back to
1170400	1176020	relationships. You need to have those relationships in place so you can have those conversations.
1176560	1181060	The third level up is static code analysis. We're going to talk about this
1181780	1187800	in much more detail later. But most tools are designed to identify potential vulnerabilities
1187800	1192860	within the code. Now, static code analysis actually looks at the code and looks for
1192860	1197220	patterns within the code without running it. Now, we're going to see a little bit later
1197220	1205300	why that's important. I think I went the wrong way. There we go. Dynamic code analysis is the
1205300	1210300	next layer up. And that is very similar to the previous layer, which was static code analysis.
1210300	1215320	But instead of just looking at the code, the code is actually executed and tested for
1215320	1224360	vulnerabilities. So, like I said, these two pieces are very, very similar. One is looking
1224360	1230000	at patterns with the code. The other is actually exercising the code and looking for issues.
1230000	1234260	The next is pin testing. Pin testing can be performed by your internal team if you have
1234260	1240560	the skills and resources, or you can use an external firm. This is usually a greater
1241240	1249100	scope than dynamic testing. It will actually look at things not just in your application,
1249100	1255320	but system-wide for vulnerabilities and attack vectors. And usually there's a person involved.
1255320	1259020	It's not just an automated exercise. We're going to see a little bit more when we talk about
1259020	1263800	dynamic testing, the difference between that and pin testing. And then finally,
1263800	1269180	when the code is in deployment, you need to have a vulnerability management process
1269180	1278440	that you can classify, assign, and track identified issues. Right now, InfoSec meets
1278440	1284600	regularly with a representative from each application team to review these and to make
1284600	1289540	sure that they're working and they're on track. Again, those relationships are absolutely key
1289540	1297780	to making this work. So now we are getting to what I consider my absolute favorite part of my job,
1297780	1303300	and that is developer training. If you remember back to that pyramid we just looked at,
1303300	1311940	this is the foundation for everything that InfoSec does with AppDev. When I came in
1312780	1319260	to Information Security about seven years ago, I started a developer training program because I
1319260	1325520	realized that the developers really are your last best hope for defense when it comes to
1325520	1331560	application security. If your application is written correctly, then everything you throw
1331560	1337960	in front of it, application firewalls, everything like that, that's gravy. That's nice, but you've
1337960	1344720	got a really tight application, and that's where real security takes place. The problem is,
1344720	1349800	I was a developer for roughly 25 years before coming to Information Security,
1350440	1355540	and nobody taught this stuff. This is stuff you learned on your own if you had an interest in it,
1355540	1363980	and it wasn't universally available to all developers. So I started small. I started with
1363980	1371860	Lunch and Learns. It was really something that, hey, I think this is important, let's try it,
1371860	1376460	let's see if it works. And it really did. We would end up having Lunch and Learns about every other
1376460	1381840	month. I would create a class on a specific topic. My first one was cross-site scripting,
1381980	1388320	a lot of fun. It's neat when you can show people cross-site scripting examples, even if you just
1388320	1393800	run WebGoat or something like that. They get excited. They kind of dig it, and they start
1393800	1399100	trying it on their own stuff. The content usually lasts about 40 or 45 minutes, a lot like this
1399100	1403680	session, and then we leave 10 to 15 minutes for open discussion and questions. Now the key here
1403680	1411160	is, make it fun. It has to be fun. I do things like, I'll go out and I'll buy cookies or some
1411160	1416480	kind of treat, so we can all sit around, eat something together. There's usually some kind
1416480	1420520	of door prize or door prizes, and they don't have to be expensive. You can just go down to
1420520	1426340	five below and get a couple of wireless Bluetooth speakers. And people dig it. It's a lot of fun.
1426700	1432600	Again, making it fun is absolutely key to a successful Lunch and Learn program.
1433180	1438440	So once we got that up and running, I decided we needed to take this up a notch.
1438920	1445680	There is a level of training that I couldn't provide that I think our developers really,
1446280	1451860	and in order to do that, I knew that I had to reach out to industry experts.
1452940	1456860	And what we ended up doing is, we were able to get commitment from management, both on
1456860	1464040	the IT development side and on InfoSec side, that we could do one or two days where most,
1464040	1468480	if not all, of the developers will attend a training event. Now here's a hint. If you are
1468480	1475280	going to try this and you're doing Agile, put it in your Innovation Sprint. If you put it in the
1475280	1480960	Innovation Sprint, which is usually the last sprint of the quarter, and the developers have
1481100	1486380	a little more leeway of what they work on during the Innovation Sprint, you'll very likely get
1486380	1492380	more participation than if you step on a middle of a sprint where there's actual deliverables due.
1492380	1499020	So keep that in mind. I was really, really lucky. My very first interaction with this program,
1499020	1505100	I was able to secure Jim Manico, and he came in and made an incredible impression on our
1505100	1509600	development teams. Jim's been back, I think, three times now. Always a pleasure.
1510420	1517760	Rich Mogul's been by twice. We've had a recent one with Ron Paris. Dr. Philippe Des Riques,
1517760	1529000	phenomenal instructor. I just took his OAuth class, so if you're looking for OAuth,
1529000	1534520	looking forward to that. The reason I actually want to call these gentlemen out by name is I
1534520	1538520	think it's important to be grateful. I am very grateful for everything they've done for me
1539080	1545580	and my development teams. And it's important to be grateful to those who help you along the way
1545580	1549940	on this journey, because none of us make it alone. So gentlemen, thank you very much. If you
1549940	1557440	see this or hear it later, your training, your mentorship has meant a lot to me and my
1557440	1564080	development teams. So one cool aside is, I mentioned this earlier, the Security Champions
1564080	1568920	team has been involved in the selection of the instructor for the last two years.
1568920	1575540	This gives your application development teams a voice in the training. And we've seen that
1575540	1584100	participation has actually increased. We had great participation before then, but when the developers
1584100	1590700	actually had a voice in picking who's coming in and what they're going to talk about,
1591300	1595980	we're finding that it's much more applicable to what's on their roadmap. It's much more
1595980	1602500	applicable to the skills they're hoping to get. And participation goes beyond what I could have
1602500	1607920	ever hoped for. So again, we're starting to see some of these initiatives are starting to
1608480	1613940	weave together and starting to build kind of a program. It's kind of cool.
1614420	1621360	So this next one, I really wish I was doing this right now. Take a developer to DEF CON.
1621360	1627480	We had management agree to provide scholarships to developers that were selected by the information
1627480	1632860	security team. So we would give them the money to buy their badges so they wouldn't have to go and
1632860	1641100	get money out of their own bank account. So each year I've been able to bring up to three developers
1641100	1650380	to DEF CON. That's been awesome. It has been just a phenomenal experience. They've gained a ton of
1650380	1657780	knowledge and a ton of awareness by DEF CON. One of the things that I found really interesting was
1657780	1662700	I really advised them not to restrict their sessions to things that they think will apply
1662700	1668920	to their job. I did that my very first DEF CON and I wasn't really sure I was going to come back,
1668920	1675400	to be honest with you. The next year, one of my co-workers strongly encouraged me to go. Thank you,
1675400	1683400	Barry. And encouraged me to go and find a few fun things that just had nothing to do with my
1683400	1689840	immediate job. The interesting thing is they ended up having more to do with the things that were
1689840	1697280	coming down the road in my job than I ever could have imagined. So I fully encourage each one of
1697280	1702380	these developers that I bring to DEF CON to experience the full DEF CON experience. Go to
1702380	1709620	all the sessions. Try some of the capture the flags. Try whatever you want. Have a good time
1709620	1717640	because it's really important for them to be immersed in the DEF CON experience.
1717720	1722920	So like I said before, they come back with a changed perspective. It's a lot like that scene
1722920	1728800	in The Matrix where that thing crawls out of Neo and he goes, my god, that thing is real?
1729100	1735520	They understand now that the security is not just something that we're teaching and preaching about
1735520	1743660	within InfoSec, that this stuff has real world application. And it's that realization that makes
1743660	1751260	them a better developer, a more secure developer going forward. We've seen a trend where a lot of
1751260	1756620	the folks that we've taken, actually, I think all but one of the developers that we've taken
1756620	1763300	to DEF CON have eventually become security champions members. And then more recently,
1763300	1769360	in the last couple of months, one of those developers, I think he was the first one I
1769360	1776160	took to DEF CON with us, ended up joining the Information Security team on a full-time manner.
1776200	1781540	And Shane, if you're listening, welcome. I'm really glad that you're part of the team.
1781960	1786220	So there's a couple of other fun activities that we do in developer training. Again,
1786220	1791920	got to keep it fun. Everything has to be fun. You learn better when you're having a good time. I
1791920	1799300	look at DEF CON. We're all having a great time and we're learning stuff. So a couple of DEF CONs
1799300	1804640	ago, for reasons of work, I couldn't make it. It was the only DEF CON since coming over to
1804640	1811820	Information Security that I was not able to go to. And I must admit I was bummed. So what I decided
1811820	1820440	to do was I was going to have my own little DEF CON. And we decided that, okay, let's just take
1820440	1826520	people along for the ride. So we identified five videos from previous DEF CONs and we had a lunch
1826520	1832800	and learn for five days that week where we watched it. We talked about it. We had drawings for DEF
1832800	1839440	CON related swag or items or giveaways. We just kind of had a good time. It was kind of built up
1839440	1846000	as, you know, hey, take a break. Come bring your lunch. Let's hang out. Watch something from DEF
1846000	1852700	CON. Talk about it. You might win a t-shirt. You might win some stickers, whatever. And people
1852700	1857720	really enjoyed it. It was a lot of fun. As a matter of fact, I'm probably going to do one this fall
1857720	1863140	for the folks that didn't get to enjoy what you're doing right now, what you're enjoying right now.
1863400	1868260	And then finally, I have an annual thank you lunch and learn. This was a tradition I started
1868260	1873660	the first year we did lunch and learns. It's always the last lunch and learn of the year.
1873660	1878240	And I typically will go out and I'll purchase quite a few giveaways. They don't have to be
1878240	1885080	expensive. We do things for, you know, like say the Bluetooth speakers, boxes of candy. We've done
1885080	1893420	some Christmas mug sets, you know, just weird things, but things that are kind of fun.
1893560	1900420	And we do a shorter lunch and learn followed by a session where we just eat some good food
1900420	1906140	and we have some drawings. And I have an opportunity to thank my development community
1906140	1911020	for the support of the lunch and learn program and for the support they've given to information
1911020	1918240	security. I think it's important to say thank you. And this has been just an incredibly popular
1918240	1925120	lunch and learn every year. It's a lot of fun and I highly recommend it. So here are the keys to a
1925120	1934220	successful training program. Make it applicable and give the developers a voice in the training.
1934300	1939360	And usually if you give them a voice in the training, it will be applicable just because
1939360	1944780	they're going to tell you what they need. I think I've mentioned this 1427 times, but
1944780	1953160	make it fun. It's got to be fun. Share how much you care about this stuff. If you're engaged,
1953160	1957740	you're passionate about it, people are going to feel that and they're going to gravitate to you
1957740	1963840	and gravitate to that topic. And then finally build relationships. Listen, reward, and always
1963840	1972200	express gratitude along the way in this journey. So we're getting into the next phase of the
1972200	1977640	toolkit and this is what I like to call the analysis hat trick. Three types of analysis
1977640	1983980	that if we implement all three, we're going to have a much more secure code base in production.
1983980	1989400	And the first is static code analysis. You might remember that from our SDLC pyramid.
1990220	1997320	So static code analysis scans the code without running it. It looks for known patterns of
1997320	2005180	vulnerabilities. You can usually select what you scan for. Some will allow you to scan for PCI,
2005180	2012260	I almost always scan for OWASP top 10. If you're not familiar with OWASP or OWASP top 10, I
2012260	2017380	strongly recommend that you check it out. Some of the deliverables from that organization are
2017380	2024060	absolutely stellar. I've been a member for I think seven years now, just after joining the InfoSec
2024060	2031940	team. And I've leveraged a ton of their stuff. So great organization, hats off to you if you're out
2031940	2037220	there. In an ideal world, the static code analysis tools would only report the confirmed
2037220	2043400	vulnerabilities. But even if you tune the daylights out of this thing, you're still going to get false
2043400	2048580	positives. And that becomes a problem. Because originally, when we first set up this process,
2048580	2054260	I was the only one that could review the false positives and whitelist them or waive them,
2054260	2060140	as it were. That slowed things down immensely, because that was not my full-time job. That was
2060140	2066180	something else on top of my full-time job, and possibly another full-time job some days,
2066180	2073260	it felt like. So that being said, I decided to leverage the security champions. These were folks
2073260	2079280	that had some basic knowledge of security. They knew their applications better than I ever would.
2079280	2087080	And they had a passion for making secure code. Seemed like a perfect fit. So what I ended up
2087080	2094840	giving several members of the security champions team the ability to review and waive false
2094840	2100920	positives. And that has worked out great. The end result was we've had a lot more
2101780	2108460	throughput. We've been able to address false positives much more quickly. And the adoption
2108460	2113720	of the static code analysis tools have actually increased as a result of not having that delay.
2114260	2120580	So I also see static code analysis as an extension of developer training. A lot of these
2120580	2129660	tools have an IDE component. So for example, I use the SpringToolSweep Eclipse. And you can
2130540	2138980	use the plugin and execute your scan from within your developer tool. And it will take you straight
2138980	2145340	to the line that's got the problem. And in some of these tools, you can even have recommendations
2145340	2151320	pop up in the bottom of one of the windows that says, hey, you might try blank, blank, and blank
2151320	2157360	to address this vulnerability. If developers are doing this while they're developing,
2157360	2165280	not waiting for the build process, it becomes a teaching tool. Because as a developer uses this,
2165280	2171000	he or she is going to identify and learn that, oh, I need to check my input fields to make sure
2171000	2175720	they're the right type and right size. Okay. You do that enough times, it becomes second nature.
2175720	2183800	So this becomes not only a way to secure your code, but a teaching opportunity as well.
2184280	2191800	So studies have shown that static code analysis can identify up to 80% of vulnerabilities.
2191800	2196600	That's pretty awesome. If I could tell you that you can do one thing and remove 80% of
2196600	2200840	vulnerabilities, I think that'd be worth doing. So this might be something you might want to
2200840	2206580	take a look at. Here's some example products. There's three products along the top that are
2206580	2214500	commercial. OWASP has an entire list of code analysis tools at that link right there. Again,
2214500	2219900	not meant as recommendations, but these were tools that I've come across in my journeys,
2219900	2226140	and they might be worth at least acting as a starting point for your own investigation.
2227200	2233760	So the second goal in our hat trick, the second piece to the tripod, is dynamic code analysis.
2233760	2240500	This was also in the SDLC pyramid. So unlike static code analysis, dynamic code analysis
2240500	2246860	actually executes the code. Now that has a prerequisite that the code has to be compiled
2246860	2252540	and free of errors and can run before you can actually perform the dynamic code analysis.
2252720	2256460	Dynamic code analysis, I mentioned this before, is different than pen testing,
2256460	2261180	whereas dynamic code analysis looks for the flaws in the code that can be exploited.
2261920	2268000	Pen testing looks for ways to exploit the system in general, often at a greater detail,
2268000	2275100	and almost always involving a human attacker. So again, two different things. They're kind of
2275100	2282540	similar, but I really think that dynamic code analysis is really, at best, a subset of a good
2282540	2291580	penetration test. So dynamic code analysis tools can identify up to 85%, but if you combine them
2291580	2299300	with the static code analysis tools on the market, up to 95% of your application's vulnerabilities
2299820	2307720	can be identified early on. 95%, that's pretty darn impressive. So again, I'm going to throw a
2307720	2314340	few products up here for you to take a look at. I've used several of these. Again, not
2314340	2319900	recommendations, but a place for you to start. Give you a word of warning. I think I told you
2319900	2324140	earlier on, I would tell you some things that work and some things that just didn't seem to
2324140	2331220	work quite so well. Some of these tools can be weapons. I'm looking at you, Burp Suite.
2332340	2338440	You need to do things like make sure you set the scope of your testing tool so it doesn't
2338440	2344280	see that, oh, I've got a link to Facebook here. I will jump and follow that link in my crawl and I
2344280	2352220	will start attacking Facebook. Set your scope absolutely to your application only. Run this
2352220	2359400	stuff in test. Don't run it in production. If you find a SQL injection exploit and you start
2359400	2366000	to execute it, it's very, very likely that you're going to corrupt or lose data on the database on
2366000	2373240	the backside of that injection vulnerability. Also, watch out for forms. So let's say you're
2373240	2379880	doing some dynamic testing and it's automated and you're getting ready to essentially attack a web
2379880	2387380	form. Make sure you find out where that web form is going before you start your test. You can easily
2387380	2395000	overrun a service desk by submitting thousands upon thousands of emails from a web mail form.
2395000	2400480	Not saying that I've ever done that, but I know that it has happened. So the third goal in our
2400480	2409140	hat trick is open source analysis. Open source is awesome. We all use open source. Some estimates
2409140	2417940	put the percentage of open source in internally developed applications at 70%. That's 70%.
2418600	2425800	Now the problem with that is, as a result, about 50% of our applications contain one or more open
2425800	2436740	source pieces that has a high security vulnerability. And keeping that out of production is a huge,
2436740	2442700	There are two different approaches I've taken when I've tried to address this problem. The first was
2442980	2447440	a firewall slash proxy solution. There are products out there that you can actually set up that will
2447440	2456420	proxy the main repos out in the wild. And they'll check as you pull down code libraries, they'll
2456420	2462680	check them for license compliance. So you can say, hey, I'm okay with this license, this license,
2462680	2468200	but I don't want that license. So you can set that up ahead of time and it'll check each time
2469020	2473540	a module is pulled down that it is in compliance with your licensing. But more importantly,
2473540	2479520	you can also set a vulnerability threshold. So you could, for example, say, hey, I don't want
2479520	2486200	any critical or high translation. I don't want any CVE 7 or higher coming into my organization's
2486200	2493640	local repository. So the problem with that is a lot of the tools that developers use,
2493640	2499440	particularly in DevOps for automating building of systems, automating testing,
2500220	2505140	have vulnerabilities because they really weren't meant to go live anywhere. They were basically
2505140	2514360	developer aids, developer tools, and your policy is going to quarantine or prevent them from coming
2514360	2521340	into your local repository. The result is you get a lot of phone calls. Hey, this module, I need it,
2521340	2527920	I need it now, and it's quarantined. Well, that requires getting out on the firewall or proxy
2528500	2534340	and looking at what's quarantined, determining how it's going to be used, where it's going to be used,
2534340	2539820	and does that CVE apply in this use case, and then deciding whether you can waive it or not.
2539820	2550220	It is a labor-intensive process. I didn't really like doing it this way at first,
2550220	2556920	so I ended up moving to a build-based solution. Build-based solutions are very similar to what
2556920	2561380	we're going to see in just a few minutes, where you basically add an open source review step to
2561380	2568060	your automated build process. If you find that same vulnerability as it's going to production,
2568060	2577180	let's say it's a CVE of 8, and it's going into production, you fail the build. At that point,
2578600	2584880	the developers have the opportunity to go, okay, I need to get a security waiver, or I need to find
2585140	2592820	a more recent or more updated library that doesn't have that CVE as an outstanding issue.
2592820	2600180	So this ended up reducing the manual intervention. The only problem is you could end up having
2600180	2607240	something on your internal network with a CVE of 7 or higher that's a developer tool. You've got
2607240	2612800	to be very careful when you're whitelisting or removing items from quarantine to understand how
2612800	2619700	it's going to be used, and if the CVE itself can be exploited in that manner. So here's some
2619700	2626860	products once again. OWASP, I've mentioned them several times. Great organization. Again,
2626860	2630600	these are not recommendations. These are simply places for you to start if you haven't
2630600	2637980	looked at these already. And finally, let's talk about automation. So those last three steps,
2637980	2645280	they seem like a lot. And originally we were doing them manually, and it became unwieldy
2645280	2654880	very quickly. It just took a lot of time. It took a lot of effort. So what we were able to
2654880	2660420	do was automate most of this into the build pipeline. So if you have a continuous integration
2660420	2666080	tool, something like Jenkins, you can add steps that execute static code analysis, dynamic code
2666080	2672520	analysis, and open source analysis with every build. So this gives you all of that protection
2672520	2679780	over 95% from the studies with every single build that you do. Now, there's some things that are
2679780	2686900	just too stinking big. So if you're still dealing with legacy applications, big old monolithic jars
2686900	2694660	and ears, if you've got something that's somewhere in the hundreds of thousands of lines of code,
2694660	2700760	sometimes close to a million lines of code, this is not going to go well for you because
2700760	2707200	the dynamic scan, the static scan, those pieces are going to take a long, long time.
2707200	2715900	I tested one application for a friend of mine just to try it. It was a legacy application,
2715900	2723380	and the scan for just the static code analysis piece took over 12 hours. You can't stop a build
2723380	2729500	for 12 hours and expect the developers not to be concerned. So I don't recommend this
2729500	2738840	for large applications. If you do, you might do it as an asynchronous build step. So basically,
2738840	2743240	it says, hey, I've got to go do static code analysis. I'm going to launch it and I'm going
2743240	2749940	to continue with my build. You lose the advantage of being able to block at that point, but you
2749940	2756180	still have an automated scan going along. You'll need to have a process in place to alert the
2756180	2761520	development team as well as information security of the results of that scan, and then decide how
2761520	2767820	you want to react. But you lose the ability to block. But fortunately, the industry is really
2767820	2775380	moving towards smaller builds, APIs, microservices, things like that. And inline scans are very,
2775380	2780340	very possible. They're very fast. These tools are tuned for this kind of work,
2780340	2788280	and they run incredibly, incredibly fast. Just aside, some of the tools
2789740	2796080	are still working on their automated pipeline plugins. So if you are looking at any of the
2796080	2801300	tools, whether it's the ones I've mentioned or something else, make sure that you ask about
2801300	2806700	the maturity of their continuous integration pipeline plugins.
2808340	2813400	Otherwise, you might find yourself stuck manually doing this for a while until you can find a way
2813400	2822060	to do that. I want to end with a couple of final thoughts. So the keys to success, hopefully you
2822060	2828800	can read this, they are start small. Do one thing. For me, it was starting with a training program
2828800	2835320	that literally led to everything else that you saw in this presentation. And as you saw in the
2835320	2843400	presentation, this stuff tended to feed upon itself. The developer training led to more
2843400	2849260	developer training, which led to a security program. So start small, and it will all come
2849260	2855800	together. Get management buy-in by both Information Security and by AppSec Management.
2856460	2862060	The centerpiece right there, that has to be the key for me. It has to be fun. When this stuff is
2862060	2867240	no longer fun, I'm not going to do it anymore. We've got to keep making it fun, particularly for
2867240	2872780	our developers. Developers are doing a lot of stuff. Adding security on top of everything else
2872780	2879980	they're doing is a pain in the butt. And for the most part, it's not welcome, unless you can make
2879980	2886120	it fun. If you can make it fun, it's going to happen. So just like we just mentioned, automate
2886120	2890940	everything you possibly can. And remember, it's all about those relationships. The relationships
2890940	2898320	with your scrum teams. It's the relationships with your developers. It's the relationships
2898320	2905400	with your vendors and the trainers that you can get to come in. So building a good relationship
2906080	2914080	is key. Don't try to reinvent the wheel. The best innovation is usually stolen from somewhere,
2914080	2919640	or given from somewhere. So if you can take one piece from this discussion and implement it in
2919640	2927760	your organization, I will feel blessed, and I'd love to hear about it. Take it slow. This stuff
2927760	2935340	is new for all of us, and it's always changing. And finally, learn what works best in your
2935340	2942540	environment. So I want to wrap up with thank yous. I want to thank you for spending the last 40,
2942540	2948060	45 minutes, whatever it ended up being, with me and with AppSec Village. I want to thank AppSec
2948060	2953260	Village. I got to volunteer there last year. I ended up spending the entire DEF CON there.
2953880	2958860	After, I think that was my sixth year in information security, I finally found my home,
2958860	2964480	and that is the AppSec Village. So thank you to the leadership of AppSec Village for doing this,
2964480	2971600	and for giving me this opportunity. I want to reiterate my thanks to Jim Manico, and he's kind
2971600	2978240	of been my spirit leader on this journey, and I appreciate all you've done. My colleagues Barry
2978240	2984340	and Shane, my partners in crime, you guys are awesome. And finally, my best friend, my wife,
2984340	2991520	who's put up with me doing this and several other things in information security. She's absolutely
2991520	2997880	the best, and I can't thank her enough. So thank you guys very much. I appreciate you spending this
2997880	3005580	time with me, and I hope you have a really, really excellent rest of your DEF CON. Thanks.