00:00:00.208-->00:00:05.167
>>We are Dan and Ben from Bishop
Fox. Uh we are penetration
testers. Uh we spend most of our

00:00:05.167-->00:00:09.417
days breaking into things like
web applications, networks,
occasionally cooler things like

00:00:09.417-->00:00:14.417
safes. Um and uh like you we’ve
been hearing a whole lot about
this AI business lately. >>Yeah

00:00:16.625-->00:00:21.167
I’m sure you’ve heard in the
news uh seems like every week
there’s a new article in the

00:00:21.167-->00:00:26.792
news about how you know AI is
going to start driving our
trucks and cars on the roads.

00:00:26.792-->00:00:31.958
It’s gonna start you know flying
our drones and uh its gonna
basically take all of our jobs.

00:00:31.958-->00:00:37.208
Um, so as pen testers we are
interested in how we can use
this uh technology and these

00:00:37.208-->00:00:42.250
these tools uh to maybe do some
offensive security stuff. Uh
we’ve heard a lot about

00:00:42.250-->00:00:47.292
defensive security applications
for machine learning uh there’s
things like firewalls out there

00:00:47.292-->00:00:53.208
that can do uh anomalous traffic
detection, those kinds of
things. Um but there weren’t too

00:00:53.208-->00:00:58.208
many uh offensive tools and so
we really wanted to explore that
space and uh this talk is

00:01:01.250-->00:01:07.667
basically gonna be about the
intersection between uh AL
programming and uh hacking. So.

00:01:07.667-->00:01:11.250
>>Yeah, so to give you a
concrete example of this, we
built a tool called Deep Hack.

00:01:11.250-->00:01:15.708
Uh that breaks into the uh
backend database of a web
application using nothing but

00:01:15.708-->00:01:20.583
machine learning. So the the
scenario here doesn’t in order
to better understand what you’re

00:01:20.583-->00:01:26.708
about to see is an attacker
taking uh an attack over the
internet um though for this bank

00:01:26.708-->00:01:31.500
website just say that has a
backend SQL uh database that has
a a what’s called a Boolean

00:01:31.500-->00:01:35.375
based blind SQL injection, so
just in case you’re not familiar
with how that works. It’s

00:01:35.375-->00:01:40.000
something like being able to ask
twenty questions with a backend
database. So you can do

00:01:40.000-->00:01:45.083
something like, could ask hey is
uh like the user’s first name
starting with an f? And it’ll

00:01:45.083-->00:01:50.667
say no. And say do does it start
with a d? And you say no. Uh you
ask well does it start b and

00:01:50.667-->00:01:54.875
it’ll say yes. And so you can
enumerate the entire database
based off of this sort of thing

00:01:54.875-->00:02:01.208
and there are tools that do this
sort of thing um but uh none
quite like how we did it. So um

00:02:01.208-->00:02:05.833
in the demo you’re about to see
um this is all just taking place
on one computer. My laptop. Um

00:02:05.833-->00:02:10.500
but it uh you know uh
importantly here um the tool
doesn’t have any say insider

00:02:10.500-->00:02:14.667
information about the backend
databases. Nor does it have any
run time information about like

00:02:14.667-->00:02:18.375
the web server itself. >>Right
it’s taking the perspective of
an attacker on the public

00:02:18.375-->00:02:23.375
internet, this is just one of us
you know. >>Yeah, completely
black box. So. Demo. So um what

00:02:32.333-->00:02:36.083
you’re seeing um right here, I’m
gonna just pause this for a
moment here is this sweet

00:02:36.083-->00:02:40.333
visualization. Now hopefully you
can read some of the text um
that we made. Um that has on the

00:02:40.333-->00:02:45.875
bottom a part are the requests
that are being made out to the
remote server um so you can see

00:02:45.875-->00:02:51.375
that its querying the database
character by character. Um
asking for information from it.

00:02:51.375-->00:02:54.958
And up on the top there are are
like the discovered strings
basically. That’s the

00:02:54.958-->00:03:00.083
information that its received
out from the uh remote and uh
endpoint. And you can see um in

00:03:00.083-->00:03:03.958
a couple of cases uh there, it’s
start to kind of freak out a
little bit and that’s not what

00:03:03.958-->00:03:07.500
you would expect it to be doing,
we’re actually gonna come back
to that a little bit and explain

00:03:07.500-->00:03:14.375
exactly why the the tool kind of
doesn’t always work exactly how
you’d expect it to. Um and it

00:03:14.375-->00:03:18.750
can uh continue on this process
brute forcing character by
character um stealing

00:03:18.750-->00:03:23.625
information out of the database
until it has basically the
entire thing. So the bp- couple

00:03:23.625-->00:03:30.167
of like important pieces here is
that our uh tool um was never
like programmed by us like we

00:03:30.167-->00:03:34.792
never programmed DPAC. It
learned how to do all of this.
And that’s gonna be pretty

00:03:34.792-->00:03:41.042
important going forward, both
for pen test, by tools and
hackers in general. >>Yeah, if

00:03:41.042-->00:03:44.667
you look through the code base
uh there is no SQL string that
you know says like select count

00:03:44.667-->00:03:50.292
from table uh there is no hard
coded SQL logic uh its all just
using uh artificial intelligence

00:03:50.292-->00:03:57.250
to kind of understand how that
works. >>Yeah so we might say it
has uh no tricks up our sleeve.

00:03:57.250-->00:04:01.500
Uh so uh we’ll be releasing the
source code to all this stuff
immediately following the

00:04:01.500-->00:04:05.833
conference and you’ll be able to
see for yourself that its about
two hundred lines of Python, its

00:04:05.833-->00:04:11.583
not some massive piece of uh new
technology that like never been
discovered before. We don’t have

00:04:11.583-->00:04:15.042
any major breakthroughs in
machine learning. This is
basically taking existing

00:04:15.042-->00:04:19.625
machine learning uh methodology
um and state of art and applying
it to the world of hacking. Um.

00:04:19.625-->00:04:25.708
So uh notably like Ben said,
there aren’t anything like any
SQL strings, there aren’t any

00:04:25.708-->00:04:30.042
SQL structures um any
information about web
applications or databases at all

00:04:30.042-->00:04:34.042
inside of the uh uh the
underlying source code thing
that actually is running this.

00:04:34.042-->00:04:39.875
Um uh the entire concept of
brute forcing things character
by character, the entire concept

00:04:39.875-->00:04:43.458
of what databases are and
information behind that are all
things that the machine has to

00:04:43.458-->00:04:48.458
learn it has to discover that on
its own. Um importantly and also
along the lines of uh no tricks

00:04:50.708-->00:04:56.125
up our sleeve here is that this
might see sort of magical to you
at this point, um and that’s

00:04:56.125-->00:04:59.833
almost on purpose that imagine
this whole presentation like a
magic trick where we reveal how

00:04:59.833-->00:05:05.042
we did it at the end, um where
this might seem kind of special
and scary to you if its magic

00:05:05.042-->00:05:10.000
then its black magic, which is
to say, is the kind that hacks
into stuff. Um but I think by

00:05:10.000-->00:05:13.875
this hopefully you’ll find that
not only is this quite
approachable but in fact of

00:05:13.875-->00:05:18.042
machine learning, rather
pedestrian. If there are any um
data scientist um in the

00:05:18.042-->00:05:22.417
audience um I think they might
find themselves uh uh like they
could have done that themselves

00:05:22.417-->00:05:29.333
and hopefully that’s in fact
what you come taking away from
this. >>Yeah, so uh are are

00:05:29.333-->00:05:33.667
program differs a little bit
from a regular program in a
couple of really important ways.

00:05:33.667-->00:05:39.417
Um in a regular program, uh you
write some source code which is
just gonna be in you know

00:05:39.417-->00:05:44.375
English and uh you’re going to
basically be compiling it down
to a binary which is you know

00:05:44.375-->00:05:47.833
machine instructions its the
thing your computer is actually
going to run and uh those

00:05:47.833-->00:05:53.958
machine instructions are gonna
perform an action uh based on
what you programmed into it. And

00:05:53.958-->00:05:58.917
uh there’s a relationship
between uh the source code and
the action you eventually take.

00:05:58.917-->00:06:03.083
Uh there’s kind of this one to
one mapping so uh if you have
enough time, you can sit there

00:06:03.083-->00:06:08.542
and reverse engineer any action
you’re program is taken back to
a line of source code. Uh there

00:06:08.542-->00:06:14.750
is uh its very very
deterministic path back to uh
you know what actually decided

00:06:14.750-->00:06:19.750
on that action. Um an AI program
differs slightly um in that uh
there is still binary and

00:06:21.875-->00:06:27.750
there’s still program that’s
being written, but it takes in
data and uh runs it though uh

00:06:27.750-->00:06:33.125
what’s called a model and uh the
model is kind of what decides an
action. So you’re not gonna be

00:06:33.125-->00:06:38.042
uh programming you know
conditional logic in there you
know if if we’re in state A,

00:06:38.042-->00:06:43.375
then we’re gonna take action A,
else we’re gonna take action B,
right? There uh, there isn’t so

00:06:43.375-->00:06:49.708
much of that an the model will
actually uh decide an action
based on probability and uh and

00:06:49.708-->00:06:54.500
and just what data it has seen
in the past. And that’s kind of
where the learning kicks in uh

00:06:54.500-->00:06:59.500
it can use this data that you
supply it with uh to kind of
learn what has happened in the

00:06:59.500-->00:07:04.917
past and uh decide what actions
its gonna take in the future. So
there’s not not its not

00:07:04.917-->00:07:09.292
necessarily as deterministic as
a regular program, it can be
kind of difficult to d- to

00:07:09.292-->00:07:14.250
actually ask ol- well why did
this you know program make the
decision it did? Why did it

00:07:14.250-->00:07:18.375
decide to you know select star?
Or why did it decide to drop my
database instead of you know

00:07:18.375-->00:07:23.125
injection and steal all the data
from it? Uh its one of the
fundamental differences. >>Yeah,

00:07:23.125-->00:07:27.333
so we’re gonna be talking quite
a bit about the model going
forward and that’s the important

00:07:27.333-->00:07:30.833
piece to take away here. But
that’s the thing that does all
the smarts in your program as

00:07:30.833-->00:07:35.833
opposed to the code that you
actually write. Um so uh >>Yeah,
its uh so so I when I wa- was uh

00:07:38.125-->00:07:42.208
learning how to do this stuff
um, I I I know a little bit of
math, but I’m not you know a

00:07:42.208-->00:07:47.292
heavy heavy math background so I
kind of, kind a had to take a
different mindset to um to the

00:07:47.292-->00:07:51.000
programming and I kind of
consider a different paradigm,
you kind of have to formulate

00:07:51.000-->00:07:54.750
your problem a little bit
differently. Kind of uh how
functional programming is a

00:07:54.750-->00:07:58.958
little bit different than uh
object oriented programming or
imperative uh programming. There

00:07:58.958-->00:08:01.833
it’s kind of like a differing
paradigm, you definitely have to
think about it slightly

00:08:01.833-->00:08:08.500
differently. Um. >>So uh I
described um machine learning
and artificial intelligence as a

00:08:08.500-->00:08:13.000
magic trick earlier and if its
magic then let’s try to
de-mystify that a little bit. Um

00:08:13.000-->00:08:17.792
by learning a little bit about
how this works um then uh maybe
you won’t be quite so scared of

00:08:17.792-->00:08:22.083
it once you discover um exactly
how this works. There’s this
fantastic cartoon I wish I could

00:08:22.083-->00:08:25.917
have put on here but the text
was too small to read. Of like
the Scooby Doo gang going just

00:08:25.917-->00:08:31.042
like pull the ma- the the wig
off of some sort of a monster,
it’s like, oh no machine

00:08:31.042-->00:08:36.917
learning, it pulls the mask off,
s- stochastic gradient descent,
it was you all along! Um uh so

00:08:36.917-->00:08:40.792
yeah something along those
lines. Eh- I definitely get a
universal sense of fear from

00:08:40.792-->00:08:45.458
people that I talk to this
about. Even my own mother. Um
where I I described uh she knows

00:08:45.458-->00:08:49.083
I do a bunch of like research
projects and things like that.
And uh not all that long ago she

00:08:49.083-->00:08:53.042
was asking me, like what are the
th- like what are you working on
these days? And I said oh you

00:08:53.042-->00:08:57.125
know making this like AI program
that learns to hack into a
computer and she just goes oh

00:08:57.125-->00:09:02.125
no. [laughter] Ha, why would you
do that? That’s ha. That’s how
it starts. Like for whatever it

00:09:04.917-->00:09:10.375
is. I don’t even know. Um yeah.
So if if you um and uh Elon Musk
are concerned about the same

00:09:10.375-->00:09:15.042
sorts of things then you know
just just keep watching, don’t
worry about it. Um so we have

00:09:15.042-->00:09:18.833
machine learning. Uh so supposed
we want to solve a much simpler
problem. We want to solve this

00:09:18.833-->00:09:23.583
maze right here. We want to make
a robot that can go up and over
and solve the maze and not fall

00:09:23.583-->00:09:29.583
into the lava or depending on
how you look at it um angry red
birds. Um. >>They’re lava birds

00:09:29.583-->00:09:35.125
really. >>Yes, lot lot a birds.
So [inaudible]. So behold the
table. This is the most simple

00:09:35.125-->00:09:40.667
uh machine learning uh algorithm
possible um basically you just
have your robot take actions and

00:09:40.667-->00:09:46.250
record the reward that it gets.
So for instance it might uh take
uh a step a starting at the the

00:09:46.250-->00:09:50.208
starting position here, take a
step left fall into the lava and
go d**n, that’s minus fifty

00:09:50.208-->00:09:53.542
points or I don’t know it just
arbitrarily knows that fifty
points is what you get for

00:09:53.542-->00:09:58.208
falling into the lava. Um and
then it knows to not try that
again or at least it knows to

00:09:58.208-->00:10:02.667
try to explore some other
states, so it might then try
moving upward and get minus one

00:10:02.667-->00:10:07.917
and say hey that was a little
bit better. And so now that it’s
in the next square it can go try

00:10:07.917-->00:10:12.375
you know turning right fall into
the lava again, God d**n it.
Gets another minus fifty points

00:10:12.375-->00:10:16.750
um and then basically just
continue on this entire process
exploring things that it hasn’t

00:10:16.750-->00:10:21.583
explored before until its found
its way to the start and it can
derive the optimal solution. And

00:10:21.583-->00:10:26.917
this is basically it. This is
machine learning. Uh you basic-
you just keep on trying things

00:10:26.917-->00:10:32.042
you record what you found until
you you know find a good way to
the exit. Um there’s only one

00:10:32.042-->00:10:36.417
problem with this. >>Yeah, the
problem is uh when you want to
do something that’s a little

00:10:36.417-->00:10:40.792
more complicated uh for example
maybe we’ll instead of solving a
maze, we’ll play a game of

00:10:40.792-->00:10:47.500
chess. Uh the problem is chess
is a little more complicated
than our maze and it holds quite

00:10:47.500-->00:10:53.417
a bit more state um as you can
see there’s about ten to the
forty-seven states in a game of

00:10:53.417-->00:10:57.792
chess and uh there’s only
sixteen pieces or you know
sixteen pieces on each side. So

00:10:57.792-->00:11:02.792
uh there’s gonna be uh a lot of
data to fit into our table. It’s
gonna be very difficult, pretty

00:11:02.792-->00:11:07.417
much impossible to you know
create a table. Enumerate every
single possible state and uh

00:11:07.417-->00:11:13.000
decide what the best course of
action is gonna be. So uh one of
the one of the key points we’re

00:11:13.000-->00:11:17.917
gonna try to do is er- key
things is we’re gonna try to do
is uh try to scale down uh the

00:11:17.917-->00:11:22.750
idea. We’re still gonna keep the
idea of choosing a random
actions figuring uh if it was

00:11:22.750-->00:11:28.375
good or bad and kind of
recording that, but we’re gonna
wanna scale it down so we can uh

00:11:28.375-->00:11:33.750
you know explore state without
having to store the entire you
know game in our uh in our

00:11:33.750-->00:11:37.042
computer at the same time.
>>Yeah, we have pretty beefy
computers but it can’t hold all

00:11:37.042-->00:11:44.833
that. Um so the main uh uh
breakthrough here the main idea
behind like actual machine

00:11:44.833-->00:11:49.375
learning as it happens like in
you know any normal sense is
that this this table is

00:11:49.375-->00:11:54.417
basically just one of these.
Which is to say a function. Uh a
mathematical function to sort of

00:11:54.417-->00:12:00.125
take numbers in and take numbers
out in a predictable way. And so
what we can do with functions?

00:12:00.125-->00:12:05.625
We can approximate them? And so
uh the we have uh a following
great little explanation of kind

00:12:05.625-->00:12:10.167
of how um function approximation
works without getting into some
of the more nitty gritty

00:12:10.167-->00:12:13.833
details. I suspect some data
scientists that are in the room
might be groaning um to

00:12:13.833-->00:12:18.292
themselves. But actually by the
way, uh data scientist is sort
of like what the AI community

00:12:18.292-->00:12:24.542
has decided to call themselves.
Its a really terrible name. Uh
like they’re no- they’re

00:12:24.542-->00:12:29.583
scientists nor have anything to
do with data really. Like if
anything to do with data

00:12:29.583-->00:12:31.208
engineers, like come on, lets be
straight. And if anything its
like actually like AI engineers.

00:12:31.208-->00:12:37.125
Get your s**t together AI
researchers. [laughter] Um so
this is usually in the point in

00:12:37.125-->00:12:41.875
these sorts of conversations
where things get really messy.
And mathematicians break out

00:12:41.875-->00:12:47.000
scary language like linear
regression and convolutional
transposition. Let’s scare away

00:12:47.000-->00:12:52.708
all the norms. Um but you don’t
actually need to know any of
this uh the frameworks are

00:12:52.708-->00:12:56.750
actually really helpful today
where you can write an AI
program without knowing a lick

00:12:56.750-->00:13:02.208
of like any of this. Um which is
the way the rest of computers
work. You don’t actually need to

00:13:02.208-->00:13:05.833
know anything of like Shannon
entropy to write a God d**m
Python program right? Like you

00:13:05.833-->00:13:12.583
can use a crypto library without
knowing any of about RSA. And so
the same is true just finally

00:13:12.583-->00:13:18.083
like recently eh- within AI
frameworks. That’s kind of why
this has been taking off. ‘Cause

00:13:18.083-->00:13:21.792
it’s no longer something that
MIT does in there like spare
time. this is something that you

00:13:21.792-->00:13:29.792
actually can do today. in a
reasonably short amount of time.
>>Yeah so uh so function

00:13:29.792-->00:13:35.417
approximation. Uh basically
we’re gonna be trying to guess
what the function uh that drew

00:13:35.417-->00:13:40.375
this line um is graph is going
to be. So uh the easiest way to
start with that is to draw the

00:13:40.375-->00:13:45.708
worst function approximation
possible which is just a
straight line. Uh this

00:13:45.708-->00:13:51.083
essentially represents uh your
first sample you have one data
point. You’re gonna take one

00:13:51.083-->00:13:53.625
action. This is the first one.
Its obviously gonna be pretty
terrible. Um but the great

00:13:53.625-->00:14:00.417
things is uh you’re basically
going to just uh take another
sample and you’re going to

00:14:00.417-->00:14:06.042
another sample and you- each
time you’re gonna plot you know
your data on your graph and uh

00:14:06.042-->00:14:12.792
you’re just gonna keep on
sampling until you kind of start
getting close to your function.

00:14:12.792-->00:14:17.083
Um its really hard to get
perfect especially the you’re in
the real world and you know

00:14:17.083-->00:14:20.833
you’re trying to model something
that’s really complex. Uh but
you can get really close and

00:14:20.833-->00:14:27.375
this idea of uh taking a random
action or taking a you know a a
smart action and uh seeing what

00:14:27.375-->00:14:32.333
happens, plotting it on your
graph and uh just repeating this
process of sampling over and

00:14:32.333-->00:14:36.708
over again is is kind of the
fundamental uh thing you’re
gonna be doing when you’re

00:14:36.708-->00:14:40.875
building your AI program. >>And
more fundamentally, this is
almost the very definition of

00:14:40.875-->00:14:50.583
learning isn’t it? What you’re
doing is taking in information.
You’re changing your model of

00:14:50.583-->00:14:53.083
the world not he base of it and
then changing your behavior
based off of the new things

00:14:53.083-->00:14:54.833
you’ve seen. Iteratively getting
closer and closer towards the
behavior that you would like.

00:14:54.833-->00:14:59.458
Like I just described learning
in a nutshell right? This is
just a math- mathematical uh way

00:14:59.458-->00:15:04.458
of representing that concept of
learning. So however we try to
do um AI and machine learning

00:15:04.458-->00:15:11.917
in the future it will probably
some form of function
approximation. >>Yeah so uh

00:15:11.917-->00:15:19.167
there’s kind of three things you
want to think about uh when you
start saying well ok, well how

00:15:19.167-->00:15:21.667
do I, you know approximating
functions, it sounds kind of
complicated. How do I actually

00:15:21.667-->00:15:24.333
start you know writing code, how
do I start getting you know
getting into this. Uh you want

00:15:24.333-->00:15:29.833
to think about three things for
your program, uh there’s
basically your input uh which is

00:15:29.833-->00:15:34.458
going to be your environment.
Its essentially everything your
AI program knows about the

00:15:34.458-->00:15:39.917
world. Um so you know for uh I’m
gonna go into the next slide uh
we’re gonna talk about chess a

00:15:39.917-->00:15:44.792
little bit, uh it’s going to be
you know everything your AI
program knows and its the only

00:15:44.792-->00:15:49.625
thing it really sees. So um
you’re gonna want to formulate
your inputs uh into you know uh

00:15:49.625-->00:15:55.500
not present too much data, but
also just present enough of the
world to be able to make smart

00:15:55.500-->00:16:01.750
decisions about it. Um you’re
also gonna have a reward so when
you take an action its either

00:16:01.750-->00:16:06.417
going to uh do something good,
do something bad or maybe do
something kind of good or bad uh

00:16:06.417-->00:16:12.500
depending on you know what your
problem is. Um and then your’e
gonna have your output. And

00:16:12.500-->00:16:16.250
that’s gonna be essentially what
action your’e gonna take. Um so,
if you wanna go to uh yeah, uh

00:16:16.250-->00:16:23.458
so you know if we’re gonna have
a real concrete example here for
our game of chess, our input

00:16:23.458-->00:16:29.833
would be you know the piece
positions of all the pieces on
the board. So um you know you

00:16:29.833-->00:16:35.875
could put those in an array and
just kind of uh just have the be
the entire state of the you know

00:16:35.875-->00:16:43.292
the program. Um you rewards are
gonna be you know if you move a
piece uh what uh what happened?

00:16:43.292-->00:16:46.667
Right if you take a piece you’re
gonna be getting some reward,
you’re gonna get that carrot

00:16:46.667-->00:16:51.208
right? The reward is you know,
your carrot on a stick, carrot
or stick um so if you capture a

00:16:51.208-->00:16:56.625
piece you’re gonna be getting
some reward and if you lose a
piece you’re gonna be uh gaining

00:16:56.625-->00:17:01.458
some negative reward or losing
some negative reward. And uh if
you move perhaps without

00:17:01.458-->00:17:06.208
actually taking a piece uh it
might be just a slight negative
reward because you didn’t really

00:17:06.208-->00:17:11.125
accomplish anything but uh you
also didn’t lose anything uh too
great either. Just some time.

00:17:11.125-->00:17:16.458
And then uh like I said, you
output is essentially going to
be the action uh wha- that

00:17:16.458-->00:17:21.625
you're gonna take. For uh your
game. So uh. >>Yeah, uh and
that’s really about it. If

00:17:21.625-->00:17:29.125
you’re wondering in the audience
right now or at home, Gerben, if
you uh can uh try to make some

00:17:29.125-->00:17:36.458
sort of a program that can uh
you know take advantage of
machine learning. Try to think

00:17:36.458-->00:17:39.417
about it in terms of these three
things um input which is a, what
are the things that my program

00:17:39.417-->00:17:42.917
needs to see in the universe.
Output, what are the actions
that it should be taking and

00:17:42.917-->00:17:45.792
then rewards. How does it know
um when it got, when it did
something good or how does it

00:17:45.792-->00:17:49.292
know when it did something bad.
Um if you can think of your
problem effectively in terms of

00:17:49.292-->00:17:52.208
those three things, then you can
make a machine learning program
that will solve your problem. Um

00:17:52.208-->00:17:57.667
so getting back to our example.
Getting back to DeepHack
specifically. Um it works a

00:17:57.667-->00:18:03.375
little bit like this. Um this
might seem at first like uh an
unintuitive way of formulating a

00:18:03.375-->00:18:07.333
problem but its actually a
pretty standard construction in
terms of text generation. So the

00:18:07.333-->00:18:10.958
input here to the system is an
incomplete query string. It’s a
partial string of text that its

00:18:10.958-->00:18:16.833
generating um and the output the
thing that it needs to decide,
the thing that it wants to

00:18:16.833-->00:18:21.417
figure out is what is the next
character in this sequence? So
if you’re looking at our example

00:18:21.417-->00:18:27.583
here, it says select star fro
and then even in the audience
you can probably thing, well M

00:18:27.583-->00:18:31.333
makes sense. Right? If we’re
talk especially if you’re
talking about SQL then it should

00:18:31.333-->00:18:36.792
be like star from. And w hat you
do is you take that M, you stick
it onto the end of the query

00:18:36.792-->00:18:39.250
string and then you just repeat
this process iteratively. And in
terms of rewards, how does it

00:18:39.250-->00:18:41.917
know like whether it did
something right or whether it
did something wrong? Is again

00:18:41.917-->00:18:46.375
based off of the output from the
remote server. So all we have,
information about are the

00:18:46.375-->00:18:50.958
Boolean based uh responses from
the web server. So did this give
a two hundred or did it give us

00:18:50.958-->00:18:56.083
a five hundred. Um which to say
like is that data correct or is
it incorrect? And so entirely

00:18:56.083-->00:19:01.125
from that basis we can figure
out um whether the sting is um
present in the database or not

00:19:01.125-->00:19:05.750
or whether this is something
that we want or whether it isn’t
Um so as a good tangible example

00:19:05.750-->00:19:11.125
of this. Um there’s uh, you can
imagine like we do this in uh
English all the time. In fact,

00:19:11.125-->00:19:14.875
you do this in your cell phone
all the time and so we’re gonna
play a little game here. I want

00:19:14.875-->00:19:19.375
you to bust out with your cell
phones right now. And open up
your text message app, because

00:19:19.375-->00:19:24.167
your cell phones do this
currently right now. Um pretty
well. So you’re gonna open up a

00:19:24.167-->00:19:29.958
text message app, whatever that
happens to be. Probably signal
‘cause we’re hackers right?

00:19:29.958-->00:19:33.208
You’re gonna go to somebody that
you really like or maybe you
really hate. And then just type

00:19:33.208-->00:19:38.208
something in, so like i’m at Def
Con space and then it’s gonna
give you a series of

00:19:38.208-->00:19:42.375
recommendations, which is like
you’re machine learning
assistant in your phone telling

00:19:42.375-->00:19:46.583
you what it things should come
next. Just go ahead and push one
of the buttons. Pick left,

00:19:46.583-->00:19:57.208
middle or right. Right and then
just keep on pressing the button
and then just keep on pressing

00:19:57.208-->00:19:59.583
the button. And you’ll notice
that it starts to generate some
pretty hilarious strings. Oh

00:19:59.583-->00:20:02.417
yeah my phone is going awesome.
I’m gonna read you this in just
a moment and the trick here is

00:20:02.417-->00:20:06.708
you have to press send. That’s
part of the rules of the game,
you agreed to it. Yeah, I’m

00:20:06.708-->00:20:12.708
pressing send. I’ve got, I’m at
Def Con this is the first time
which is not, I have been every

00:20:12.708-->00:20:19.250
right to be mad at me for not
being to that day for last
weekend and d- this is getting

00:20:19.250-->00:20:30.250
serious. And wanted to see if I
could get a copy of the receipt
for the it was a weird night in

00:20:30.250-->00:20:38.375
Vegas last night apparently.
Haha. I’m married. [audience]
[laughter] I’ll get back to you

00:20:38.375-->00:20:44.542
on that. The receipts. And I’m
hitting send. So this is
basically um like Deep Hack is

00:20:44.542-->00:20:48.667
doing right? So its taking uh
the information from the string
it has seen before as well as in

00:20:48.667-->00:20:54.083
your cellphone case um the like
what the is in the history of
things you’ve sent in the past

00:20:54.083-->00:20:58.583
and tries to figure out a good
suggestion for that. Um and
that’s really all there is to

00:20:58.583-->00:21:03.917
it. So you can imagine here if
you have a sentence like a lot
of foo, then the machine will

00:21:03.917-->00:21:07.833
try to c- create a good
suggestion for you know what
letter should come next. Some

00:21:07.833-->00:21:12.583
good examples might be D like a
lot of food. Or maybe a lot of
foot. Maybe there’s a lot of

00:21:12.583-->00:21:16.458
football you’ve been watching or
something like that or maybe in
foosball. And so strictly

00:21:16.458-->00:21:20.583
speaking um the machine doesn’t
pick the next character um
actually what it does is produce

00:21:20.583-->00:21:25.000
a uh probability distribution
across all letters and so it
might give D a very high

00:21:25.000-->00:21:28.833
probability, T a very low
probability. And so the entire
nature of the program is

00:21:28.833-->00:21:35.042
probabilistic. And that now
leads us to why um Deep Hack um
sometimes wigs out and does

00:21:35.042-->00:21:38.958
stupid s**t. Um because if you
start out your sentence in a
text message came um with

00:21:38.958-->00:21:44.333
something non sensical or if you
just kind of like pick something
random, uh then it goes off the

00:21:44.333-->00:21:49.333
rails quickly. So if the very
first like couple of words um
have nothing to do with SQL and

00:21:49.333-->00:21:53.458
start talking about like food
then the Deep Hack has no idea
how to recover from that and

00:21:53.458-->00:22:02.750
basically just s***s the bed.
>>Yeah so uh some of the things
to keep in mind. Some of the uh

00:22:02.750-->00:22:09.458
things we ran into when we’re
trying to write our program uh
were uh having good data is

00:22:09.458-->00:22:17.875
really key. Um you want to have
good labeled data to uh feed
into your model to uh have a

00:22:17.875-->00:22:23.500
good basis um like Dan was
saying you could pretty much uh
throw Deep Hack at a a web

00:22:23.500-->00:22:32.458
server with uh no training and
it would just barf all over the
place. It would be really good

00:22:32.458-->00:22:33.833
at generating random texts and
eventually you might run into
some SQL queries you know like

00:22:33.833-->00:22:37.292
maybe some simple ones and one
equals one or whatever that
would actually work and it would

00:22:37.292-->00:22:40.500
be you know uh a light bulb
would go off in it. But that
would take a really long time

00:22:40.500-->00:22:44.417
and uh we don’t have a lot of
time on you know engagements you
want to basically train your

00:22:44.417-->00:22:53.083
model uh with uh labeled data
that you’ve collected over time
uh so your you know model can

00:22:53.083-->00:22:56.417
you know learn a little bit
about the problem you’re trying
to solve before you just unleash

00:22:56.417-->00:23:01.417
it on the world. So uh that’s
what uh bootstrapping your model
with experiences, that’s thats

00:23:01.417-->00:23:05.500
kind of what its called. Uh
you’re gonna give your uh model
some experiences um and try to

00:23:05.500-->00:23:09.875
you know teach it. Kind of like
a small child perhaps. You’re
gonna um. >>Yeah making a

00:23:09.875-->00:23:15.000
machine learning program is a
bit like having a child and
teaching it to solve your

00:23:15.000-->00:23:18.833
problem instead of you having to
do it yourself. Um it can be a
rather frustrating experience

00:23:18.833-->00:23:24.333
and get messy at times. >>Yeah,
indirect control over you know
you’re program like can you

00:23:24.333-->00:23:28.458
please maybe do that, that would
be really great if you could you
know. Start hacking that’d be

00:23:28.458-->00:23:32.917
great. >>Yeah, so turns that um
computers don’t necessarily have
to be taking action themselves

00:23:32.917-->00:23:37.792
in order to learn from the
brain, you can give it
experiences from you know other

00:23:37.792-->00:23:42.375
machines or things that its seen
in the past to try to get it um
bootstrapped into better

00:23:42.375-->00:23:46.875
behavior. Like Ben had said, um
if you just start Deep Hack from
scratch it just produces

00:23:46.875-->00:23:50.500
garbage, um it doesn’t do much
of anything else and if you give
it an infinite amount of time it

00:23:50.500-->00:23:54.000
will eventually come up with a
good SQL string. Um but that’s
not really good enough, right?

00:23:54.000-->00:23:59.333
Um so um uh the getting a good
sample of training data is
usually the difference between a

00:23:59.333-->00:24:03.625
solvable and one that isn’t. So
again in your problem, if you’re
thinking, I wonder if I can

00:24:03.625-->00:24:09.833
solve x with machine learning.
Try to think about how can I get
a metric s**t ton of data um on

00:24:09.833-->00:24:14.500
my problem that’s labeled data.
And it has to be labeled data
which is to say that you have to

00:24:14.500-->00:24:18.083
know ahead of time um what it is
and what it isn’t. So for
example if you’re making a

00:24:18.083-->00:24:22.417
computer program that tells the
difference between a hotdog and
not a hotdog, then its not

00:24:22.417-->00:24:26.500
enough to have a lots and lots
of pictures of hotdogs, you have
to have lots and lots of

00:24:26.500-->00:24:30.917
pictures of hotdogs and to know
ahead of time that they are
hotdogs. ‘Cause you are, you

00:24:30.917-->00:24:34.750
have to tell the program that
its what it is, alright its not
just gonna figure that out on

00:24:34.750-->00:24:38.833
its own. Um or you can get your
users to do it for you. This is
uh a picture of reCAPTCHA.

00:24:38.833-->00:24:44.292
>>Yeah and I’m sure you’ve
filled out a lot of reCAPTCHAs
in your day and uh you’re

00:24:44.292-->00:24:49.292
basically just feeding a machine
learning algorithm and you’re
helping train uh a you know a

00:24:49.292-->00:24:53.792
really impressive algorithm that
can decide if you’re a bot or
not and its great because uh

00:24:53.792-->00:24:58.542
you’re just kind of wringing out
your brain power and helping you
know the greater good of the

00:24:58.542-->00:25:04.125
internet pretty much um with
that. So yeah if you can coerce
your er- convince your users to

00:25:04.125-->00:25:09.125
perhaps help you out, train your
models and uh label some data
for you all the better. so.

00:25:09.125-->00:25:12.750
>>Google’s remarkably open about
the reCAPTCHA thing too. I
thought this was really

00:25:12.750-->00:25:16.000
surreptitious but no, it says
like right on their homepage.
They’re like you’re feeding our

00:25:16.000-->00:25:19.667
machine learning algorithm bro.
Like that’s that’s what’s
happening here. >>Yeah.

00:25:19.667-->00:25:22.375
>>Curiously, they actually do it
really cleverly uh in this
picture here that has the

00:25:22.375-->00:25:26.792
picture of a cat and says
selects all the images that
match this one below. And my

00:25:26.792-->00:25:29.917
brain immediately went to cats
and so I was like, well I was
like these ones are cats and

00:25:29.917-->00:25:34.292
these ones aren’t um I I was
showing this to my wife ahead of
this time and she says well uh

00:25:34.292-->00:25:37.458
everything except for the bottom
left one. And I was like what,
what are you talking about? And

00:25:37.458-->00:25:40.875
she’s like well that’s the only
one that’s not an animal. I was
like oh that’s actually really

00:25:40.875-->00:25:43.667
clever and if you read the
prompt it doesn’t say, pick all
the ones that are cats. It just

00:25:43.667-->00:25:47.833
says all the ones that are
similar to this below. And so
that way they actually get a

00:25:47.833-->00:25:52.750
really big sample of uh results
from things that are just
similar for whatever that means

00:25:52.750-->00:25:56.750
to you. And so they can get an
idea of what things are similar
to human beings without actually

00:25:56.750-->00:26:01.458
telling you the categories for
what those means. So that was
actually really clever. >>Yeah,

00:26:01.458-->00:26:05.500
very smart. >>Um, so and maybe
at this point you’re thinking
like so what. So you made some

00:26:05.500-->00:26:11.125
computer program that hacks into
a web applications database like
doesn’t them already have

00:26:11.125-->00:26:16.375
programs that do that? And
that’s true. Um but there’s uh
part of what makes DeepHack

00:26:16.375-->00:26:20.583
special and part of what uh
might make you want to make um
AI based hacking tools in past

00:26:20.583-->00:26:24.833
is there’s nothing actually in
what we were talking about
that’s specific to uh the

00:26:24.833-->00:26:28.083
underlying database type. Um in
fact part of the entire thrust
is that its agnostic to the

00:26:28.083-->00:26:35.500
underlying database type, the
structures and things like that.
So in the previous example we

00:26:35.500-->00:26:38.958
showed you um exploit uh SQLite
database because SQLite was
easy. Um um here it is again

00:26:38.958-->00:26:45.500
using um post rest. And so in
this case, you notice that the
database type is this the

00:26:45.500-->00:26:48.750
completely different this the
query that it’s making is
totally different um but it

00:26:48.750-->00:26:54.375
works nonetheless. And this is
the reason that AI is going to
take off. This is the reason

00:26:54.375-->00:26:58.125
that um this is going to be big
and the reason basically that
every major pen testing like

00:26:58.125-->00:27:04.250
tool uh certainly in 2018 but
maybe in mid 2017 is gonna start
to add this. You’re going to see

00:27:04.250-->00:27:09.083
web inspect and burp and
basically everybody else adding
machine learning um to augment

00:27:09.083-->00:27:13.667
existing tooling because it can
take care of this kind of fuzzy
logic for you. You don’t have to

00:27:13.667-->00:27:19.625
re-write um your tool to do any
of this. And in fact the SQLite
and uh PostgreSQL uh attacks

00:27:19.625-->00:27:25.625
here that we sh- that we just
showed had exactly zero lines of
code changed in them. So without

00:27:25.625-->00:27:31.167
changing a single line of code
you can train your uh tool to do
a completely different job. As

00:27:31.167-->00:27:37.833
in uh completely different
details. Uh completely different
job. >>Yeah and that generalized

00:27:37.833-->00:27:42.167
kind of uh you kind of that this
infinitely flexible general you
know function and that that’s

00:27:42.167-->00:27:46.833
gonna be super valuable for you
know tools going forward. Not
just injection attacks but uh a

00:27:46.833-->00:27:52.083
lot of other attacks. So.
>>Yeah. That was demonstration.
So um some lessons learned, um

00:27:52.083-->00:27:56.042
it turns out that quality
training data is really
important. Um we uh originally

00:27:56.042-->00:27:59.875
had a couple of ideas about how
we were going to gather training
data. Um one of our first ideas

00:27:59.875-->00:28:06.417
was to just scrape the internet
for SQL strings. And this
actually works um fairly well,

00:28:06.417-->00:28:13.042
you can just find uh projects a
open source on GitHub that have
uh SQL structures and a tool can

00:28:13.042-->00:28:18.250
look at that and learn and um
what SQL is what sort of
structures of it are. What are

00:28:18.250-->00:28:22.875
the kind of uh terms that it
has. Um but what it doesn’t have
a whole lot of our injection

00:28:22.875-->00:28:27.833
strings specifically. So we can
teach our bot to speak SQL, but
it doesn’t know how to inject.

00:28:27.833-->00:28:32.583
And so as nerds we try to breach
that gap, we thought well hey
maybe we can just use uh SQL

00:28:32.583-->00:28:37.750
map. Since uh SQL map is another
tool that you know does
basically this. It turns out

00:28:37.750-->00:28:41.833
that SQL map um generates
strings, they’re just really
f*****g complicated. Um the its

00:28:41.833-->00:28:47.250
in this um vein of tools that we
refer to as um the worst, best
tools. In that its the clearly

00:28:47.250-->00:28:51.250
the best tool at doing what it
does, it works really well, we
use it on gadgets, but at the

00:28:51.250-->00:28:56.792
same time its just the worst. Um
and the the strings that it
generate um are just really

00:28:56.792-->00:28:59.875
complicated. The model had a
really hard time picking up on
it ‘cause it does things like uh

00:28:59.875-->00:29:03.917
a binary um search algorithm.
>>Yeah, that was- >>just
character by character. >>Yeah,

00:29:03.917-->00:29:08.958
it was really hard for uh the
model to derive that binary
search from uh from SQL map. So

00:29:08.958-->00:29:12.792
we uh had to simplify it down
and we basically made bad SQL
map, which uh does regular uh

00:29:12.792-->00:29:18.458
Boolean search uh Boolean blind
injection. Uh it doesn’t ne- use
binary search. It just kind of

00:29:18.458-->00:29:23.292
brute forces character by
character. Um so, yeah so
training data super super

00:29:23.292-->00:29:28.333
important to get. Um scraping
internet, becoming Google is
pretty hard. Um they do a pretty

00:29:28.333-->00:29:32.375
good job of that so uh you want
to try to get really good
training data and also you want

00:29:32.375-->00:29:37.167
to be incredibly careful about
what you’re rewarding. Uh this
is its seems like kind of common

00:29:37.167-->00:29:41.708
sense but um it can really sneak
up on you in really interesting
ways uh one of our one of our

00:29:41.708-->00:29:45.958
big problems uh that we ran
into, well not like a big
problem, but one of the funny

00:29:45.958-->00:29:50.500
things that kind of happened was
uh we didn’t incentivize um
characters we had already found

00:29:50.500-->00:29:56.458
and uh we thought that oh well
we’re rewarding characters
found, we’ll we’ll try this we

00:29:56.458-->00:30:01.583
ran it for you a long time. And
uh then we just found out that
our database, our our tool was

00:30:01.583-->00:30:05.500
really good at just brute
forcing the same character over
and over again, so we knew the

00:30:05.500-->00:30:10.292
table started with A really
well, you know, that was uh, we
had that definitely locked down.

00:30:10.292-->00:30:13.375
>>Yeah, it was the bird getting
a piece of seed every time it
found a letter, right. So it got

00:30:13.375-->00:30:16.375
A, we gave it a reward. Got A
again, it kept on giving a
reward. Mother f****r kept on

00:30:16.375-->00:30:21.750
finding A. [laughter] Uh so
we’re like oh ok, like you you
will get more of whatever it is

00:30:21.750-->00:30:25.667
that you reward and so you have
to be really careful about that.
You might think that you’ve

00:30:25.667-->00:30:30.375
described the problem accurately
when in fact your computer has
found some clever loophole in

00:30:30.375-->00:30:35.875
your own rules in order to just
get an arbitrary amount of
reward. >>Mm, yeah, and uh

00:30:35.875-->00:30:41.917
another thing we learned is uh
get a GPU uh or better yet, get
a lot of them. Um we had a

00:30:41.917-->00:30:46.458
significant speed up from GPUs
and uh there’s a lot of uh great
companies out there that are you

00:30:46.458-->00:30:52.167
know selling cheap GPUs and you
can just rent them on the cloud,
its like super easy spin up. Um

00:30:52.167-->00:30:56.125
and it really helps. We had like
uh something like uh twenty
times speed up when we were

00:30:56.125-->00:31:00.917
training our models. Um so its
really easy to you know kind of
get started with this stuff and

00:31:00.917-->00:31:05.125
then once you kind of get an
idea for something you want to
try, definitely get a GPU. Uh

00:31:05.125-->00:31:08.833
it’ll speed up your debug time.
Uh that was one of the
frustrating thing uh kind of

00:31:08.833-->00:31:13.542
early. Uh it, you know you have
to train up your model overtime
you make a change, so you wanna

00:31:13.542-->00:31:19.000
try to uh you know make good
changes and then an hour later
you can find out if it was

00:31:19.000-->00:31:22.958
actually a good idea or not. So
uh. >>You thought compile times
were bad. >>You know going back,

00:31:22.958-->00:31:29.125
yeah massive C plus plus so
yeah. The uh getting GPUs, super
helpful. Um more the better.

00:31:29.125-->00:31:36.083
More the better. >>So some other
considerations here about neural
networks um and uh AI and

00:31:36.083-->00:31:41.208
hacking in general. Um neural
networks have this kind of
inherent proprietary-ness to

00:31:41.208-->00:31:46.875
them. Um in that, uh there’s no
really good way to debug them or
to know what they’re doing, like

00:31:46.875-->00:31:51.667
even really in principle. Your
code, your logic, the thing that
determines what your program is

00:31:51.667-->00:31:55.708
doing, is a shape. Its a literal
geometric shape, not
figuratively its like actually

00:31:55.708-->00:32:01.125
a, its a function, right that
has a shape. And so if you try
to ask the question like, hey

00:32:01.125-->00:32:05.250
why did my machine learning car
just like run over a dog. The
answer is f**k you, that’s why.

00:32:05.250-->00:32:08.417
[laughter] >>Like it it did it
because that’s why it did it.
Like it there’s no good reason

00:32:08.417-->00:32:14.042
for it. It just. >>Yeah. >>It
just followed the shape that it
thought it was supposed to

00:32:14.042-->00:32:19.292
follow and so uh this is a
problem both for you know like
debugging purposes in a former

00:32:19.292-->00:32:24.500
life, um I was actually uh
responsible for like a writing
test cases for um a flight

00:32:24.500-->00:32:29.375
software. And we had uh
extremely strict, what are
called code coverage

00:32:29.375-->00:32:34.708
requirements. Which is to say
that the uh I had to write test
cases that hit every single line

00:32:34.708-->00:32:39.833
of code in the entire like suite
of programs in order to like
test that the whole thing wasn’t

00:32:39.833-->00:32:42.917
just going to fall out of the
sky. We care about that sort of
thing in terms of reliability if

00:32:42.917-->00:32:48.458
not critical safety um
environments. And I don’t even
know how wh- what does that even

00:32:48.458-->00:32:51.958
look like for an AI program?
Like you can make a self driving
car that will on average be

00:32:51.958-->00:32:57.542
safer than a human being. ‘Cause
you can make a you know a chess
playing robot or a go playing

00:32:57.542-->00:33:01.917
robot that’s better than people.
But can you have any strong
assurances that this won’t go

00:33:01.917-->00:33:05.208
haywire? Like not really. It
doesn’t even uh, I don’t know
what that looks like in

00:33:05.208-->00:33:08.458
principal. >>Yeah and who’s
responsible when it does go
haywire? You know I mean th-

00:33:08.458-->00:33:11.958
>>Does it entail. Aside from the
entire liability issues.
>>Right. >>Um also if you’re the

00:33:11.958-->00:33:16.125
sort of person who cares about
freeing open source software um
this should also sort of worry

00:33:16.125-->00:33:23.083
you. In that the program
behavior is not debuggable, its
not um reversible to any

00:33:23.083-->00:33:28.667
original form. Even if you’re
given the the entire set of data
the person trained and the

00:33:28.667-->00:33:32.417
source code to it, you can’t
combine them to create the same
program. It’s just stochastic

00:33:32.417-->00:33:38.833
which is saying is fancy math
terms for random. Um and so
that’s also an inherent problem.

00:33:38.833-->00:33:43.625
So what does it even mean to
have an open source AI program.
Um we should start having a

00:33:43.625-->00:33:49.083
conversation about that. >>Yeah,
um you know the big cloud
providers, you know Google,

00:33:49.083-->00:33:51.792
Facebook, all those big
companies that do a lot of this
kind of research, um they will

00:33:51.792-->00:33:58.417
happily rent you out their spare
CPU cycles and they will happily
you know rent you out your GPUs,

00:33:58.417-->00:34:03.833
but they will never ever rent
out their models and their um
mass collections of data to you.

00:34:03.833-->00:34:08.208
That, you know that kind of the
secret sauce, right? So uh if
all that is hidden, um you know

00:34:08.208-->00:34:13.667
you’re kind of at a disadvantage
if uh you wanna get started with
stuff. Uh big hill to climb.

00:34:13.667-->00:34:17.500
Yeah, in a way that’s not
present with ordinary programs.
There’s a reason you see um you

00:34:17.500-->00:34:21.417
know all the major breakthroughs
in this sort of things coming
from large companies that just

00:34:21.417-->00:34:26.458
coincidentally happen to have a
have to access to everyone’s
data. Because you need vast

00:34:26.458-->00:34:31.917
amounts of data in order to make
these sorts of programs happen.
Um but if you you know want to

00:34:31.917-->00:34:35.667
tell the difference between a
stop sign and a dog, then you
really need to have lots and

00:34:35.667-->00:34:40.250
lots of pictures of stop signs
and dogs. And if you the average
programmer want to make a

00:34:40.250-->00:34:45.292
program that does that, uh you
just don’t have that data and
the people who do aren’t

00:34:45.292-->00:34:50.667
sharing. Um uh and and more um
moreover, like programs, AI
writing, writing an AI program

00:34:50.667-->00:34:55.708
is like incredibly fun, its
actually really rewarding and a
lot easier than you probably

00:34:55.708-->00:35:00.042
think it is. Uh there are some
great libraries out there. There
are tensor flow or actually

00:35:00.042-->00:35:05.667
Caris which wraps that up into a
a very easy um uh sort of higher
level library. Um but at the

00:35:05.667-->00:35:11.125
same time it can be a really
frustrating experience. Uh
because every once in a while

00:35:11.125-->00:35:14.917
the program will just do
something that you didn’t think
you told it to do or it’ll just

00:35:14.917-->00:35:19.167
like start going haywire. You
have no idea why and uh you try
to ask the question, like why

00:35:19.167-->00:35:23.125
did it do that. Is it ‘cause you
didn’t train it enough? Like
that can always be a problem.

00:35:23.125-->00:35:26.917
Did you train it too much?
‘Cause that’s a thing too. Uh
like its just kind of really

00:35:26.917-->00:35:31.625
hard. Uh there’s also really fun
problems like uh, what can best
be described as um uh uh oh

00:35:31.625-->00:35:42.500
gosh. The the the >>forfeiting?
>>Yeah, you can forfeit database
to play from the from the data

00:35:42.500-->00:35:46.083
to try to make it better
basically is where I’m getting
at, yeah. Um so some future work

00:35:46.083-->00:35:51.542
here um if you’re like looking
at making some uh data that or
some programs that will you know

00:35:51.542-->00:35:56.750
exploit various things. I know
we have some ideas about what we
want to take care of uh coming

00:35:56.750-->00:36:01.292
here in the new future or things
that we’re actually already
working on. Um and here are some

00:36:01.292-->00:36:05.125
like good crunchy problems. But
as a good rule of thumb, there’s
a really prominent researcher

00:36:05.125-->00:36:09.375
named uh Andrew When, who uh uh
is responsible for a bunch of
this stuff. Um so a good really

00:36:09.375-->00:36:14.375
of thumb is, like if a human
expert uh can solve the problem
in like a couple of seconds, uh

00:36:14.375-->00:36:19.375
that’s probably something that’s
eminently solvable by a a
machine learning algorithm right

00:36:19.375-->00:36:24.250
now. And so if you think back
into all the things you do in
terms of uh like your tasks as a

00:36:24.250-->00:36:27.708
pen tester or as a hacker or
whatever, there’s probably quite
a bit in there that can be

00:36:27.708-->00:36:32.167
automated in terms of what is
fuzzy logic, that maybe you do
as a human that would be a

00:36:32.167-->00:36:37.500
really hard thing to um uh make
a computer that would do that.
>>Yeah, kind of hand program

00:36:37.500-->00:36:42.625
logic in there um yeah. So uh
one thing that I’m really
excited about is uh password

00:36:42.625-->00:36:48.417
brute forcing, um because uh
when you’re trying to you know
brute force some passwords, it

00:36:48.417-->00:36:54.667
it starts with a word list and
then you kind of you know hand
jam some uh some rules to kind

00:36:54.667-->00:36:59.250
of mangle the word list. Uh you
know if if uh you see a you know
the month of or the current

00:36:59.250-->00:37:03.000
season you say ok well for the
end of the year you know the
current year on there and maybe

00:37:03.000-->00:37:08.500
a bang at the end. Uh those
kinds of rules are definitely uh
just kind of uh things you’ve

00:37:08.500-->00:37:15.958
though are a good idea and uh
they seem to fit a pattern uh
from the data. Well uh machine

00:37:15.958-->00:37:20.083
learning algorithm would be
really good at picking up on
those patterns from uh you know

00:37:20.083-->00:37:23.500
password dumps that have been
dumped and uh could potentially
generate rules for you

00:37:23.500-->00:37:28.292
automatically and uh they might
be better than the ones you can
think of you know yourself. Uh

00:37:28.292-->00:37:33.167
they might be able to see things
and see patterns and uh have
relationships that you wouldn’t

00:37:33.167-->00:37:36.542
really be able to see. >>Yeah in
fact that’s something we
frequently do on engagements.

00:37:36.542-->00:37:40.167
When we uh are cracking
passwords that we’ve you know
dumped from whatever. Um we’ll

00:37:40.167-->00:37:44.667
frequently get a little ways in,
look at the passwords that we’ve
already brute forced, and then

00:37:44.667-->00:37:48.833
try to kind of manually sift it
for patterns. Like maybe the
first character is always

00:37:48.833-->00:37:53.292
capital and the last character
is always a number or perhaps it
follows some like company

00:37:53.292-->00:37:58.125
pattern or uh if you have a
organization like everybody
else’s that uh expires passwords

00:37:58.125-->00:38:02.833
every quarter, then they tend to
have things like spring 2016,
fall 2017 as passwords. Uh and

00:38:02.833-->00:38:08.333
so the uh if you have a machine
learning algorithm that can
automatically pick up on these

00:38:08.333-->00:38:12.708
patterns that would be a really
good addition to existing
passwords of a password brute

00:38:12.708-->00:38:17.875
forcing uh methodology. And in
addition um to that, we have
things like an instrumented web

00:38:17.875-->00:38:22.167
app buzzer, something that’s
given the internal structures
from an instrumented Python web

00:38:22.167-->00:38:27.750
application, can see the
internal data structures and
logic and try to produce inputs

00:38:27.750-->00:38:31.542
that will exercise as much fact.
Basically like AFL, if you’re
familiar with that, um but

00:38:31.542-->00:38:35.375
basically the potentially much
smarter in that it uses a neural
network, but also for uh web

00:38:35.375-->00:38:41.750
application that’s a you know
much um larger set of um sort of
things to attack. >>Yeah and uh

00:38:41.750-->00:38:48.625
like I was saying before um
there’s a there’s a lot of hype
about this kind of stuff. Um

00:38:48.625-->00:38:56.667
there’s you know tons of media
coverage about it. You know
saying Hollywood and all kinds

00:38:56.667-->00:39:00.125
of stuff, all these you know AI
programs, but uh what its
actually kind of bad is uh

00:39:00.125-->00:39:04.750
discovering like a new class of
vulnerability right? Um so it
its not uh AI isn’t really going

00:39:04.750-->00:39:09.458
to at least in the short term
its not going to uh be able to
discover new types of

00:39:09.458-->00:39:12.875
vulnerabilities. You know, its
not gonna, its probably not
gonna be able to find um some

00:39:12.875-->00:39:19.208
kind of new crazy you know
injection flaw that we just
don’t you know, know about. Um

00:39:19.208-->00:39:24.208
and a lot of that comes down to
uh context especially you know
uh with security testing, um

00:39:24.208-->00:39:29.542
context is pretty key around
some uh application behavior and
functionality. Uh sometimes its

00:39:29.542-->00:39:34.292
its totally alright to be able
to run remote commands on uh you
know machine. Because that’s a

00:39:34.292-->00:39:39.000
tools purpose, uh the machine
learning algorithm doesn’t know
that so it doesn’t really have

00:39:39.000-->00:39:44.417
the context to decide what is a
security risk and what is not.
So. >>Yeah, suppose you a make a

00:39:44.417-->00:39:48.667
request and a social security
number comes back. Like is that
a bug? Is that intended

00:39:48.667-->00:39:52.375
behavior? It might actually be a
legitimate website that’s
supposed to do that. How’s the

00:39:52.375-->00:39:56.417
computer supposed to know
whether that’s a bug or not. Um
this is kind of the more

00:39:56.417-->00:40:00.625
fundamental questions about like
what is a security bug right?
Reasonable people can disagree

00:40:00.625-->00:40:06.042
on this fact and moreover the
computer is missing context. So
imagine teaching a computer or

00:40:06.042-->00:40:10.708
yourself in fact, learning a
whole new language. Entirely on
the basis of dictionaries. So

00:40:10.708-->00:40:14.958
you read a whole dictionary, you
read a thesaurus, you can read
all as much structure and syntax

00:40:14.958-->00:40:20.750
you can learn languages you want
right. Say French um and then
you go to France and try to talk

00:40:20.750-->00:40:24.875
to people and you’d be
completely s**t out of luck.
Somebody’s gonna tell you some

00:40:24.875-->00:40:27.917
sort of idioms. Say like you
know, take a hike or something
like that. And you have no idea

00:40:27.917-->00:40:32.417
what that means in context
‘cause the language is about
more than just the words on

00:40:32.417-->00:40:36.542
paper or words as they’re
spoken, it’s a part of culture.
And so the same thing is true as

00:40:36.542-->00:40:42.000
security in general. That what
is and is not a security general
uh problem often just depends.

00:40:42.000-->00:40:46.792
And so uh I suspect the machine
learning algorithms are going to
be quite bad at doing this sort

00:40:46.792-->00:40:52.250
of thing and quite good at
automating tasks that used to be
fuzzy logic. That uh computers

00:40:52.250-->00:40:56.167
um previously quite bad at. And
I think that is a very large
space in computer security um

00:40:56.167-->00:41:06.083
that could be taken advantage
of. >>Yeah unfortunately no sky
net right now. Um, but yeah.

00:41:06.083-->00:41:09.833
>>Um that’s it. We’ll uh, we
have a couple minutes for
questions if anyone wants ‘em.

00:41:09.833-->00:00:00.000
[applause]