start	end	text
1550	11010	So, welcome to our next talk by Nahid Ghali and Vincent Pham, who I believe are both researchers
11010	17710	at Capital One, and they're presenting a detection system for ML-based threats.
18890	22070	So let's let them take it away.
23570	28570	To our talk, Spectrum, an end-to-end framework for ML-based threat monitoring and detection.
28570	35730	I'm Vincent Pham, and my co-author, Nahid Farhadi. We're both software developers at
35730	44270	Capital One. We're going to talk about a platform that targets insider threat. So for those
44270	49550	of you who are unfamiliar with what insider threat is, we have the formal definition here.
49550	54010	So insider threat is the potential for an individual who has or had authorized access
54010	59390	to an organization's asset to use that access, either maliciously or unintentionally, to
59390	64390	act in a way that could negatively affect the organization. This unintentionally is
64390	69450	also important because we're not looking at only malicious or suspicious people. Also
69450	74670	people who have naively or unknowingly leaked data from the company as well, since they
74670	80730	are a potential threat. I think this comic on the right side pretty much clarifies who
80730	86470	are insider threats. Basically when you think of the entire compass of who are possible
86470	90990	threats in the world, it's pretty much the entire population and then who are insider
90990	97570	threats. That's the people who work within a company. Even if you have zero risk or very
97570	106330	minimal risk, there's still a potential threat there. And just to give you a little bit about
106930	113850	the insider threat stats, 90% of organizations feel vulnerable to insider attacks. 53% of
113850	119410	companies had insider threat attack against their organization in the last 12 months. 37%
119410	125330	of employees have excessive access privileges. This could be important because think about
125830	131650	your most important data and what would happen if that your company and who has access to that
131650	138010	data. 94% of companies monitor their employees digital footprint. So this is the data sources
138010	143870	that we could use to track where the data is going, who has access to what. And then 86%
143870	150050	of companies have or are building an insider threat program to date. And then also another
150050	157590	important fact is the average cost of insider threats is currently about estimated at $1.6
157590	168510	million. So we mentioned that there's 86% companies who are thinking about or are currently
168510	173830	building an insider threat system. You might be thinking should you buy a platform that is
173830	179830	currently have a solution developed or should you build your own solution to insider threat?
180210	185990	We list some of the pros of each of the different buying versus build. So if you think about buying
185990	192810	a platform, the pros of that is that a solution is already developed. So it could possibly be
192810	199370	simple as a plug and play. Although in most cases, it'll take a couple of months or maybe
199370	204870	even a couple of years just to integrate that platform into your system. Also support is
204870	210310	provided so you don't have to build up your own team, you could pay for support to install that
210310	217590	system and grant you the analysis to insider threat. Lastly, the data management and handling
217590	221450	solution is also provided by the company as well. So once again, you don't have to build your own
221450	226930	team. The pros of building your own system is that technology is always changing. So if your
226930	231970	company is thinking about changing its technology, changing its database, or just changing the way
232590	238530	everything is handled, moving from maybe a server or a local server to the cloud,
238530	245050	you could quickly adapt to that need. If your system is constantly being audited, you have
245050	249530	full knowledge of the systems, you don't have to wait for your vendor to give you back some
249530	255110	information, or maybe they're not even going to release the information. When you're building
255110	261210	it yourself, you already have that information. And then once again, due to ever-changing technology,
261210	266710	you might have changing data as well. So then you could adapt your platform really quickly to
266710	278910	that changing data. So in order to combat insider threat, we built a system called Spectrum,
278910	284710	which operates on two types of entities and the relationship. The entities are the subjects and
284710	293810	the objects. The subjects are identified as the types that take action, such as in a very
293810	298530	simple sense for insider threat are the employees. But you could also consider
299330	306890	AM roles in AWS, which is like a group of profiles gathered together. Or if you're doing insider
306890	312530	threats, or I guess external threats, you can also think about customers as your subjects.
312530	319190	And then the objects are entities in which actions are taken. And those actions are recorded as
319190	323590	event trails. So you could think of like anytime someone logs into a VPN, what time they logged
323590	331490	in at, if they batch into a building, if they opened a laptop, events such as that are recordable
331490	344140	and provides evidence to whether an insider is happening or not. So the proposed framework
344920	352100	for Spectrum follows such that if we go from the very top, we define a set of entities. So
352540	358360	once again, in this case, that would be the employees, we find out the different data
358360	366420	sources that are linked to the entity. So we might have, in this case, three different events source.
366780	372840	And then from each of the individual events source, we have features for that event source.
372840	379300	For example, let's say you're using your email data set, that email data set will have features
379300	385240	such as where the email is being sent to, where it's being sent from, the time of day the email
385240	391400	is sent, the subject, and so on and so on. And then from the features, we can construct patterns.
392520	400040	For example, has the email been sent within the past 24 hours? Does the content of the email
400500	408520	display a positive or negative emotion? And so on. We could construct rules from it,
408520	413160	we could also do anomaly detection on the feature itself. So is there something suspicious
413160	418700	based away from the baseline? And then we can also build machinery models such as classifier
418700	425540	to identify, once again, whether particular events or set of feature shuffle up something
425540	432200	that's suspicious or not. So once we construct all of these rules, anomaly classifiers,
432780	436740	we tied back the dependencies to extractor, which ties back to event source.
436740	442840	And then that's how we built the entire pipeline from the bottom up. And then when we run the
442840	450120	pipeline, it then reconstructs this entire scheme and tied back all the data sources so that if you
450120	454980	turn off one pattern, then it might turn off a particular source just to save some cost in the
454980	464840	meantime. Our data is then enriched to the role, the entity, and that is stored into storage with
464840	472920	the entity and the enriched data of whether a particular role is suspicious or not or whether,
473620	479020	I guess, the classification is marking as a zero to one probability of suspiciousness.
479500	487060	These information are then used to flag a particular entity, maybe another reduced flag,
487060	492420	or maybe open up a case management so that analysts can look into, or it could even be
492420	498760	displayed in a UI so that analysts can go through this ranked order of suspicious entities, suspicious
498760	503480	events, and see whether this is really a true positive or false positive in terms of
503480	510280	inside threat or not. So, in a nutshell, this is what our system of spectrum looks like.
512100	521400	All right. So, this proposed framework had some techniques being done on it in order to effectively
523620	530480	save money on cost of data handling or making sure that we don't lose any data.
530660	536600	So, using modes of operation and also dependency orchestration. I'm going to go through some of
536600	541820	the techniques that we use to build this whole framework. The first one is that we have
542340	548860	a unified source of the data. Basically, when we look at different sources of information,
548860	553520	like digital footprint, it could be any digital interaction that the employee has
553520	558620	with the outside world. It could be proxy monitoring, mail servers, different sensors
558620	567240	that we have on endpoints or user machines or on cloud accounts that monitor all of these
567980	574580	traffics and the communication that we have with the outside world. We can have different
574580	581940	tools or event monitoring like windows, event log, sysmon, snort, etc. And each of these resources
581940	589240	provide a specific type of information for us. But the important thing is that when we want to
589240	595200	run the job and we want to detect whether there has been any insider threats actions over the
595200	603860	past 24 hours, it is important to only query the past 24 hours for that specific data.
603860	609680	Now, at the development time, your developers are going to develop different features and they
609680	616840	are going to each of them query against all of these data sources, which is very costly.
616840	626180	So our proposed solution to have a cost efficiency in a spectrum development phase is that
626820	634060	we actually had all of these data sources. We processed them and we sessionized them and we
634060	641020	extracted the information that we really need from each data source into a specific table. And then
641020	648340	we had a Matillion job running on a timely basis, like five times a day, six times a day, as your
648340	655840	customer requests you to extract that information, process them and keep them in a table for us.
655840	663100	The size of this table is an order of magnitude smaller compared to the size of the data sources
663100	669100	that you originally have. Plus your developers wouldn't go by themselves and each of them query
669100	675000	those huge tables. Now you have unified all of this information in one table and it is going
675000	679320	every time you want to develop, you want to test, you want to add a new feature,
679320	686460	you are going to query the smaller table. So this actually saved us 90 percent of the cost
686460	695420	of the querying. We use Snowflake as the source of the data where we keep our original information
695420	702660	and then we use Matillion in order to have a schedule for running jobs on extracting information
702660	710940	and processing them from Snowflake. So here is an example of how we keep the data.
711640	718780	The other benefit or other feature that we propose in implementation of a spectrum is that
718780	725280	we unify the data. This brings a second benefit for us. First one was that, okay,
725280	735180	we had some cost savings. The second benefit is that this data frame is like an object and it is
735180	743400	moving along the pipeline and different actions are being executed on this data frame. The
743400	750320	important thing is that we don't complicate our design by separating different data sources in
750320	756520	different tables, having each of those tables have different number of roles. For example,
756520	763600	in email, you have a data type, definitely a hash or a unique ID. It is related to an entity ID,
763600	770120	it could be an employee, it could be an IAM role or whatever entity that is doing the action,
770120	776860	and it has a timestamp. But in meta, you have to from number of attachments, etc.
776860	783920	The same information for a proxy data is very different. Of course, you have the first four
783920	791080	columns, but in meta, you have probably the URL that has been accessed, the type of connection,
791080	797920	is it HTTP or what type of connection is it, is it blocked by the company or not,
797920	805740	or etc. Other type of information. So imagine if we had different data frames for each of these
805740	813540	data types. And you know, you can have up to 20 data sources and having different tables for each
813540	820260	of these is going to make your design very complicated and very inflexible against making
820260	827960	changes. But with the proposed method, we keep information related to each data type in a JSON
827960	837040	format in just one column, and that column is named meta. This way, our data frame can have
837040	843120	different types of information, but all of them are in the same format, all of them are in one
843120	849120	data frame, and that data frame is going through the pipeline, and each of the modules is responsible
849120	856600	for doing specific actions on it. That's why we had extractors. Those extractors actually extract
856600	860740	features and stuff from the meta field.
863600	870500	Another design thinking that was behind this proposed framework is modes of operation.
871060	878980	So as you know, like any other machine learning system, we have a normal job that is running,
878980	888360	usually in the POC phase, we call it the test one, which is actually running the trained model
888360	895500	against your real-time data and extracting results. We call that mode modeling, and we
895500	903360	run it six times a day, or more per user's request. We have a training phase. In that phase,
903360	909740	we only train the models. We have several baseliners, we have several classifiers.
909740	914980	Those baseliners and anomaly detections are for user and entity behavior monitoring.
914980	921140	So we compare employees against their own behavior, or against their peers' behavior,
921140	926820	or role-based type of anomaly detection. So another phase of operation is training.
926820	932920	In training, obviously, we extract more or longer period of the data.
933920	940580	And that's actually where the Matillion job and having a unified table helps us.
940580	946500	In modeling, we only look at the past 24 hours of the data. In training, we look at, for example,
946500	955840	90 days of the data, or a longer period of time. So in this case, now that we have two different
955840	963420	phases, and we separated these duties from each other, we make sure not to run really long queries
963420	969540	for training. We have performance monitoring. In this performance monitoring, we monitor the
969540	976120	thresholds and metrics for our machine learning models, for classifiers and anomaly detectors,
976120	984840	and make sure that they are acting normally. They are not having any problems with the sources of
984840	992920	the data, or with the baseliners, or if we generate alerts off of these, we go back to
992920	999660	training and we fine-tune our model. We also have a backfill mode, which is something that happens
999660	1005240	usually in production, is that sometimes your sensors don't work, sometimes you don't have the
1005240	1013120	source of the data available, there are problems with running jobs in order to filling out those
1013120	1021880	tables. So in backfill mode, we run the same modeling, but this time on a longer period of
1021880	1028980	time to make sure that we haven't missed on any of the events due to problems in production and
1028980	1038960	providing the data. Another important part of this modeling job is a dependency orchestrator.
1038960	1045340	As I said before, we have different, and Vincent mentioned before, we have different phases of
1046680	1054940	operation. First, we extract stuff from our data sources, then after we extract those data,
1054940	1063800	we have three main techniques in order to find out if an event is malicious or not. We have
1063800	1069500	rule-based techniques, which we call pattern matchers, we have baseliners, which is anomaly
1069500	1076180	detection type of job, and then we have machine learning component. Now, each of these might
1076180	1083680	depend on each other. For example, on a baseliner, you might want to run the baseliner on a feature
1083680	1090580	that is extracted, or you want to run a baseliner on a specific data type, for example, on print
1090580	1097260	operations. That doesn't necessarily need to wait on extracting information from email,
1097260	1103840	right? It's just a baseliner on print operation, so there is no dependency between email events
1103840	1110680	and print events. So, in the dependency orchestrator, we come up with a plan that
1110680	1117840	what is the most optimized way to run these modules based on their dependencies. You may
1117840	1123560	have machine learning components that depend on several baseliners or several pattern matchers,
1123560	1129440	you may run different baseliners that depend on two features, and finally, you may detect the
1129440	1138820	maliciousness of something if a specific feature is extracted and it is violating an anomaly. So,
1138820	1145420	there is all these dependencies. Here is an example in the module dependency. For example,
1145420	1152420	here, we read the data from our original database and imagine that each of these colors is an event
1152420	1158780	type. For example, this is email, this is proxy, and this is print. We extract information from email,
1158780	1163440	we extract information from print, for example, number of pages, printer that is used, and we
1163440	1172540	extract information from proxy. Then, we may want to run a baseliner on the email events and see that,
1172540	1178420	amount of emails that someone is sending on a daily basis, or we may want to look at the number
1178420	1183580	of proxy, what is the number of, what is the amount of upload and download for proxy that we
1183580	1192000	use in a baseliner. Then, later on, we call these as detectors. These detectors might say that I
1192000	1198840	think this person is malicious if they printed something and if they had some sort of anomaly
1198840	1206000	behavior in terms of proxy. So, the dependency of this proxy is both on the baseliner and on the
1206000	1214100	extractor of the second event type. So, basically, the job of dependency orchestrator is to come up
1214100	1221240	with a plan and find out what is the dependency for each of these modules and make sure that they
1221240	1227740	are ordered in a way that each module meets its dependency when it's time for the module to be
1227740	1233920	executed. Finally, we have a store operation where we store all of those results in Postgres
1234680	1242780	database or any other type of database and we have a list of threats that we have detected for
1242780	1252000	each of the entities. Another capability that we have in this implementation in the proposed
1252000	1260140	framework, it provides us scalability. First of all, for the processing engine,
1260140	1269640	we use Spark on AWS EMR and it is composed of fleet instance and we have some spot instances,
1269640	1276400	some on-demand instances on EC2. As soon as the number of jobs increase, the number of data,
1276400	1280800	the number of events that we are monitoring or the period that we are monitoring,
1281120	1289700	it can totally scale out and perform the job successfully. Another scalability from another
1289700	1298340	point of view is that we have added RBAC restricted views for our events. For example,
1298340	1305640	if you have different customers with different levels of access control to the entities that
1305640	1312680	you are monitoring or to the events that you are monitoring, you can use RBAC and role-based
1312680	1323320	access control in order to make sure that, for example, the head of insider threat monitoring
1323320	1330560	can have access to all of the threats, can see all of the activities of, for example,
1330560	1338020	executive or leadership of the company, but maybe a junior security analyst doesn't have
1338020	1347980	as much capability. So we added into our detectors, we have a field that we set a level of access
1348620	1354880	for each of those threats, or it can be based on role, based on a threat, or based on the
1354880	1362680	activity that has been executed. This will give us the capability to provide service for multiple
1362680	1370620	customers as well. The same framework that is looking at the same exact same data,
1370620	1377200	it can be used for insider threat detection, but if we have other customers, we can see that
1377200	1383480	if there was any network intrusion, so this is basically CSOC's interest, if there is any website
1383480	1390060	abuse, or in case of financial companies, we can see if there is any insider trading
1390680	1396440	or any fraud done by the employees. This doesn't necessarily have to be looking at specific
1397860	1405520	entities or limited to any relationship between those entities. As long as the application is
1405520	1410880	monitoring and looking at anomaly detection, we can simply change the source of the data
1410880	1415860	and modify the dependencies and actually extract the results.
1417380	1421760	Thank you so much for your time and attention.
1422080	1426060	Here's our emails. We're happy to answer any questions.