1
00:00:01,550 --> 00:00:17,710
So, welcome to our next talk by Nahid Ghali and Vincent Pham, who I believe are both researchers at Capital One, and they're presenting a detection system for ML-based threats.

2
00:00:18,890 --> 00:00:22,070
So let's let them take it away.

3
00:00:23,570 --> 00:00:28,570
To our talk, Spectrum, an end-to-end framework for ML-based threat monitoring and detection.

4
00:00:28,570 --> 00:00:32,950
I'm Vincent Pham, and my co-author, Nahid Farhadi.

5
00:00:32,950 --> 00:00:37,070
We're both software developers at Capital One.

6
00:00:38,310 --> 00:00:43,670
We're going to talk about a platform that targets insider threat.

7
00:00:43,670 --> 00:00:49,550
So for those of you who are unfamiliar with what insider threat is, we have the formal definition here.

8
00:00:49,550 --> 00:01:02,690
So insider threat is the potential for an individual who has or had authorized access to an organization's asset to use that access, either maliciously or unintentionally, to act in a way that could negatively affect the organization.

9
00:01:02,990 --> 00:01:09,250
This unintentionally is also important because we're not looking at only malicious or suspicious people.

10
00:01:09,250 --> 00:01:16,050
Also people who have naively or unknowingly leaked data from the company as well, since they are a potential threat.

11
00:01:16,150 --> 00:01:22,210
I think this comic on the right side pretty much clarifies who are insider threats.

12
00:01:22,210 --> 00:01:31,670
Basically when you think of the entire compass of who are possible threats in the world, it's pretty much the entire population and then who are insider threats.

13
00:01:31,670 --> 00:01:34,170
That's the people who work within a company.

14
00:01:34,670 --> 00:01:41,170
Even if you have zero risk or very minimal risk, there's still a potential threat there.

15
00:01:44,370 --> 00:01:52,970
And just to give you a little bit about the insider threat stats, 90% of organizations feel vulnerable to insider attacks.

16
00:01:52,970 --> 00:01:58,630
53% of companies had insider threat attack against their organization in the last 12 months.

17
00:01:58,630 --> 00:02:03,050
37% of employees have excessive access privileges.

18
00:02:03,050 --> 00:02:12,490
This could be important because think about your most important data and what would happen if that your company and who has access to that data.

19
00:02:12,870 --> 00:02:17,210
94% of companies monitor their employees digital footprint.

20
00:02:17,210 --> 00:02:22,570
So this is the data sources that we could use to track where the data is going, who has access to what.

21
00:02:22,570 --> 00:02:28,850
And then 86% of companies have or are building an insider threat program to date.

22
00:02:29,010 --> 00:02:38,170
And then also another important fact is the average cost of insider threats is currently about estimated at $1.6 million.

23
00:02:41,370 --> 00:02:50,450
So we mentioned that there's 86% companies who are thinking about or are currently building an insider threat system.

24
00:02:50,550 --> 00:02:59,830
You might be thinking should you buy a platform that is currently have a solution developed or should you build your own solution to insider threat?

25
00:03:00,210 --> 00:03:04,610
We list some of the pros of each of the different buying versus build.

26
00:03:04,710 --> 00:03:11,150
So if you think about buying a platform, the pros of that is that a solution is already developed.

27
00:03:11,250 --> 00:03:15,050
So it could possibly be simple as a plug and play.

28
00:03:15,110 --> 00:03:23,410
Although in most cases, it'll take a couple of months or maybe even a couple of years just to integrate that platform into your system.

29
00:03:23,670 --> 00:03:35,310
Also support is provided so you don't have to build up your own team, you could pay for support to install that system and grant you the analysis to insider threat.

30
00:03:35,470 --> 00:03:40,110
Lastly, the data management and handling solution is also provided by the company as well.

31
00:03:40,110 --> 00:03:42,210
So once again, you don't have to build your own team.

32
00:03:42,210 --> 00:03:46,550
The pros of building your own system is that technology is always changing.

33
00:03:46,550 --> 00:04:00,570
So if your company is thinking about changing its technology, changing its database, or just changing the way everything is handled, moving from maybe a server or a local server to the cloud, you could quickly adapt to that need.

34
00:04:01,270 --> 00:04:13,710
If your system is constantly being audited, you have full knowledge of the systems, you don't have to wait for your vendor to give you back some information, or maybe they're not even going to release the information.

35
00:04:14,350 --> 00:04:17,190
When you're building it yourself, you already have that information.

36
00:04:17,190 --> 00:04:23,170
And then once again, due to ever-changing technology, you might have changing data as well.

37
00:04:23,170 --> 00:04:28,630
So then you could adapt your platform really quickly to that changing data.

38
00:04:32,990 --> 00:04:43,030
So in order to combat insider threat, we built a system called Spectrum, which operates on two types of entities and the relationship.

39
00:04:43,030 --> 00:04:45,970
The entities are the subjects and the objects.

40
00:04:46,150 --> 00:04:57,070
The subjects are identified as the types that take action, such as in a very simple sense for insider threat are the employees.

41
00:04:57,070 --> 00:05:04,990
But you could also consider AM roles in AWS, which is like a group of profiles gathered together.

42
00:05:04,990 --> 00:05:12,530
Or if you're doing insider threats, or I guess external threats, you can also think about customers as your subjects.

43
00:05:12,530 --> 00:05:16,150
And then the objects are entities in which actions are taken.

44
00:05:16,190 --> 00:05:20,010
And those actions are recorded as event trails.

45
00:05:20,010 --> 00:05:37,510
So you could think of like anytime someone logs into a VPN, what time they logged in at, if they batch into a building, if they opened a laptop, events such as that are recordable and provides evidence to whether an insider is happening or not.

46
00:05:42,240 --> 00:05:51,980
So the proposed framework for Spectrum follows such that if we go from the very top, we define a set of entities.

47
00:05:51,980 --> 00:06:01,900
So once again, in this case, that would be the employees, we find out the different data sources that are linked to the entity.

48
00:06:02,220 --> 00:06:06,420
So we might have, in this case, three different events source.

49
00:06:06,780 --> 00:06:12,840
And then from each of the individual events source, we have features for that event source.

50
00:06:12,840 --> 00:06:27,820
For example, let's say you're using your email data set, that email data set will have features such as where the email is being sent to, where it's being sent from, the time of day the email is sent, the subject, and so on and so on.

51
00:06:27,860 --> 00:06:31,400
And then from the features, we can construct patterns.

52
00:06:32,520 --> 00:06:37,560
For example, has the email been sent within the past 24 hours?

53
00:06:38,200 --> 00:06:42,900
Does the content of the email display a positive or negative emotion?

54
00:06:43,040 --> 00:06:44,160
And so on.

55
00:06:45,040 --> 00:06:51,660
We could construct rules from it, we could also do anomaly detection on the feature itself.

56
00:06:51,840 --> 00:06:55,820
So is there something suspicious based away from the baseline?

57
00:06:55,940 --> 00:07:07,000
And then we can also build machinery models such as classifier to identify, once again, whether particular events or set of feature shuffle up something that's suspicious or not.

58
00:07:08,040 --> 00:07:16,740
So once we construct all of these rules, anomaly classifiers, we tied back the dependencies to extractor, which ties back to event source.

59
00:07:16,740 --> 00:07:21,120
And then that's how we built the entire pipeline from the bottom up.

60
00:07:21,140 --> 00:07:35,820
And then when we run the pipeline, it then reconstructs this entire scheme and tied back all the data sources so that if you turn off one pattern, then it might turn off a particular source just to save some cost in the meantime.

61
00:07:37,380 --> 00:07:59,020
Our data is then enriched to the role, the entity, and that is stored into storage with the entity and the enriched data of whether a particular role is suspicious or not or whether, I guess, the classification is marking as a zero to one probability of suspiciousness.

62
00:07:59,500 --> 00:08:18,480
These information are then used to flag a particular entity, maybe another reduced flag, or maybe open up a case management so that analysts can look into, or it could even be displayed in a UI so that analysts can go through this ranked order of suspicious entities,

63
00:08:18,480 --> 00:08:25,440
suspicious events, and see whether this is really a true positive or false positive in terms of inside threat or not.

64
00:08:25,500 --> 00:08:30,280
So, in a nutshell, this is what our system of spectrum looks like.

65
00:08:32,100 --> 00:08:33,020
All right.

66
00:08:33,020 --> 00:08:50,480
So, this proposed framework had some techniques being done on it in order to effectively save money on cost of data handling or making sure that we don't lose any data.

67
00:08:50,660 --> 00:08:55,060
So, using modes of operation and also dependency orchestration.

68
00:08:55,060 --> 00:09:00,080
I'm going to go through some of the techniques that we use to build this whole framework.

69
00:09:00,080 --> 00:09:04,580
The first one is that we have a unified source of the data.

70
00:09:04,580 --> 00:09:15,180
Basically, when we look at different sources of information, like digital footprint, it could be any digital interaction that the employee has with the outside world.

71
00:09:15,180 --> 00:09:33,140
It could be proxy monitoring, mail servers, different sensors that we have on endpoints or user machines or on cloud accounts that monitor all of these traffics and the communication that we have with the outside world.

72
00:09:33,140 --> 00:09:40,260
We can have different tools or event monitoring like windows, event log, sysmon, snort, etc.

73
00:09:40,260 --> 00:09:45,680
And each of these resources provide a specific type of information for us.

74
00:09:45,680 --> 00:10:03,860
But the important thing is that when we want to run the job and we want to detect whether there has been any insider threats actions over the past 24 hours, it is important to only query the past 24 hours for that specific data.

75
00:10:03,860 --> 00:10:16,840
Now, at the development time, your developers are going to develop different features and they are going to each of them query against all of these data sources, which is very costly.

76
00:10:16,840 --> 00:10:30,780
So our proposed solution to have a cost efficiency in a spectrum development phase is that we actually had all of these data sources.

77
00:10:30,860 --> 00:10:40,500
We processed them and we sessionized them and we extracted the information that we really need from each data source into a specific table.

78
00:10:40,540 --> 00:10:55,840
And then we had a Matillion job running on a timely basis, like five times a day, six times a day, as your customer requests you to extract that information, process them and keep them in a table for us.

79
00:10:55,840 --> 00:11:04,900
The size of this table is an order of magnitude smaller compared to the size of the data sources that you originally have.

80
00:11:05,000 --> 00:11:10,420
Plus your developers wouldn't go by themselves and each of them query those huge tables.

81
00:11:10,420 --> 00:11:22,240
Now you have unified all of this information in one table and it is going every time you want to develop, you want to test, you want to add a new feature, you are going to query the smaller table.

82
00:11:22,240 --> 00:11:27,400
So this actually saved us 90 percent of the cost of the querying.

83
00:11:27,400 --> 00:11:46,160
We use Snowflake as the source of the data where we keep our original information and then we use Matillion in order to have a schedule for running jobs on extracting information and processing them from Snowflake.

84
00:11:46,760 --> 00:11:50,940
So here is an example of how we keep the data.

85
00:11:51,640 --> 00:12:00,720
The other benefit or other feature that we propose in implementation of a spectrum is that we unify the data.

86
00:12:00,720 --> 00:12:03,780
This brings a second benefit for us.

87
00:12:03,780 --> 00:12:07,940
First one was that, okay, we had some cost savings.

88
00:12:07,940 --> 00:12:23,240
The second benefit is that this data frame is like an object and it is moving along the pipeline and different actions are being executed on this data frame.

89
00:12:23,240 --> 00:12:35,820
The important thing is that we don't complicate our design by separating different data sources in different tables, having each of those tables have different number of roles.

90
00:12:35,820 --> 00:12:41,740
For example, in email, you have a data type, definitely a hash or a unique ID.

91
00:12:41,740 --> 00:12:51,600
It is related to an entity ID, it could be an employee, it could be an IAM role or whatever entity that is doing the action, and it has a timestamp.

92
00:12:51,600 --> 00:12:56,860
But in meta, you have to from number of attachments, etc.

93
00:12:56,860 --> 00:13:01,060
The same information for a proxy data is very different.

94
00:13:01,060 --> 00:13:19,300
Of course, you have the first four columns, but in meta, you have probably the URL that has been accessed, the type of connection, is it HTTP or what type of connection is it, is it blocked by the company or not, or etc.

95
00:13:19,300 --> 00:13:21,260
Other type of information.

96
00:13:21,260 --> 00:13:26,740
So imagine if we had different data frames for each of these data types.

97
00:13:26,740 --> 00:13:41,260
And you know, you can have up to 20 data sources and having different tables for each of these is going to make your design very complicated and very inflexible against making changes.

98
00:13:41,260 --> 00:13:53,540
But with the proposed method, we keep information related to each data type in a JSON format in just one column, and that column is named meta.

99
00:13:53,920 --> 00:14:11,820
This way, our data frame can have different types of information, but all of them are in the same format, all of them are in one data frame, and that data frame is going through the pipeline, and each of the modules is responsible for doing specific actions on it.

100
00:14:11,820 --> 00:14:13,740
That's why we had extractors.

101
00:14:13,740 --> 00:14:20,740
Those extractors actually extract features and stuff from the meta field.

102
00:14:23,600 --> 00:14:30,500
Another design thinking that was behind this proposed framework is modes of operation.

103
00:14:31,060 --> 00:14:53,580
So as you know, like any other machine learning system, we have a normal job that is running, usually in the POC phase, we call it the test one, which is actually running the trained model against your real-time data and extracting results.

104
00:14:53,580 --> 00:15:00,560
We call that mode modeling, and we run it six times a day, or more per user's request.

105
00:15:00,580 --> 00:15:02,460
We have a training phase.

106
00:15:02,460 --> 00:15:06,360
In that phase, we only train the models.

107
00:15:06,360 --> 00:15:09,740
We have several baseliners, we have several classifiers.

108
00:15:09,740 --> 00:15:14,980
Those baseliners and anomaly detections are for user and entity behavior monitoring.

109
00:15:14,980 --> 00:15:23,620
So we compare employees against their own behavior, or against their peers' behavior, or role-based type of anomaly detection.

110
00:15:23,620 --> 00:15:26,820
So another phase of operation is training.

111
00:15:26,820 --> 00:15:32,920
In training, obviously, we extract more or longer period of the data.

112
00:15:33,920 --> 00:15:40,580
And that's actually where the Matillion job and having a unified table helps us.

113
00:15:40,580 --> 00:15:44,420
In modeling, we only look at the past 24 hours of the data.

114
00:15:44,420 --> 00:15:50,780
In training, we look at, for example, 90 days of the data, or a longer period of time.

115
00:15:50,780 --> 00:16:04,960
So in this case, now that we have two different phases, and we separated these duties from each other, we make sure not to run really long queries for training.

116
00:16:05,080 --> 00:16:07,080
We have performance monitoring.

117
00:16:07,080 --> 00:16:20,420
In this performance monitoring, we monitor the thresholds and metrics for our machine learning models, for classifiers and anomaly detectors, and make sure that they are acting normally.

118
00:16:20,420 --> 00:16:35,360
They are not having any problems with the sources of the data, or with the baseliners, or if we generate alerts off of these, we go back to training and we fine-tune our model.

119
00:16:35,360 --> 00:16:54,280
We also have a backfill mode, which is something that happens usually in production, is that sometimes your sensors don't work, sometimes you don't have the source of the data available, there are problems with running jobs in order to filling out those tables.

120
00:16:54,280 --> 00:17:11,060
So in backfill mode, we run the same modeling, but this time on a longer period of time to make sure that we haven't missed on any of the events due to problems in production and providing the data.

121
00:17:12,920 --> 00:17:18,960
Another important part of this modeling job is a dependency orchestrator.

122
00:17:18,960 --> 00:17:27,720
As I said before, we have different, and Vincent mentioned before, we have different phases of operation.

123
00:17:27,720 --> 00:17:43,220
First, we extract stuff from our data sources, then after we extract those data, we have three main techniques in order to find out if an event is malicious or not.

124
00:17:43,220 --> 00:17:53,460
We have rule-based techniques, which we call pattern matchers, we have baseliners, which is anomaly detection type of job, and then we have machine learning component.

125
00:17:53,460 --> 00:17:57,660
Now, each of these might depend on each other.

126
00:17:57,660 --> 00:18:11,560
For example, on a baseliner, you might want to run the baseliner on a feature that is extracted, or you want to run a baseliner on a specific data type, for example, on print operations.

127
00:18:11,560 --> 00:18:17,600
That doesn't necessarily need to wait on extracting information from email, right?

128
00:18:17,600 --> 00:18:25,420
It's just a baseliner on print operation, so there is no dependency between email events and print events.

129
00:18:25,420 --> 00:18:36,860
So, in the dependency orchestrator, we come up with a plan that what is the most optimized way to run these modules based on their dependencies.

130
00:18:36,860 --> 00:18:58,520
You may have machine learning components that depend on several baseliners or several pattern matchers, you may run different baseliners that depend on two features, and finally, you may detect the maliciousness of something if a specific feature is extracted and it is violating an anomaly.

131
00:18:58,520 --> 00:19:00,860
So, there is all these dependencies.

132
00:19:00,860 --> 00:19:04,740
Here is an example in the module dependency.

133
00:19:04,740 --> 00:19:12,840
For example, here, we read the data from our original database and imagine that each of these colors is an event type.

134
00:19:12,840 --> 00:19:16,220
For example, this is email, this is proxy, and this is print.

135
00:19:16,220 --> 00:19:25,760
We extract information from email, we extract information from print, for example, number of pages, printer that is used, and we extract information from proxy.

136
00:19:25,900 --> 00:19:45,400
Then, we may want to run a baseliner on the email events and see that, amount of emails that someone is sending on a daily basis, or we may want to look at the number of proxy, what is the number of, what is the amount of upload and download for proxy that we use in a baseliner.

137
00:19:46,080 --> 00:19:49,540
Then, later on, we call these as detectors.

138
00:19:49,540 --> 00:20:00,680
These detectors might say that I think this person is malicious if they printed something and if they had some sort of anomaly behavior in terms of proxy.

139
00:20:00,680 --> 00:20:09,880
So, the dependency of this proxy is both on the baseliner and on the extractor of the second event type.

140
00:20:10,180 --> 00:20:28,780
So, basically, the job of dependency orchestrator is to come up with a plan and find out what is the dependency for each of these modules and make sure that they are ordered in a way that each module meets its dependency when it's time for the module to be executed.

141
00:20:28,780 --> 00:20:45,280
Finally, we have a store operation where we store all of those results in Postgres database or any other type of database and we have a list of threats that we have detected for each of the entities.

142
00:20:47,340 --> 00:20:55,420
Another capability that we have in this implementation in the proposed framework, it provides us scalability.

143
00:20:55,940 --> 00:21:12,500
First of all, for the processing engine, we use Spark on AWS EMR and it is composed of fleet instance and we have some spot instances, some on-demand instances on EC2.

144
00:21:12,520 --> 00:21:27,220
As soon as the number of jobs increase, the number of data, the number of events that we are monitoring or the period that we are monitoring, it can totally scale out and perform the job successfully.

145
00:21:27,220 --> 00:21:37,680
Another scalability from another point of view is that we have added RBAC restricted views for our events.

146
00:21:37,680 --> 00:21:58,200
For example, if you have different customers with different levels of access control to the entities that you are monitoring or to the events that you are monitoring, you can use RBAC and role-based access control in order to make sure that, for example,

147
00:21:58,200 --> 00:22:19,720
the head of insider threat monitoring can have access to all of the threats, can see all of the activities of, for example, executive or leadership of the company, but maybe a junior security analyst doesn't have as much capability.

148
00:22:19,720 --> 00:22:37,700
So we added into our detectors, we have a field that we set a level of access for each of those threats, or it can be based on role, based on a threat, or based on the activity that has been executed.

149
00:22:37,700 --> 00:22:44,800
This will give us the capability to provide service for multiple customers as well.

150
00:22:45,860 --> 00:23:04,560
The same framework that is looking at the same exact same data, it can be used for insider threat detection, but if we have other customers, we can see that if there was any network intrusion, so this is basically CSOC's interest, if there is any website abuse,

151
00:23:04,560 --> 00:23:13,640
or in case of financial companies, we can see if there is any insider trading or any fraud done by the employees.

152
00:23:13,640 --> 00:23:23,060
This doesn't necessarily have to be looking at specific entities or limited to any relationship between those entities.

153
00:23:23,060 --> 00:23:35,860
As long as the application is monitoring and looking at anomaly detection, we can simply change the source of the data and modify the dependencies and actually extract the results.

154
00:23:37,380 --> 00:23:41,760
Thank you so much for your time and attention.

155
00:23:42,080 --> 00:23:43,480
Here's our emails.

156
00:23:43,480 --> 00:23:46,060
We're happy to answer any questions.