WEBVTT 00:01.550 --> 00:17.710 So, welcome to our next talk by Nahid Ghali and Vincent Pham, who I believe are both researchers at Capital One, and they're presenting a detection system for ML-based threats. 00:18.890 --> 00:22.070 So let's let them take it away. 00:23.570 --> 00:28.570 To our talk, Spectrum, an end-to-end framework for ML-based threat monitoring and detection. 00:28.570 --> 00:32.950 I'm Vincent Pham, and my co-author, Nahid Farhadi. 00:32.950 --> 00:37.070 We're both software developers at Capital One. 00:38.310 --> 00:43.670 We're going to talk about a platform that targets insider threat. 00:43.670 --> 00:49.550 So for those of you who are unfamiliar with what insider threat is, we have the formal definition here. 00:49.550 --> 01:02.690 So insider threat is the potential for an individual who has or had authorized access to an organization's asset to use that access, either maliciously or unintentionally, to act in a way that could negatively affect the organization. 01:02.990 --> 01:09.250 This unintentionally is also important because we're not looking at only malicious or suspicious people. 01:09.250 --> 01:16.050 Also people who have naively or unknowingly leaked data from the company as well, since they are a potential threat. 01:16.150 --> 01:22.210 I think this comic on the right side pretty much clarifies who are insider threats. 01:22.210 --> 01:31.670 Basically when you think of the entire compass of who are possible threats in the world, it's pretty much the entire population and then who are insider threats. 01:31.670 --> 01:34.170 That's the people who work within a company. 01:34.670 --> 01:41.170 Even if you have zero risk or very minimal risk, there's still a potential threat there. 01:44.370 --> 01:52.970 And just to give you a little bit about the insider threat stats, 90% of organizations feel vulnerable to insider attacks. 01:52.970 --> 01:58.630 53% of companies had insider threat attack against their organization in the last 12 months. 01:58.630 --> 02:03.050 37% of employees have excessive access privileges. 02:03.050 --> 02:12.490 This could be important because think about your most important data and what would happen if that your company and who has access to that data. 02:12.870 --> 02:17.210 94% of companies monitor their employees digital footprint. 02:17.210 --> 02:22.570 So this is the data sources that we could use to track where the data is going, who has access to what. 02:22.570 --> 02:28.850 And then 86% of companies have or are building an insider threat program to date. 02:29.010 --> 02:38.170 And then also another important fact is the average cost of insider threats is currently about estimated at $1.6 million. 02:41.370 --> 02:50.450 So we mentioned that there's 86% companies who are thinking about or are currently building an insider threat system. 02:50.550 --> 02:59.830 You might be thinking should you buy a platform that is currently have a solution developed or should you build your own solution to insider threat? 03:00.210 --> 03:04.610 We list some of the pros of each of the different buying versus build. 03:04.710 --> 03:11.150 So if you think about buying a platform, the pros of that is that a solution is already developed. 03:11.250 --> 03:15.050 So it could possibly be simple as a plug and play. 03:15.110 --> 03:23.410 Although in most cases, it'll take a couple of months or maybe even a couple of years just to integrate that platform into your system. 03:23.670 --> 03:35.310 Also support is provided so you don't have to build up your own team, you could pay for support to install that system and grant you the analysis to insider threat. 03:35.470 --> 03:40.110 Lastly, the data management and handling solution is also provided by the company as well. 03:40.110 --> 03:42.210 So once again, you don't have to build your own team. 03:42.210 --> 03:46.550 The pros of building your own system is that technology is always changing. 03:46.550 --> 04:00.570 So if your company is thinking about changing its technology, changing its database, or just changing the way everything is handled, moving from maybe a server or a local server to the cloud, you could quickly adapt to that need. 04:01.270 --> 04:13.710 If your system is constantly being audited, you have full knowledge of the systems, you don't have to wait for your vendor to give you back some information, or maybe they're not even going to release the information. 04:14.350 --> 04:17.190 When you're building it yourself, you already have that information. 04:17.190 --> 04:23.170 And then once again, due to ever-changing technology, you might have changing data as well. 04:23.170 --> 04:28.630 So then you could adapt your platform really quickly to that changing data. 04:32.990 --> 04:43.030 So in order to combat insider threat, we built a system called Spectrum, which operates on two types of entities and the relationship. 04:43.030 --> 04:45.970 The entities are the subjects and the objects. 04:46.150 --> 04:57.070 The subjects are identified as the types that take action, such as in a very simple sense for insider threat are the employees. 04:57.070 --> 05:04.990 But you could also consider AM roles in AWS, which is like a group of profiles gathered together. 05:04.990 --> 05:12.530 Or if you're doing insider threats, or I guess external threats, you can also think about customers as your subjects. 05:12.530 --> 05:16.150 And then the objects are entities in which actions are taken. 05:16.190 --> 05:20.010 And those actions are recorded as event trails. 05:20.010 --> 05:37.510 So you could think of like anytime someone logs into a VPN, what time they logged in at, if they batch into a building, if they opened a laptop, events such as that are recordable and provides evidence to whether an insider is happening or not. 05:42.240 --> 05:51.980 So the proposed framework for Spectrum follows such that if we go from the very top, we define a set of entities. 05:51.980 --> 06:01.900 So once again, in this case, that would be the employees, we find out the different data sources that are linked to the entity. 06:02.220 --> 06:06.420 So we might have, in this case, three different events source. 06:06.780 --> 06:12.840 And then from each of the individual events source, we have features for that event source. 06:12.840 --> 06:27.820 For example, let's say you're using your email data set, that email data set will have features such as where the email is being sent to, where it's being sent from, the time of day the email is sent, the subject, and so on and so on. 06:27.860 --> 06:31.400 And then from the features, we can construct patterns. 06:32.520 --> 06:37.560 For example, has the email been sent within the past 24 hours? 06:38.200 --> 06:42.900 Does the content of the email display a positive or negative emotion? 06:43.040 --> 06:44.160 And so on. 06:45.040 --> 06:51.660 We could construct rules from it, we could also do anomaly detection on the feature itself. 06:51.840 --> 06:55.820 So is there something suspicious based away from the baseline? 06:55.940 --> 07:07.000 And then we can also build machinery models such as classifier to identify, once again, whether particular events or set of feature shuffle up something that's suspicious or not. 07:08.040 --> 07:16.740 So once we construct all of these rules, anomaly classifiers, we tied back the dependencies to extractor, which ties back to event source. 07:16.740 --> 07:21.120 And then that's how we built the entire pipeline from the bottom up. 07:21.140 --> 07:35.820 And then when we run the pipeline, it then reconstructs this entire scheme and tied back all the data sources so that if you turn off one pattern, then it might turn off a particular source just to save some cost in the meantime. 07:37.380 --> 07:59.020 Our data is then enriched to the role, the entity, and that is stored into storage with the entity and the enriched data of whether a particular role is suspicious or not or whether, I guess, the classification is marking as a zero to one probability of suspiciousness. 07:59.500 --> 08:18.480 These information are then used to flag a particular entity, maybe another reduced flag, or maybe open up a case management so that analysts can look into, or it could even be displayed in a UI so that analysts can go through this ranked order of suspicious entities, 08:18.480 --> 08:25.440 suspicious events, and see whether this is really a true positive or false positive in terms of inside threat or not. 08:25.500 --> 08:30.280 So, in a nutshell, this is what our system of spectrum looks like. 08:32.100 --> 08:33.020 All right. 08:33.020 --> 08:50.480 So, this proposed framework had some techniques being done on it in order to effectively save money on cost of data handling or making sure that we don't lose any data. 08:50.660 --> 08:55.060 So, using modes of operation and also dependency orchestration. 08:55.060 --> 09:00.080 I'm going to go through some of the techniques that we use to build this whole framework. 09:00.080 --> 09:04.580 The first one is that we have a unified source of the data. 09:04.580 --> 09:15.180 Basically, when we look at different sources of information, like digital footprint, it could be any digital interaction that the employee has with the outside world. 09:15.180 --> 09:33.140 It could be proxy monitoring, mail servers, different sensors that we have on endpoints or user machines or on cloud accounts that monitor all of these traffics and the communication that we have with the outside world. 09:33.140 --> 09:40.260 We can have different tools or event monitoring like windows, event log, sysmon, snort, etc. 09:40.260 --> 09:45.680 And each of these resources provide a specific type of information for us. 09:45.680 --> 10:03.860 But the important thing is that when we want to run the job and we want to detect whether there has been any insider threats actions over the past 24 hours, it is important to only query the past 24 hours for that specific data. 10:03.860 --> 10:16.840 Now, at the development time, your developers are going to develop different features and they are going to each of them query against all of these data sources, which is very costly. 10:16.840 --> 10:30.780 So our proposed solution to have a cost efficiency in a spectrum development phase is that we actually had all of these data sources. 10:30.860 --> 10:40.500 We processed them and we sessionized them and we extracted the information that we really need from each data source into a specific table. 10:40.540 --> 10:55.840 And then we had a Matillion job running on a timely basis, like five times a day, six times a day, as your customer requests you to extract that information, process them and keep them in a table for us. 10:55.840 --> 11:04.900 The size of this table is an order of magnitude smaller compared to the size of the data sources that you originally have. 11:05.000 --> 11:10.420 Plus your developers wouldn't go by themselves and each of them query those huge tables. 11:10.420 --> 11:22.240 Now you have unified all of this information in one table and it is going every time you want to develop, you want to test, you want to add a new feature, you are going to query the smaller table. 11:22.240 --> 11:27.400 So this actually saved us 90 percent of the cost of the querying. 11:27.400 --> 11:46.160 We use Snowflake as the source of the data where we keep our original information and then we use Matillion in order to have a schedule for running jobs on extracting information and processing them from Snowflake. 11:46.760 --> 11:50.940 So here is an example of how we keep the data. 11:51.640 --> 12:00.720 The other benefit or other feature that we propose in implementation of a spectrum is that we unify the data. 12:00.720 --> 12:03.780 This brings a second benefit for us. 12:03.780 --> 12:07.940 First one was that, okay, we had some cost savings. 12:07.940 --> 12:23.240 The second benefit is that this data frame is like an object and it is moving along the pipeline and different actions are being executed on this data frame. 12:23.240 --> 12:35.820 The important thing is that we don't complicate our design by separating different data sources in different tables, having each of those tables have different number of roles. 12:35.820 --> 12:41.740 For example, in email, you have a data type, definitely a hash or a unique ID. 12:41.740 --> 12:51.600 It is related to an entity ID, it could be an employee, it could be an IAM role or whatever entity that is doing the action, and it has a timestamp. 12:51.600 --> 12:56.860 But in meta, you have to from number of attachments, etc. 12:56.860 --> 13:01.060 The same information for a proxy data is very different. 13:01.060 --> 13:19.300 Of course, you have the first four columns, but in meta, you have probably the URL that has been accessed, the type of connection, is it HTTP or what type of connection is it, is it blocked by the company or not, or etc. 13:19.300 --> 13:21.260 Other type of information. 13:21.260 --> 13:26.740 So imagine if we had different data frames for each of these data types. 13:26.740 --> 13:41.260 And you know, you can have up to 20 data sources and having different tables for each of these is going to make your design very complicated and very inflexible against making changes. 13:41.260 --> 13:53.540 But with the proposed method, we keep information related to each data type in a JSON format in just one column, and that column is named meta. 13:53.920 --> 14:11.820 This way, our data frame can have different types of information, but all of them are in the same format, all of them are in one data frame, and that data frame is going through the pipeline, and each of the modules is responsible for doing specific actions on it. 14:11.820 --> 14:13.740 That's why we had extractors. 14:13.740 --> 14:20.740 Those extractors actually extract features and stuff from the meta field. 14:23.600 --> 14:30.500 Another design thinking that was behind this proposed framework is modes of operation. 14:31.060 --> 14:53.580 So as you know, like any other machine learning system, we have a normal job that is running, usually in the POC phase, we call it the test one, which is actually running the trained model against your real-time data and extracting results. 14:53.580 --> 15:00.560 We call that mode modeling, and we run it six times a day, or more per user's request. 15:00.580 --> 15:02.460 We have a training phase. 15:02.460 --> 15:06.360 In that phase, we only train the models. 15:06.360 --> 15:09.740 We have several baseliners, we have several classifiers. 15:09.740 --> 15:14.980 Those baseliners and anomaly detections are for user and entity behavior monitoring. 15:14.980 --> 15:23.620 So we compare employees against their own behavior, or against their peers' behavior, or role-based type of anomaly detection. 15:23.620 --> 15:26.820 So another phase of operation is training. 15:26.820 --> 15:32.920 In training, obviously, we extract more or longer period of the data. 15:33.920 --> 15:40.580 And that's actually where the Matillion job and having a unified table helps us. 15:40.580 --> 15:44.420 In modeling, we only look at the past 24 hours of the data. 15:44.420 --> 15:50.780 In training, we look at, for example, 90 days of the data, or a longer period of time. 15:50.780 --> 16:04.960 So in this case, now that we have two different phases, and we separated these duties from each other, we make sure not to run really long queries for training. 16:05.080 --> 16:07.080 We have performance monitoring. 16:07.080 --> 16:20.420 In this performance monitoring, we monitor the thresholds and metrics for our machine learning models, for classifiers and anomaly detectors, and make sure that they are acting normally. 16:20.420 --> 16:35.360 They are not having any problems with the sources of the data, or with the baseliners, or if we generate alerts off of these, we go back to training and we fine-tune our model. 16:35.360 --> 16:54.280 We also have a backfill mode, which is something that happens usually in production, is that sometimes your sensors don't work, sometimes you don't have the source of the data available, there are problems with running jobs in order to filling out those tables. 16:54.280 --> 17:11.060 So in backfill mode, we run the same modeling, but this time on a longer period of time to make sure that we haven't missed on any of the events due to problems in production and providing the data. 17:12.920 --> 17:18.960 Another important part of this modeling job is a dependency orchestrator. 17:18.960 --> 17:27.720 As I said before, we have different, and Vincent mentioned before, we have different phases of operation. 17:27.720 --> 17:43.220 First, we extract stuff from our data sources, then after we extract those data, we have three main techniques in order to find out if an event is malicious or not. 17:43.220 --> 17:53.460 We have rule-based techniques, which we call pattern matchers, we have baseliners, which is anomaly detection type of job, and then we have machine learning component. 17:53.460 --> 17:57.660 Now, each of these might depend on each other. 17:57.660 --> 18:11.560 For example, on a baseliner, you might want to run the baseliner on a feature that is extracted, or you want to run a baseliner on a specific data type, for example, on print operations. 18:11.560 --> 18:17.600 That doesn't necessarily need to wait on extracting information from email, right? 18:17.600 --> 18:25.420 It's just a baseliner on print operation, so there is no dependency between email events and print events. 18:25.420 --> 18:36.860 So, in the dependency orchestrator, we come up with a plan that what is the most optimized way to run these modules based on their dependencies. 18:36.860 --> 18:58.520 You may have machine learning components that depend on several baseliners or several pattern matchers, you may run different baseliners that depend on two features, and finally, you may detect the maliciousness of something if a specific feature is extracted and it is violating an anomaly. 18:58.520 --> 19:00.860 So, there is all these dependencies. 19:00.860 --> 19:04.740 Here is an example in the module dependency. 19:04.740 --> 19:12.840 For example, here, we read the data from our original database and imagine that each of these colors is an event type. 19:12.840 --> 19:16.220 For example, this is email, this is proxy, and this is print. 19:16.220 --> 19:25.760 We extract information from email, we extract information from print, for example, number of pages, printer that is used, and we extract information from proxy. 19:25.900 --> 19:45.400 Then, we may want to run a baseliner on the email events and see that, amount of emails that someone is sending on a daily basis, or we may want to look at the number of proxy, what is the number of, what is the amount of upload and download for proxy that we use in a baseliner. 19:46.080 --> 19:49.540 Then, later on, we call these as detectors. 19:49.540 --> 20:00.680 These detectors might say that I think this person is malicious if they printed something and if they had some sort of anomaly behavior in terms of proxy. 20:00.680 --> 20:09.880 So, the dependency of this proxy is both on the baseliner and on the extractor of the second event type. 20:10.180 --> 20:28.780 So, basically, the job of dependency orchestrator is to come up with a plan and find out what is the dependency for each of these modules and make sure that they are ordered in a way that each module meets its dependency when it's time for the module to be executed. 20:28.780 --> 20:45.280 Finally, we have a store operation where we store all of those results in Postgres database or any other type of database and we have a list of threats that we have detected for each of the entities. 20:47.340 --> 20:55.420 Another capability that we have in this implementation in the proposed framework, it provides us scalability. 20:55.940 --> 21:12.500 First of all, for the processing engine, we use Spark on AWS EMR and it is composed of fleet instance and we have some spot instances, some on-demand instances on EC2. 21:12.520 --> 21:27.220 As soon as the number of jobs increase, the number of data, the number of events that we are monitoring or the period that we are monitoring, it can totally scale out and perform the job successfully. 21:27.220 --> 21:37.680 Another scalability from another point of view is that we have added RBAC restricted views for our events. 21:37.680 --> 21:58.200 For example, if you have different customers with different levels of access control to the entities that you are monitoring or to the events that you are monitoring, you can use RBAC and role-based access control in order to make sure that, for example, 21:58.200 --> 22:19.720 the head of insider threat monitoring can have access to all of the threats, can see all of the activities of, for example, executive or leadership of the company, but maybe a junior security analyst doesn't have as much capability. 22:19.720 --> 22:37.700 So we added into our detectors, we have a field that we set a level of access for each of those threats, or it can be based on role, based on a threat, or based on the activity that has been executed. 22:37.700 --> 22:44.800 This will give us the capability to provide service for multiple customers as well. 22:45.860 --> 23:04.560 The same framework that is looking at the same exact same data, it can be used for insider threat detection, but if we have other customers, we can see that if there was any network intrusion, so this is basically CSOC's interest, if there is any website abuse, 23:04.560 --> 23:13.640 or in case of financial companies, we can see if there is any insider trading or any fraud done by the employees. 23:13.640 --> 23:23.060 This doesn't necessarily have to be looking at specific entities or limited to any relationship between those entities. 23:23.060 --> 23:35.860 As long as the application is monitoring and looking at anomaly detection, we can simply change the source of the data and modify the dependencies and actually extract the results. 23:37.380 --> 23:41.760 Thank you so much for your time and attention. 23:42.080 --> 23:43.480 Here's our emails. 23:43.480 --> 23:46.060 We're happy to answer any questions.