WEBVTT 00:03.720 --> 00:05.680 Sorry about the no sound, people. 00:05.680 --> 00:11.340 I will figure this out. 00:11.960 --> 00:19.140 Every day, it just decides to grab the wrong audio input. 00:22.240 --> 00:26.540 So let's go back to the beginning and see if this works. 00:26.540 --> 00:27.700 Hi everyone. 00:27.940 --> 00:30.820 Today's talk is about machine learning privacy. 00:30.820 --> 00:36.620 The title of the talk is secrets are lies, sharing is caring, privacy is theft. 00:36.920 --> 00:40.700 My name is Nahid Farhadi and I'm a software developer in Capital One. 00:40.700 --> 00:44.120 I will be giving this talk with my colleague Vincent Pham. 00:45.320 --> 00:47.460 The outline of our talk is as below. 00:47.460 --> 01:00.280 We will give an introduction on the importance of privacy and we will talk about privacy attacks, we propose some defense techniques, and finally we will show a demo of an attack as well as defense techniques. 01:01.920 --> 01:04.040 Machine learning needs data. 01:04.040 --> 01:10.300 So what if this data has sensitive information like medical data or financial data? 01:10.400 --> 01:24.860 Even if we trust the algorithms that are generating the model or model developers, sometimes just using the statistics by looking at the output of a model, we can find out some information about the input of the model. 01:24.880 --> 01:35.780 This type of threat to ML can be black box or white box, meaning that the adversary doesn't necessarily need to have any access to the model specifics. 01:36.320 --> 01:38.140 So what is really privacy? 01:38.140 --> 01:52.440 Privacy means that if we are using some data from our users for training a model, but just by looking at the output of the model, no one should be able to get information on the input of the model. 01:53.760 --> 01:57.980 There is a trade-off between privacy and utility. 01:58.280 --> 02:03.160 If we have too much privacy, then it will be very difficult to use the model. 02:03.160 --> 02:10.800 If we don't have much privacy, then it will be useful, but then we will endanger our customers. 02:10.880 --> 02:17.340 PPML is there to define or find the optimized point between utility and privacy. 02:17.340 --> 02:22.360 How can we sacrifice less by getting enough privacy? 02:23.460 --> 02:27.220 My colleague will be talking about privacy attacks next. 02:28.060 --> 02:31.400 In terms of privacy attack, there are two main categories. 02:31.400 --> 02:35.440 One is the classical, where the private data is in the train, where it's in the raw set. 02:35.440 --> 02:39.040 And there are the ML-enabled one, which I'll be focusing more on. 02:39.260 --> 02:47.300 In the next slide, you can see that there are several attack surfaces that attackers can utilize to perform a privacy attack. 02:47.300 --> 02:53.340 The first is the physical domain, in terms of network protection, it could be attack network traffic. 02:53.400 --> 02:57.080 Then there's the digital representation, which is the TCP dump. 02:57.120 --> 03:05.340 There's also the machine learning model, which is just the model itself, where you have an input and do a prediction on the output. 03:05.740 --> 03:08.020 In terms of this, it's the attack probability. 03:08.020 --> 03:11.120 And then you have the physical domain, which is the shutdown infrastructure. 03:11.260 --> 03:13.960 For this talk, we'll focus on the machine learning model. 03:14.440 --> 03:19.260 In the next slide, we will see our first type of attack, which is the linkage attack. 03:19.360 --> 03:34.640 With this attack, in the early 90s, Tina Sweeney found that she was able to identify the identification of the governor of Massachusetts by linking his health records to a voter information just by using three variables, his gender, date of birth, and a zip code. 03:34.820 --> 03:44.400 In the next slide, she was able to identify that using these three variables alone, she was able to uniquely identify 87% of the population in the U.S. 03:44.500 --> 03:47.600 using the census, the 1990 census. 03:47.600 --> 03:56.640 In 2000, Philip Goh identified that using these three records, he was able to identify only 64% of the population. 03:56.640 --> 04:03.160 This could be a result of people urbanizing more, moving from the rural areas to the cities. 04:03.160 --> 04:11.400 But this provides an important point where using just very simple features, attackers can identify back to the original source. 04:11.930 --> 04:15.980 In the next slide, we look at another popular linkage attack. 04:16.380 --> 04:22.560 In 2006, Netflix released a public data set, a competition where they're offering a prize to improve their recommendation. 04:23.080 --> 04:40.000 And their FAQs denoted that there is no private information in the data set itself, but researchers have found that if they were able to link the Netflix accounts back to the IMDb, just by using the name of the person on the IMDb, their public dates, and public ratings, 04:40.000 --> 04:45.000 they'll identify personable information back to Netflix. 04:45.240 --> 04:55.820 You might be wondering why this might be damaging since the people tend to put their public information on IMDb. 04:55.920 --> 05:04.100 This could be because they might be biasing or selecting a set of records to disclose on IMDb, but not on Netflix recommendation. 05:04.100 --> 05:19.140 For example, they were able to identify that for a particular person, they were able to identify just based on his Netflix record, that he might have a voting preference from his voting record on Power and Terror, Norm Chomsky and Our Times, and Periodic 9-11. 05:19.140 --> 05:29.160 They might be able to infer on his political preference based on the Jesus of Nazareth and Gospel of John ratings, and also his eating habit based on his recommendations on supersizing. 05:29.580 --> 05:34.680 In the next slide, we look at a new type of attack called the reconstruction attack. 05:34.680 --> 05:44.700 This is more prone to models that tend to keep the training information in the structure itself, such as near-neighbor classifier and kernel-based SVM. 05:44.760 --> 06:11.900 But this is also prone to neural networks as well, as we can see in the next slide, where in production environments, researchers and even implementers tend to reuse neural network models just because of something called style transfer, or style transfer where they're able to utilize one model and create a new prediction from that same model using the inputs and architecture of it, 06:11.900 --> 06:14.000 and reconstruct back to the original input. 06:14.500 --> 06:33.000 In the next slide, we also see a similar type of attack called model inversion attack, where in this case, a feature such as an image is fed into a neural network model and produces a set of probabilities, and then the researcher can use these probabilities and name and reconstruct an image back to the real thing. 06:33.140 --> 06:43.660 For example, in the right side of the slide, you can see that in the left image, there's this reconstructed image, and in the right is the original image, and they're very similar to each other. 06:44.180 --> 07:00.860 In the next slide, we also have a membership inference attack, where an adversary can generate multiple shadow network models that resembles the production environment, produce the probabilities from these models, create a new model called the attack model to feed in these probabilities, 07:00.860 --> 07:10.400 and learn what observations are trained by the production model and what are not trained by the production model, identify what are the members and what are not the members. 07:10.900 --> 07:14.460 In the next slide, we also have a core attack. 07:14.460 --> 07:23.680 This is more prone to models such as LSTM, or models that learn from a time series of data points. 07:23.740 --> 07:38.660 For this example, you might have a model that learns from social security, and it does an autocomplete, and one of the autocomplete could be some sort of sensitive information, such as the phone number or the social security of a person . 07:38.920 --> 07:50.600 This is more prone to maybe like if you're over-memorizing from the neural network model itself , where the records might appear at least five times. 07:51.040 --> 07:57.780 In the next slide, we have a summary of different privacy attacks that I've talked about earlier on. 07:57.840 --> 08:07.600 You could take a pause here and just look through the difference, compare the different approaches, and look at the examples, but we can move on if you're ready to move on. 08:09.870 --> 08:10.950 Next slide. 08:11.830 --> 08:18.690 All right, so now we're really talking about methods and techniques to preserve privacy. 08:19.650 --> 08:24.050 To preserve privacy, we have four main points to protect. 08:24.050 --> 08:33.030 The training data and input and output of the model shouldn't be visible to anyone who is not authorized to visit them. 08:33.030 --> 08:39.930 And finally, the model privacy, which makes sure that the model cannot be stolen by any malicious party. 08:40.170 --> 08:46.570 Two main categories to preserve privacy are secure computation and privacy preservation. 08:46.570 --> 08:56.270 In case of secure computation, we transform or distribute our computations in a way that is not readable by any unauthorized person. 08:56.270 --> 09:04.950 And in terms of privacy preservation techniques, we use injection, noise injection, or masking and hiding techniques in order to preserve privacy. 09:05.790 --> 09:11.130 The first method of privacy preservation is homomorphic encryption. 09:11.130 --> 09:25.130 Homomorphic encryption is a type of encryption where you are doing the same computations and a third party can do your computations for you, but not on the original form of the data, on an encrypted form of data. 09:25.130 --> 09:44.270 This means that if you use a third party company for your ML computations, you can totally give your data to them in a transformed form, which is using homomorphic encryption, and they can do the computations for you and give you back the results. 09:44.270 --> 09:49.350 And then you can decrypt the data and get the plain text or cipher text as you want. 09:50.330 --> 09:53.450 The second method is secure multi-party computation. 09:53.450 --> 10:04.810 In case of this technique, we have multiple parties who all contribute into building an ML model or a function or to compute a value for a function. 10:04.810 --> 10:10.530 But the thing is that the input is distributed between all of these parties. 10:10.530 --> 10:17.030 So as you see, part of input A1 is here, A2 is here, A3 is here, and so on. 10:17.030 --> 10:28.570 So this way, if an adversary wants to attack the outputs, they need to be able to hack into all of the parties in order to make sense of the data. 10:28.570 --> 10:33.310 Otherwise, getting information from one of the parties is meaningless. 10:34.450 --> 10:37.110 Another technique is federated machine learning. 10:37.110 --> 10:41.350 Federated machine learning is very similar to the previous method. 10:41.350 --> 10:52.030 However, the difference is that we have multiple sets of users, and these users all contribute into building a classifier. 10:52.750 --> 11:05.950 The way that we handle the creation of the outputs is that we assign some sort of weights and locally update the average of classifiers by assigning weights to each of the trained models. 11:07.830 --> 11:12.550 The most important technique to preserve privacy is differential privacy. 11:12.930 --> 11:25.550 Basically, differential privacy means that it's a noise injection technique so that the output of your model shouldn't reveal any information about the input of the model. 11:25.550 --> 11:35.110 Whether your model is trained using the data of this person or the data of this person shouldn't be revealed in the outcome of your algorithm. 11:37.230 --> 11:41.470 It can be non-interactive or interactive. 11:41.690 --> 11:53.870 For example, in case of non-interactive, we pre-compute the stats, the amount of noise, the amount of relationship between the input features, and then inject noise to that. 11:53.870 --> 12:03.770 In case of interactive, the amount of noise is injected based on user's requests. 12:05.350 --> 12:08.150 What does differential privacy mean? 12:08.150 --> 12:21.570 It means that if person i change their information from x i to x prime i, then the probability of their output should not change by that much. 12:21.570 --> 12:23.330 What does it mean again? 12:23.330 --> 12:38.430 It means that as a person, if one of my features is changing, and as a person who is contributing into training a model, that change shouldn't cause a huge change in the output. 12:38.430 --> 12:46.890 Because if that causes that huge change, it will reveal some information about this instance or observance as an input. 12:46.890 --> 13:03.550 For example, if I change the color of my hair from brown to green, and that has a huge effect on the output of the model, then just by observing that change, an adversary will know that I changed the color of my hair. 13:03.550 --> 13:10.490 So using differential privacy, we want to make that change as minimal as possible. 13:10.490 --> 13:11.870 And how do we do that? 13:11.870 --> 13:19.050 By computing what is the effect of that change and making that as a noise input into our model. 13:19.870 --> 13:26.350 So looking again at the differential privacy, you see here that we have two datasets. 13:26.350 --> 13:36.570 We call these neighbor datasets, meaning that all of the features in both datasets are actually equal, except that one feature. 13:36.570 --> 13:39.650 In here, it is x3 and x3 prime. 13:39.650 --> 13:51.470 We want to make sure that when we change the input data from x3 to x3 prime, the output that is generated by the model is changing minimally. 13:51.470 --> 14:00.370 So as you see, the probability of generating a specific output in the first case to the second case is minimal, or epsilon. 14:01.310 --> 14:05.930 The next method is called PATE, private aggregation of teacher ensembles. 14:05.930 --> 14:16.330 In this case, we have several classifiers that use some data use some features in order to train a model. 14:16.330 --> 14:18.750 Then we have two sets of voters. 14:18.750 --> 14:22.030 It's like an inversion programming type of technique. 14:22.030 --> 14:27.430 These voters choose the majority voting generated by all of these classifiers. 14:27.590 --> 14:43.410 And if we have equal amount of votes, for example, in this case, we have two cancers and two healthy results, then we will calculate a Gaussian noise to inject to the voters data. 14:43.470 --> 14:55.530 So basically, we again go back into our training data, and we take a look and see that how much noise does each classifier needs to be injected into. 14:55.530 --> 15:03.490 Then again, we look at we inject that noise and we get the results off of this chart based on the results. 15:03.490 --> 15:10.090 So based on the amount of noise injected, the model might generate the value cancer or might generate the value healthy. 15:10.090 --> 15:25.010 The important thing is that whatever the output is, for example, in this case, it is cancer, the adversary cannot correlate it back to the data that is trained in order to get the value of cancer here. 15:25.010 --> 15:32.530 Because even if we don't have we have the value healthy here by the noise injection, these might also generate the value cancer. 15:32.530 --> 15:45.050 So we don't really know is James Smith the cause of generating the result cancer or if someone similar to James Smith in this training data set is contributing into that result. 15:45.690 --> 16:00.550 Here is the comparison of the defense methods in terms of what they're emphasizing on, what are they trying to protect, whether it is the data owner or the model protection, and their use cases in different applications. 16:00.910 --> 16:08.610 Most importantly, most of these methods are really effective when they're used in combination with each other, one from each category. 16:08.610 --> 16:19.690 For example, homomorphic encryption and differential privacy or SMP and differential privacy, PATE and federated lightning plus homomorphic encryption. 16:20.650 --> 16:24.330 Here are the practical methods for privacy preservation. 16:24.330 --> 16:26.370 They are very easy to use. 16:26.570 --> 16:34.530 There are already packages that are generated for that like TensorFlow or PyTorch or IBM privacy package. 16:34.530 --> 16:41.450 We are going to show an example of how to use one of these packages in a real world sample. 16:44.350 --> 16:55.570 In this demo, we're going to show how you're going to implement a solution from an adversary point of view, and then from a defense point of view, how you can easily apply differentiable private method into your model. 16:56.250 --> 17:03.170 So here in the references, you'll see the data that we use, which is a data source on Kaggle of the purchase dataset. 17:03.510 --> 17:10.010 This contains information that you can use to create labels, such as a binary classifier. 17:10.130 --> 17:14.910 In some cases, it could be like fraud, or if it's fraudulent or not, or it could be abusive or not. 17:14.910 --> 17:30.950 Then we can also create a set of multi-class labels itself, such as the class for a customer, or it could be the category of a merchant , or any other multi-class that you can think of relevant to your use case. 17:31.970 --> 17:36.550 And you can see that we're also going to show you the TensorFlow privacy method as well. 17:36.950 --> 17:40.490 So first, we're going to look at how a shadow network is created. 17:40.490 --> 17:41.850 It's really easy. 17:41.850 --> 17:44.790 First, you create the set of shadow networks. 17:44.850 --> 18:03.890 Then you output the predictions from the shadow networks, feed that into a TAC network, and then train a target network that is similar to what you think the production environment would look like, and then see whether the prediction from that target network is predicted as an in-member or out-member from the TAC network. 18:04.190 --> 18:09.770 If we look at the shadow network, all you have to do is create a very simple for loop. 18:09.830 --> 18:17.190 So if you go down, you can see that all you have to do is create n number of shadow networks. 18:17.190 --> 18:20.330 They could be similar to each other, or they could be of different structure. 18:20.330 --> 18:40.210 But for our use case, we're going to create one type of TensorFlow model, a very simple one with an initial density of 64 nodes, and then dropout, and then another dense layer of 24 nodes, and then finally the softmax for a prediction of the multi-class or binary class. 18:40.490 --> 18:50.470 Each of these n shadow networks will have the same networks, and it'll produce a prediction of how likely it is to be in a certain class or not. 18:51.470 --> 18:56.690 If we look at the TAC network, this is where then you're feeding the predictions of the shadow networks. 18:56.690 --> 19:04.690 We're then training a very fine model to distinguish the threshold of whether a particular observation is within a network or not. 19:04.750 --> 19:05.890 So this is very simple. 19:06.430 --> 19:09.230 You're training samples from the shadow network. 19:09.230 --> 19:15.990 You're going to identify if it came from that particular shadow network or not, and that would be pretty much your label. 19:17.370 --> 19:31.050 If we go to the target DP, this will reveal how easy it is to implement a differential private method for your defense mechanism if you already have a deep learning TensorFlow model in production. 19:31.170 --> 19:36.710 So all you really have to do is import the TensorFlow privacy library and then update your optimizer. 19:36.710 --> 19:49.850 So for in cell 11, you can see that the model is the same, but in cell 12, you can see that the only thing that changes is the optimizer, which is a DP gradient descent constant optimizer with four inputs that you would have to fine-tune. 19:49.850 --> 19:53.530 Dial-to-norm clip, noise multiplier, non-micro-batches, and learning rate. 19:53.530 --> 19:56.410 And then you have your differential private method. 19:59.170 --> 20:14.170 And then if we go back to the first one, here we show results of the different number of shadow networks with the different number of classes, 10 classes, one-inch classes, and the binary two classes, and whether it was differential private or not. 20:14.170 --> 20:21.510 And you can see the different types of accuracy, test accuracy, and then the recall and precision rate as well. 20:22.470 --> 20:24.230 This is just a summary. 20:24.230 --> 20:28.350 If you're interested, you can pause here and look at it for further details. 20:35.590 --> 20:38.410 Okay, so this is the end of our presentation. 20:38.410 --> 20:40.190 Thank you so much for your time. 20:40.190 --> 20:45.350 If you have questions, please reach out to our emails included in the slide deck.