1
00:00:03,720 --> 00:00:05,680
Sorry about the no sound, people.

2
00:00:05,680 --> 00:00:11,340
I will figure this out.

3
00:00:11,960 --> 00:00:19,140
Every day, it just decides to grab the wrong audio input.

4
00:00:22,240 --> 00:00:26,540
So let's go back to the beginning and see if this works.

5
00:00:26,540 --> 00:00:27,700
Hi everyone.

6
00:00:27,940 --> 00:00:30,820
Today's talk is about machine learning privacy.

7
00:00:30,820 --> 00:00:36,620
The title of the talk is secrets are lies, sharing is caring, privacy is theft.

8
00:00:36,920 --> 00:00:40,700
My name is Nahid Farhadi and I'm a software developer in Capital One.

9
00:00:40,700 --> 00:00:44,120
I will be giving this talk with my colleague Vincent Pham.

10
00:00:45,320 --> 00:00:47,460
The outline of our talk is as below.

11
00:00:47,460 --> 00:01:00,280
We will give an introduction on the importance of privacy and we will talk about privacy attacks, we propose some defense techniques, and finally we will show a demo of an attack as well as defense techniques.

12
00:01:01,920 --> 00:01:04,040
Machine learning needs data.

13
00:01:04,040 --> 00:01:10,300
So what if this data has sensitive information like medical data or financial data?

14
00:01:10,400 --> 00:01:24,860
Even if we trust the algorithms that are generating the model or model developers, sometimes just using the statistics by looking at the output of a model, we can find out some information about the input of the model.

15
00:01:24,880 --> 00:01:35,780
This type of threat to ML can be black box or white box, meaning that the adversary doesn't necessarily need to have any access to the model specifics.

16
00:01:36,320 --> 00:01:38,140
So what is really privacy?

17
00:01:38,140 --> 00:01:52,440
Privacy means that if we are using some data from our users for training a model, but just by looking at the output of the model, no one should be able to get information on the input of the model.

18
00:01:53,760 --> 00:01:57,980
There is a trade-off between privacy and utility.

19
00:01:58,280 --> 00:02:03,160
If we have too much privacy, then it will be very difficult to use the model.

20
00:02:03,160 --> 00:02:10,800
If we don't have much privacy, then it will be useful, but then we will endanger our customers.

21
00:02:10,880 --> 00:02:17,340
PPML is there to define or find the optimized point between utility and privacy.

22
00:02:17,340 --> 00:02:22,360
How can we sacrifice less by getting enough privacy?

23
00:02:23,460 --> 00:02:27,220
My colleague will be talking about privacy attacks next.

24
00:02:28,060 --> 00:02:31,400
In terms of privacy attack, there are two main categories.

25
00:02:31,400 --> 00:02:35,440
One is the classical, where the private data is in the train, where it's in the raw set.

26
00:02:35,440 --> 00:02:39,040
And there are the ML-enabled one, which I'll be focusing more on.

27
00:02:39,260 --> 00:02:47,300
In the next slide, you can see that there are several attack surfaces that attackers can utilize to perform a privacy attack.

28
00:02:47,300 --> 00:02:53,340
The first is the physical domain, in terms of network protection, it could be attack network traffic.

29
00:02:53,400 --> 00:02:57,080
Then there's the digital representation, which is the TCP dump.

30
00:02:57,120 --> 00:03:05,340
There's also the machine learning model, which is just the model itself, where you have an input and do a prediction on the output.

31
00:03:05,740 --> 00:03:08,020
In terms of this, it's the attack probability.

32
00:03:08,020 --> 00:03:11,120
And then you have the physical domain, which is the shutdown infrastructure.

33
00:03:11,260 --> 00:03:13,960
For this talk, we'll focus on the machine learning model.

34
00:03:14,440 --> 00:03:19,260
In the next slide, we will see our first type of attack, which is the linkage attack.

35
00:03:19,360 --> 00:03:34,640
With this attack, in the early 90s, Tina Sweeney found that she was able to identify the identification of the governor of Massachusetts by linking his health records to a voter information just by using three variables, his gender, date of birth, and a zip code.

36
00:03:34,820 --> 00:03:44,400
In the next slide, she was able to identify that using these three variables alone, she was able to uniquely identify 87% of the population in the U.S.

37
00:03:44,500 --> 00:03:47,600
using the census, the 1990 census.

38
00:03:47,600 --> 00:03:56,640
In 2000, Philip Goh identified that using these three records, he was able to identify only 64% of the population.

39
00:03:56,640 --> 00:04:03,160
This could be a result of people urbanizing more, moving from the rural areas to the cities.

40
00:04:03,160 --> 00:04:11,400
But this provides an important point where using just very simple features, attackers can identify back to the original source.

41
00:04:11,930 --> 00:04:15,980
In the next slide, we look at another popular linkage attack.

42
00:04:16,380 --> 00:04:22,560
In 2006, Netflix released a public data set, a competition where they're offering a prize to improve their recommendation.

43
00:04:23,080 --> 00:04:40,000
And their FAQs denoted that there is no private information in the data set itself, but researchers have found that if they were able to link the Netflix accounts back to the IMDb, just by using the name of the person on the IMDb, their public dates, and public ratings,

44
00:04:40,000 --> 00:04:45,000
they'll identify personable information back to Netflix.

45
00:04:45,240 --> 00:04:55,820
You might be wondering why this might be damaging since the people tend to put their public information on IMDb.

46
00:04:55,920 --> 00:05:04,100
This could be because they might be biasing or selecting a set of records to disclose on IMDb, but not on Netflix recommendation.

47
00:05:04,100 --> 00:05:19,140
For example, they were able to identify that for a particular person, they were able to identify just based on his Netflix record, that he might have a voting preference from his voting record on Power and Terror, Norm Chomsky and Our Times, and Periodic 9-11.

48
00:05:19,140 --> 00:05:29,160
They might be able to infer on his political preference based on the Jesus of Nazareth and Gospel of John ratings, and also his eating habit based on his recommendations on supersizing.

49
00:05:29,580 --> 00:05:34,680
In the next slide, we look at a new type of attack called the reconstruction attack.

50
00:05:34,680 --> 00:05:44,700
This is more prone to models that tend to keep the training information in the structure itself, such as near-neighbor classifier and kernel-based SVM.

51
00:05:44,760 --> 00:06:11,900
But this is also prone to neural networks as well, as we can see in the next slide, where in production environments, researchers and even implementers tend to reuse neural network models just because of something called style transfer, or style transfer where they're able to utilize one model and create a new prediction from that same model using the inputs and architecture of it,

52
00:06:11,900 --> 00:06:14,000
and reconstruct back to the original input.

53
00:06:14,500 --> 00:06:33,000
In the next slide, we also see a similar type of attack called model inversion attack, where in this case, a feature such as an image is fed into a neural network model and produces a set of probabilities, and then the researcher can use these probabilities and name and reconstruct an image back to the real thing.

54
00:06:33,140 --> 00:06:43,660
For example, in the right side of the slide, you can see that in the left image, there's this reconstructed image, and in the right is the original image, and they're very similar to each other.

55
00:06:44,180 --> 00:07:00,860
In the next slide, we also have a membership inference attack, where an adversary can generate multiple shadow network models that resembles the production environment, produce the probabilities from these models, create a new model called the attack model to feed in these probabilities,

56
00:07:00,860 --> 00:07:10,400
and learn what observations are trained by the production model and what are not trained by the production model, identify what are the members and what are not the members.

57
00:07:10,900 --> 00:07:14,460
In the next slide, we also have a core attack.

58
00:07:14,460 --> 00:07:23,680
This is more prone to models such as LSTM, or models that learn from a time series of data points.

59
00:07:23,740 --> 00:07:38,660
For this example, you might have a model that learns from social security, and it does an autocomplete, and one of the autocomplete could be some sort of sensitive information, such as the phone number or the social security of a person .

60
00:07:38,920 --> 00:07:50,600
This is more prone to maybe like if you're over-memorizing from the neural network model itself , where the records might appear at least five times.

61
00:07:51,040 --> 00:07:57,780
In the next slide, we have a summary of different privacy attacks that I've talked about earlier on.

62
00:07:57,840 --> 00:08:07,600
You could take a pause here and just look through the difference, compare the different approaches, and look at the examples, but we can move on if you're ready to move on.

63
00:08:09,870 --> 00:08:10,950
Next slide.

64
00:08:11,830 --> 00:08:18,690
All right, so now we're really talking about methods and techniques to preserve privacy.

65
00:08:19,650 --> 00:08:24,050
To preserve privacy, we have four main points to protect.

66
00:08:24,050 --> 00:08:33,030
The training data and input and output of the model shouldn't be visible to anyone who is not authorized to visit them.

67
00:08:33,030 --> 00:08:39,930
And finally, the model privacy, which makes sure that the model cannot be stolen by any malicious party.

68
00:08:40,170 --> 00:08:46,570
Two main categories to preserve privacy are secure computation and privacy preservation.

69
00:08:46,570 --> 00:08:56,270
In case of secure computation, we transform or distribute our computations in a way that is not readable by any unauthorized person.

70
00:08:56,270 --> 00:09:04,950
And in terms of privacy preservation techniques, we use injection, noise injection, or masking and hiding techniques in order to preserve privacy.

71
00:09:05,790 --> 00:09:11,130
The first method of privacy preservation is homomorphic encryption.

72
00:09:11,130 --> 00:09:25,130
Homomorphic encryption is a type of encryption where you are doing the same computations and a third party can do your computations for you, but not on the original form of the data, on an encrypted form of data.

73
00:09:25,130 --> 00:09:44,270
This means that if you use a third party company for your ML computations, you can totally give your data to them in a transformed form, which is using homomorphic encryption, and they can do the computations for you and give you back the results.

74
00:09:44,270 --> 00:09:49,350
And then you can decrypt the data and get the plain text or cipher text as you want.

75
00:09:50,330 --> 00:09:53,450
The second method is secure multi-party computation.

76
00:09:53,450 --> 00:10:04,810
In case of this technique, we have multiple parties who all contribute into building an ML model or a function or to compute a value for a function.

77
00:10:04,810 --> 00:10:10,530
But the thing is that the input is distributed between all of these parties.

78
00:10:10,530 --> 00:10:17,030
So as you see, part of input A1 is here, A2 is here, A3 is here, and so on.

79
00:10:17,030 --> 00:10:28,570
So this way, if an adversary wants to attack the outputs, they need to be able to hack into all of the parties in order to make sense of the data.

80
00:10:28,570 --> 00:10:33,310
Otherwise, getting information from one of the parties is meaningless.

81
00:10:34,450 --> 00:10:37,110
Another technique is federated machine learning.

82
00:10:37,110 --> 00:10:41,350
Federated machine learning is very similar to the previous method.

83
00:10:41,350 --> 00:10:52,030
However, the difference is that we have multiple sets of users, and these users all contribute into building a classifier.

84
00:10:52,750 --> 00:11:05,950
The way that we handle the creation of the outputs is that we assign some sort of weights and locally update the average of classifiers by assigning weights to each of the trained models.

85
00:11:07,830 --> 00:11:12,550
The most important technique to preserve privacy is differential privacy.

86
00:11:12,930 --> 00:11:25,550
Basically, differential privacy means that it's a noise injection technique so that the output of your model shouldn't reveal any information about the input of the model.

87
00:11:25,550 --> 00:11:35,110
Whether your model is trained using the data of this person or the data of this person shouldn't be revealed in the outcome of your algorithm.

88
00:11:37,230 --> 00:11:41,470
It can be non-interactive or interactive.

89
00:11:41,690 --> 00:11:53,870
For example, in case of non-interactive, we pre-compute the stats, the amount of noise, the amount of relationship between the input features, and then inject noise to that.

90
00:11:53,870 --> 00:12:03,770
In case of interactive, the amount of noise is injected based on user's requests.

91
00:12:05,350 --> 00:12:08,150
What does differential privacy mean?

92
00:12:08,150 --> 00:12:21,570
It means that if person i change their information from x i to x prime i, then the probability of their output should not change by that much.

93
00:12:21,570 --> 00:12:23,330
What does it mean again?

94
00:12:23,330 --> 00:12:38,430
It means that as a person, if one of my features is changing, and as a person who is contributing into training a model, that change shouldn't cause a huge change in the output.

95
00:12:38,430 --> 00:12:46,890
Because if that causes that huge change, it will reveal some information about this instance or observance as an input.

96
00:12:46,890 --> 00:13:03,550
For example, if I change the color of my hair from brown to green, and that has a huge effect on the output of the model, then just by observing that change, an adversary will know that I changed the color of my hair.

97
00:13:03,550 --> 00:13:10,490
So using differential privacy, we want to make that change as minimal as possible.

98
00:13:10,490 --> 00:13:11,870
And how do we do that?

99
00:13:11,870 --> 00:13:19,050
By computing what is the effect of that change and making that as a noise input into our model.

100
00:13:19,870 --> 00:13:26,350
So looking again at the differential privacy, you see here that we have two datasets.

101
00:13:26,350 --> 00:13:36,570
We call these neighbor datasets, meaning that all of the features in both datasets are actually equal, except that one feature.

102
00:13:36,570 --> 00:13:39,650
In here, it is x3 and x3 prime.

103
00:13:39,650 --> 00:13:51,470
We want to make sure that when we change the input data from x3 to x3 prime, the output that is generated by the model is changing minimally.

104
00:13:51,470 --> 00:14:00,370
So as you see, the probability of generating a specific output in the first case to the second case is minimal, or epsilon.

105
00:14:01,310 --> 00:14:05,930
The next method is called PATE, private aggregation of teacher ensembles.

106
00:14:05,930 --> 00:14:16,330
In this case, we have several classifiers that use some data use some features in order to train a model.

107
00:14:16,330 --> 00:14:18,750
Then we have two sets of voters.

108
00:14:18,750 --> 00:14:22,030
It's like an inversion programming type of technique.

109
00:14:22,030 --> 00:14:27,430
These voters choose the majority voting generated by all of these classifiers.

110
00:14:27,590 --> 00:14:43,410
And if we have equal amount of votes, for example, in this case, we have two cancers and two healthy results, then we will calculate a Gaussian noise to inject to the voters data.

111
00:14:43,470 --> 00:14:55,530
So basically, we again go back into our training data, and we take a look and see that how much noise does each classifier needs to be injected into.

112
00:14:55,530 --> 00:15:03,490
Then again, we look at we inject that noise and we get the results off of this chart based on the results.

113
00:15:03,490 --> 00:15:10,090
So based on the amount of noise injected, the model might generate the value cancer or might generate the value healthy.

114
00:15:10,090 --> 00:15:25,010
The important thing is that whatever the output is, for example, in this case, it is cancer, the adversary cannot correlate it back to the data that is trained in order to get the value of cancer here.

115
00:15:25,010 --> 00:15:32,530
Because even if we don't have we have the value healthy here by the noise injection, these might also generate the value cancer.

116
00:15:32,530 --> 00:15:45,050
So we don't really know is James Smith the cause of generating the result cancer or if someone similar to James Smith in this training data set is contributing into that result.

117
00:15:45,690 --> 00:16:00,550
Here is the comparison of the defense methods in terms of what they're emphasizing on, what are they trying to protect, whether it is the data owner or the model protection, and their use cases in different applications.

118
00:16:00,910 --> 00:16:08,610
Most importantly, most of these methods are really effective when they're used in combination with each other, one from each category.

119
00:16:08,610 --> 00:16:19,690
For example, homomorphic encryption and differential privacy or SMP and differential privacy, PATE and federated lightning plus homomorphic encryption.

120
00:16:20,650 --> 00:16:24,330
Here are the practical methods for privacy preservation.

121
00:16:24,330 --> 00:16:26,370
They are very easy to use.

122
00:16:26,570 --> 00:16:34,530
There are already packages that are generated for that like TensorFlow or PyTorch or IBM privacy package.

123
00:16:34,530 --> 00:16:41,450
We are going to show an example of how to use one of these packages in a real world sample.

124
00:16:44,350 --> 00:16:55,570
In this demo, we're going to show how you're going to implement a solution from an adversary point of view, and then from a defense point of view, how you can easily apply differentiable private method into your model.

125
00:16:56,250 --> 00:17:03,170
So here in the references, you'll see the data that we use, which is a data source on Kaggle of the purchase dataset.

126
00:17:03,510 --> 00:17:10,010
This contains information that you can use to create labels, such as a binary classifier.

127
00:17:10,130 --> 00:17:14,910
In some cases, it could be like fraud, or if it's fraudulent or not, or it could be abusive or not.

128
00:17:14,910 --> 00:17:30,950
Then we can also create a set of multi-class labels itself, such as the class for a customer, or it could be the category of a merchant , or any other multi-class that you can think of relevant to your use case.

129
00:17:31,970 --> 00:17:36,550
And you can see that we're also going to show you the TensorFlow privacy method as well.

130
00:17:36,950 --> 00:17:40,490
So first, we're going to look at how a shadow network is created.

131
00:17:40,490 --> 00:17:41,850
It's really easy.

132
00:17:41,850 --> 00:17:44,790
First, you create the set of shadow networks.

133
00:17:44,850 --> 00:18:03,890
Then you output the predictions from the shadow networks, feed that into a TAC network, and then train a target network that is similar to what you think the production environment would look like, and then see whether the prediction from that target network is predicted as an in-member or out-member from the TAC network.

134
00:18:04,190 --> 00:18:09,770
If we look at the shadow network, all you have to do is create a very simple for loop.

135
00:18:09,830 --> 00:18:17,190
So if you go down, you can see that all you have to do is create n number of shadow networks.

136
00:18:17,190 --> 00:18:20,330
They could be similar to each other, or they could be of different structure.

137
00:18:20,330 --> 00:18:40,210
But for our use case, we're going to create one type of TensorFlow model, a very simple one with an initial density of 64 nodes, and then dropout, and then another dense layer of 24 nodes, and then finally the softmax for a prediction of the multi-class or binary class.

138
00:18:40,490 --> 00:18:50,470
Each of these n shadow networks will have the same networks, and it'll produce a prediction of how likely it is to be in a certain class or not.

139
00:18:51,470 --> 00:18:56,690
If we look at the TAC network, this is where then you're feeding the predictions of the shadow networks.

140
00:18:56,690 --> 00:19:04,690
We're then training a very fine model to distinguish the threshold of whether a particular observation is within a network or not.

141
00:19:04,750 --> 00:19:05,890
So this is very simple.

142
00:19:06,430 --> 00:19:09,230
You're training samples from the shadow network.

143
00:19:09,230 --> 00:19:15,990
You're going to identify if it came from that particular shadow network or not, and that would be pretty much your label.

144
00:19:17,370 --> 00:19:31,050
If we go to the target DP, this will reveal how easy it is to implement a differential private method for your defense mechanism if you already have a deep learning TensorFlow model in production.

145
00:19:31,170 --> 00:19:36,710
So all you really have to do is import the TensorFlow privacy library and then update your optimizer.

146
00:19:36,710 --> 00:19:49,850
So for in cell 11, you can see that the model is the same, but in cell 12, you can see that the only thing that changes is the optimizer, which is a DP gradient descent constant optimizer with four inputs that you would have to fine-tune.

147
00:19:49,850 --> 00:19:53,530
Dial-to-norm clip, noise multiplier, non-micro-batches, and learning rate.

148
00:19:53,530 --> 00:19:56,410
And then you have your differential private method.

149
00:19:59,170 --> 00:20:14,170
And then if we go back to the first one, here we show results of the different number of shadow networks with the different number of classes, 10 classes, one-inch classes, and the binary two classes, and whether it was differential private or not.

150
00:20:14,170 --> 00:20:21,510
And you can see the different types of accuracy, test accuracy, and then the recall and precision rate as well.

151
00:20:22,470 --> 00:20:24,230
This is just a summary.

152
00:20:24,230 --> 00:20:28,350
If you're interested, you can pause here and look at it for further details.

153
00:20:35,590 --> 00:20:38,410
Okay, so this is the end of our presentation.

154
00:20:38,410 --> 00:20:40,190
Thank you so much for your time.

155
00:20:40,190 --> 00:20:45,350
If you have questions, please reach out to our emails included in the slide deck.