Sorry about the no sound, people. I will figure this out. Every day,
it just decides to grab the wrong audio input. So let's go back to the beginning and see if
this works. Hi everyone. Today's talk is about machine learning privacy. The title of the talk
is secrets are lies, sharing is caring, privacy is theft. My name is Nahid Farhadi and I'm a
software developer in Capital One. I will be giving this talk with my colleague Vincent Pham.
The outline of our talk is as below. We will give an introduction on the importance of privacy
and we will talk about privacy attacks, we propose some defense techniques,
and finally we will show a demo of an attack as well as defense techniques.
Machine learning needs data. So what if this data has sensitive information like medical data
or financial data? Even if we trust the algorithms that are generating the model or model developers,
sometimes just using the statistics by looking at the output of a model, we can find out some
information about the input of the model. This type of threat to ML can be black box or white
box, meaning that the adversary doesn't necessarily need to have any access to the
model specifics. So what is really privacy? Privacy means that if we are using some data
from our users for training a model, but just by looking at the output of the model,
no one should be able to get information on the input of the model.
There is a trade-off between privacy and utility. If we have too much privacy,
then it will be very difficult to use the model. If we don't have much privacy, then
it will be useful, but then we will endanger our customers. PPML is there to define or find
the optimized point between utility and privacy. How can we sacrifice less by getting enough privacy?
My colleague will be talking about privacy attacks next.
In terms of privacy attack, there are two main categories. One is the classical, where the
private data is in the train, where it's in the raw set. And there are the ML-enabled one, which
I'll be focusing more on. In the next slide, you can see that there are several attack surfaces
that attackers can utilize to perform a privacy attack. The first is the physical domain,
in terms of network protection, it could be attack network traffic. Then there's the digital
representation, which is the TCP dump. There's also the machine learning model, which is just
the model itself, where you have an input and do a prediction on the output. In terms of this,
it's the attack probability. And then you have the physical domain, which is the shutdown
infrastructure. For this talk, we'll focus on the machine learning model. In the next slide,
we will see our first type of attack, which is the linkage attack. With this attack, in the early
90s, Tina Sweeney found that she was able to identify the identification of the governor of
Massachusetts by linking his health records to a voter information just by using three variables,
his gender, date of birth, and a zip code. In the next slide, she was able to identify that
using these three variables alone, she was able to uniquely identify 87% of the population in the
U.S. using the census, the 1990 census. In 2000, Philip Goh identified that using these three
records, he was able to identify only 64% of the population. This could be a result of people
urbanizing more, moving from the rural areas to the cities. But this provides an important point
where using just very simple features, attackers can identify back to the original source.
In the next slide, we look at another popular linkage attack. In 2006, Netflix released a
public data set, a competition where they're offering a prize to improve their recommendation.
And their FAQs denoted that there is no private information in the data set itself,
but researchers have found that if they were able to link the Netflix accounts back to the IMDb,
just by using the name of the person on the IMDb, their public dates, and public ratings,
they'll identify personable information back to Netflix. You might be wondering why this might be
damaging since the people tend to put their public information on IMDb. This could be because they
might be biasing or selecting a set of records to disclose on IMDb, but not on Netflix recommendation.
For example, they were able to identify that for a particular person, they were able to identify
just based on his Netflix record, that he might have a voting preference from his voting record on
Power and Terror, Norm Chomsky and Our Times, and Periodic 9-11. They might be able to infer on his political
preference based on the Jesus of Nazareth and Gospel of John ratings, and also his eating habit based
on his recommendations on supersizing. In the next slide, we look at a new type of attack called
the reconstruction attack. This is more prone to models that tend to keep the training information
in the structure itself, such as near-neighbor classifier and kernel-based SVM. But this is also
prone to neural networks as well, as we can see in the next slide, where in production environments,
researchers and even implementers tend to reuse neural network models just because of something
called style transfer, or style transfer where they're able to utilize one model and
create a new prediction from that same model using the inputs and architecture of it,
and reconstruct back to the original input. In the next slide, we also see a similar type of
attack called model inversion attack, where in this case, a feature such as an image is fed into
a neural network model and produces a set of probabilities, and then the researcher can use
these probabilities and name and reconstruct an image back to the real thing. For example,
in the right side of the slide, you can see that in the left image, there's this reconstructed image,
and in the right is the original image, and they're very similar to each other.
In the next slide, we also have a membership inference attack, where an adversary can
generate multiple shadow network models that resembles the production environment,
produce the probabilities from these models, create a new model called the attack model to
feed in these probabilities, and learn what observations are trained by the production model
and what are not trained by the production model, identify what are the members and what are not the
members. In the next slide, we also have a core attack. This is more prone to models such as LSTM,
or models that learn from a time series of data points. For this example, you might have a
model that learns from social security, and it does an autocomplete, and one of the autocomplete
could be some sort of sensitive information, such as the phone number or the social security of a
person. This is more prone to maybe like if you're over-memorizing from the neural network model
itself, where the records might appear at least five times. In the next slide, we have a summary
of different privacy attacks that I've talked about earlier on. You could take a pause here
and just look through the difference, compare the different approaches, and look at the examples,
but we can move on if you're ready to move on. Next slide.
All right, so now we're really talking about methods and techniques to preserve privacy.
To preserve privacy, we have four main points to protect. The training data and input and output
of the model shouldn't be visible to anyone who is not authorized to visit them. And finally,
the model privacy, which makes sure that the model cannot be stolen by any malicious party.
Two main categories to preserve privacy are secure computation and privacy preservation. In case of
secure computation, we transform or distribute our computations in a way that is not readable
by any unauthorized person. And in terms of privacy preservation techniques, we use injection,
noise injection, or masking and hiding techniques in order to preserve privacy.
The first method of privacy preservation is homomorphic encryption. Homomorphic encryption
is a type of encryption where you are doing the same computations and a third party can
do your computations for you, but not on the original form of the data, on an encrypted form
of data. This means that if you use a third party company for your ML computations, you can totally
give your data to them in a transformed form, which is using homomorphic encryption, and they
can do the computations for you and give you back the results. And then you can decrypt the data
and get the plain text or cipher text as you want. The second method is secure multi-party
computation. In case of this technique, we have multiple parties who all contribute into building
an ML model or a function or to compute a value for a function. But the thing is that the input
is distributed between all of these parties. So as you see, part of input A1 is here, A2 is here,
A3 is here, and so on. So this way, if an adversary wants to attack the outputs, they need to
be able to hack into all of the parties in order to make sense of the data. Otherwise, getting
information from one of the parties is meaningless. Another technique is federated machine learning.
Federated machine learning is very similar to the previous method. However, the difference is that
we have multiple sets of users, and these users all contribute into building a classifier.
The way that we handle the creation of the outputs is that we assign some sort of weights
and locally update the average of classifiers by assigning weights to each of the trained models.
The most important technique to preserve privacy is differential privacy.
Basically, differential privacy means that it's a noise injection technique so that
the output of your model shouldn't reveal any information about the input of the model.
Whether your model is trained using the data of this person or the data of this person
shouldn't be revealed in the outcome of your algorithm.
It can be non-interactive or interactive. For example, in case of non-interactive,
we pre-compute the stats, the amount of noise, the amount of relationship between the input
features, and then inject noise to that. In case of interactive, the amount of noise is
injected based on user's requests.
What does differential privacy mean? It means that if person i change their information
from x i to x prime i, then the probability of their output should not change by that much.
What does it mean again? It means that as a person, if one of my features is changing,
and as a person who is contributing into training a model, that change shouldn't cause a huge change
in the output. Because if that causes that huge change, it will reveal some information
about this instance or observance as an input. For example, if I change the color of my hair
from brown to green, and that has a huge effect on the output of the model, then just by observing
that change, an adversary will know that I changed the color of my hair. So using differential
privacy, we want to make that change as minimal as possible. And how do we do that? By computing
what is the effect of that change and making that as a noise input into our model.
So looking again at the differential privacy, you see here that we have two datasets. We call these
neighbor datasets, meaning that all of the features in both datasets are actually equal,
except that one feature. In here, it is x3 and x3 prime. We want to make sure that when we change
the input data from x3 to x3 prime, the output that is generated by the model is changing
minimally. So as you see, the probability of generating a specific output in the first case
to the second case is minimal, or epsilon. The next method is called PATE, private
aggregation of teacher ensembles. In this case, we have several classifiers that use some data
use some features in order to train a model. Then we have two sets of voters. It's like an
inversion programming type of technique. These voters choose the majority voting generated by
all of these classifiers. And if we have equal amount of votes, for example, in this case,
we have two cancers and two healthy results, then we will calculate a Gaussian noise to
inject to the voters data. So basically, we again go back into our training data,
and we take a look and see that how much noise does each classifier needs to be injected into.
Then again, we look at we inject that noise and we get the results off of this chart based on the
results. So based on the amount of noise injected, the model might generate the value cancer or
might generate the value healthy. The important thing is that whatever the output is, for example,
in this case, it is cancer, the adversary cannot correlate it back to the data that is trained
in order to get the value of cancer here. Because even if we don't have we have the value healthy
here by the noise injection, these might also generate the value cancer. So we don't really
know is James Smith the cause of generating the result cancer or if someone similar to
James Smith in this training data set is contributing into that result. Here is the
comparison of the defense methods in terms of what they're emphasizing on, what are they trying to
protect, whether it is the data owner or the model protection, and their use cases in different
applications. Most importantly, most of these methods are really effective when they're used
in combination with each other, one from each category. For example, homomorphic encryption
and differential privacy or SMP and differential privacy, PATE and federated lightning plus
homomorphic encryption. Here are the practical methods for privacy preservation. They are very
easy to use. There are already packages that are generated for that like TensorFlow or PyTorch
or IBM privacy package. We are going to show an example of how to use one of these packages in a
real world sample. In this demo, we're going to show how you're going to implement a solution
from an adversary point of view, and then from a defense point of view, how you can easily apply
differentiable private method into your model. So here in the references, you'll see the data
that we use, which is a data source on Kaggle of the purchase dataset. This contains information
that you can use to create labels, such as a binary classifier. In some cases, it could be like
fraud, or if it's fraudulent or not, or it could be abusive or not. Then we can also create a set
of multi-class labels itself, such as the class for a customer, or it could be the category of a
merchant, or any other multi-class that you can think of relevant to your use case. And you can
see that we're also going to show you the TensorFlow privacy method as well. So first, we're going to
look at how a shadow network is created. It's really easy. First, you create the set of shadow
networks. Then you output the predictions from the shadow networks, feed that into a TAC network,
and then train a target network that is similar to what you think the production environment would
look like, and then see whether the prediction from that target network is predicted as an in-member
or out-member from the TAC network. If we look at the shadow network, all you have to do is create
a very simple for loop. So if you go down, you can see that all you have to do is create
n number of shadow networks. They could be similar to each other, or they could be of different
structure. But for our use case, we're going to create one type of TensorFlow model, a very simple
one with an initial density of 64 nodes, and then dropout, and then another dense layer of 24 nodes,
and then finally the softmax for a prediction of the multi-class or binary class. Each of these
n shadow networks will have the same networks, and it'll produce a prediction of how likely it is
to be in a certain class or not. If we look at the TAC network, this is where then you're feeding
the predictions of the shadow networks. We're then training a very fine model to distinguish
the threshold of whether a particular observation is within a network or not. So this is very
simple. You're training samples from the shadow network. You're going to identify if it came from
that particular shadow network or not, and that would be pretty much your label. If we go to the
target DP, this will reveal how easy it is to implement a differential private method for your
defense mechanism if you already have a deep learning TensorFlow model in production. So all
you really have to do is import the TensorFlow privacy library and then update your optimizer.
So for in cell 11, you can see that the model is the same, but in cell 12, you can see that
the only thing that changes is the optimizer, which is a DP gradient descent constant optimizer
with four inputs that you would have to fine-tune. Dial-to-norm clip, noise multiplier,
non-micro-batches, and learning rate. And then you have your differential private method.
And then if we go back to the first one, here we show results of the different number of shadow
networks with the different number of classes, 10 classes, one-inch classes, and the binary two
classes, and whether it was differential private or not. And you can see the different types of
accuracy, test accuracy, and then the recall and precision rate as well. This is just a summary.
If you're interested, you can pause here and look at it for further details.
Okay, so this is the end of our presentation. Thank you so much for your time.
If you have questions, please reach out to our emails included in the slide deck.