start	end	text
0	3700	Hello, I am Miguel Guevara.
3800	9920	And I'm Brian Gibson. I work in privacy infrastructure at Google, in particular on differential privacy.
10080	15060	And we will talk to you about today our differential privacy.
15420	19260	Before we start, let me play this video for you.
21140	25260	At Google, we use data to make our services work better for you.
25260	29880	And it's our responsibility to keep your data private, safe, and secure.
30000	34880	That's why we continually invest in tools that protect individual privacy.
35020	38320	Differentially private analysis is one of those tools.
39100	46980	Many of our features, like popular times and maps, are only possible because we can learn from the activity patterns of millions of users.
46980	54680	With differential privacy, we can protect an individual's data while still preserving the usefulness of the data at scale.
54680	57340	Let's take a closer look at how it works.
57340	63180	Imagine you're trying to plan a trip to the grocery store and want to know how busy it is at the moment.
63880	72320	We use differentially private analysis on the historical and real-time data of other shoppers to show you how busy a place is.
72720	79460	The analysis adds noise to results so you can look at how crowded a place is without identifying who was there.
79460	91240	Differentially private analysis is what enables us to show millions of our users every day how busy a place is while protecting people's privacy with confidence.
91380	97820	Many important research, business, and social questions can be answered by analyzing large data sets.
97820	104240	Which is why, after years of development, Google has made its differential privacy library open source.
104240	118660	Whether you are a researcher working on new medical treatments or a developer building an exciting app, this open source library can help you easily and affordably gain insights from data in a private, secure way.
118920	125200	Differential privacy is just one of many ways we build privacy protections into our products.
125200	131520	And we will continue to innovate to new technologies that keep your data private, safe, and secure.
132680	144320	Great. Now that we've watched the video, I'll get into trying to talk a little bit more from a high level about differential privacy.
144860	152760	The naive question that we always get is, why don't we just remove identifiers to anonymize data?
153360	161500	Unfortunately, removing identifiers might not be good enough, as well as removing PII.
161520	168600	There are multiple well-known data breaches resulting from publishing the quote-unquote de-identified data.
168600	175240	One classical example involves the publication of de-identified medical records in the state of Washington.
175240	181360	Those records could be partially joined with the voting registry using birthday and postcode.
181360	186680	This was sufficient to identify a majority of individuals in the data set.
186680	196260	In another classical example, as a part of a competition in 2006 that intended to improve the movie recommendation system,
196260	202240	Netflix published movie ratings from its users. The records had the user identifiers removed,
202240	209180	but researchers could partially join these records with IMDB databases, which were tied to identity.
209540	215480	Such a join made possible to partially tie the full history of movie ranking to the user identifiers.
215480	222540	This is a classical example of joining de-identified data sets and re-identifying some of the users.
222680	229560	Finally, de-identified search history can also, in certain cases, reveal the identity of a user.
229660	236600	So, there is a pattern here. Even if the records are published without direct identifying information,
236600	245460	the content of those records together with some auxiliary information may make re-identification possible in certain cases.
245480	249840	Thus, we need something better than de-identification in general.
250340	256300	The second question that is very common to have is, well, why don't we just aggregate data?
256560	264680	In fact, it is a very well-known technique to publish aggregated statistics with buckets of users.
264740	273240	Usually, this involves grouping by a set of features. For example, grouping by restaurant visits, by location, and hour.
273240	282260	This is commonly referred to as k-anonymity, since each user in the anonymized dataset is indistinguishable from at least k other users.
285580	292060	Now, let me explain to you why we need differential privacy and how it works.
292060	298240	Let's imagine you are attending a two-day conference and the moderator shares some insights about the audience.
298340	302580	Here, you can see the country of origin of the participants on day one.
302580	308480	It's aggregated, so it does not reveal anything about any of the individuals in the room, right?
308700	317160	And it looks like the top country of origin is Italy, whereas Germany has the least participants and France and Spain are in the middle.
317760	325080	On day two, the moderator shows the stats for the day. They look pretty similar, right?
325080	333380	Except, someone has been paying attention and noticed that there was only one new person on day two.
333700	337960	And when they look at the data from both days together, look at that.
338940	343160	Can you tell where the specific individual is coming from? Yes.
343420	353780	Germany. This is how, with two aggregated datasets, the moderator has managed to leak information about a particular participant.
354620	362760	K-anonymity provides stronger guarantees than simple de-identification, but it is not perfect either.
362760	372760	For example, it is problematic if I'm part of a bucket that is associated with a sensitive attribute, say a particular medical condition.
372860	378680	It's not helpful that I'm indistinguishable from K other users having the same condition.
378680	388120	While I cannot be associated with any particular record in the database, my sensitive attribute in that example is revealed.
388120	398220	Here is another example. Imagine two aggregate buckets, 99 users plus me and 99 users without me.
398220	410600	Publishing aggregate information about those two buckets is K-anonymous, but it's possible to identify me and my sensitive attributes by differencing those two buckets.
410600	423440	At last, if the attacker can modify the database and create fake entries, there's no anonymity for my data if I'm part of a crowd of 99 fake users.
423440	430480	Those type of attacks aren't always feasible or relevant, but they cannot be excluded in general.
430620	439650	Is there a better way to define anonymity? I think so. Let's use differential privacy.
440650	453690	The moderator would similarly announce that they're showing insights about the audience, but they would like to announce, I'm going to share insights about the audience, randomly changing the numbers just a little.
454070	469630	Look at the day one stats, how these stats look compared to the real numbers. They're slightly different. So yes, it is less precise, but you still get the same insights about the statistical trends.
472170	487510	Similarly, let's look at day two. Same thing. But now, if you compare the stats for day one and day two, you can't tell which country the single person is coming from.
487510	495410	And you know it's not worth trying since you know the numbers were changed a little. This is the principle behind differential privacy.
495570	514510	Differential privacy introduces the right amount of noise to the data, so we're able to provide useful statistically significant insights while ensuring that no one can tell if a particular individual's information was used in the competition in a mathematically proven way.
515810	527950	Just to recap, and Brian will talk a little bit more about it in a bit, an algorithm is differentially private if its output doesn't change much when a single person is added to the database.
528010	541990	This has a rigorous mathematical definition, which basically considers a pair of two neighboring data sets, calculates the output of those data sets, and defines a threshold of the probability density functions.
541990	548450	This threshold is what defines how much the output is allowed to change, and it's called epsilon.
549050	557010	Now, let me talk to you about some open sourcing efforts we've done at Google to share some of this technology with the world.
557010	562750	We publish our code in an open source library so that others can reuse our work.
562750	569890	As I mentioned before, differential privacy has a lot of implementation subtopics, and it is hard to get it right.
569890	576150	As all of you in the crypto community probably know, people should enroll their own crypto.
576570	582230	And this is also a really good advice for differential privacy and their implementations.
582430	586510	Reusing a hardened implementation is generally a good idea.
586890	594150	We open source the technology, and you can read more of the articles that were written back then.
595210	599510	I'll now give you some insights as to how we've used the technology.
601350	605130	You're probably aware of the Google Mobility Reports.
605130	612430	These Mobility Reports were a response to the emerging crisis back in March around COVID-19.
612430	620890	They provide policymakers with an idea of mobility trends at a city and state level for different categories.
621030	627610	Before I continue going into the Mobility Reports, let me talk to you a little bit about location data at Google.
629430	635270	Location data is an optional account feature that says where you go in your private timeline.
635270	639390	It's off by default and can be paused or unpaused anytime.
639990	644390	From a product perspective, it's available on Android and iOS.
644990	656690	It helps users remember where they have been and gives users personalized information such as better restaurant recommendations, suggestions for a faster commute, etc.
656690	667090	Users can view, edit, and delete some or all of their timeline at any time, as well as set up auto deletion after three or 18 months.
667090	672270	More recently, we announced that auto deletion is now automatic for new accounts.
672590	681110	And if you're curious as to how it looks like, you can access it as it's been shown on this GIF.
682690	685230	Now, let me talk to you about the reports.
685230	689270	The reports are the Mobility Reports.
689270	695630	These reports were the first product that Google launched that was built with differential privacy from the ground up.
696130	704570	The reports do not share data about any individual's locations, movements, or conflicts.
704950	714930	The insights are created with aggregated, anonymized datasets from users who have turned on the location history setting, which is off by default.
715230	719590	People can turn off that setting at any time and delete their history.
720130	727310	We took the additional step of utilizing differential privacy to ensure that no individual can be identified.
727870	730990	So how exactly are we using differential privacy?
731030	737310	First, we produce a count of people at certain locations within a geographical area.
737310	741930	Let's say that we count the visitors to parks in Vegas on a given day.
741930	746590	That might give us a number of 8,238 people.
747030	751170	Secondly, we add what we call some random noise to that number.
751310	757550	Let's assume, for example, that we added an artificial number of 79.
757870	762950	We add these two and get a new count of 8,317.
763310	770270	This count is differentially private, and we can now use it to calculate the metrics that we publish.
771150	779170	This is an example of a differentially private query that we do to produce reports like the mobility report.
779310	790210	It's helpful for people who are familiar with the SQL environment because it uses the general syntax by only introducing new anonymization functions.
790210	793650	In this case, an anonymization count.
794910	806110	Generally, to produce the metrics, in the reports, we compute a baseline using differential privacy for a given weekday in the past.
806310	810330	Second, we compute those same metrics for a given day.
810330	816910	Let's say for Friday, March 27, 5,000 people visited parks in the same region.
816910	825530	Then we divide that result by the first result, showing that there was a 50% reduction in visits to parks in that region.
825530	829210	And then we repeat this process for all the metrics that we are producing.
829210	833910	And therefore, all the metrics that we're reporting are produced with differential privacy.
834730	840410	There are some interesting practical aspects on how we use these reports.
840410	845070	We limit the number of times that a person is counted on each metric.
845070	852290	For example, if for some reason I go to the grocery store 100 times in a day, I will only be counted once.
852390	858330	This prevents specific patterns like delivery worker people from being too revealing.
859490	863310	Second, we remove metrics that are too noisy.
863450	873570	By using confidence intervals, we can figure out when a number has a less than 5% chance of being wrong by more than 10%.
873570	880630	And finally, if you're curious, we published a technical blog detailing the anonymization process.
882570	891950	Differential privacy has been around for a while, and mobility reports are not the only example out there that's using it.
891950	896050	The US Census is using it for its 2020 census.
896050	899450	Facebook recently released an URL data set.
899690	902530	Apple has been using it.
902530	904550	We released it on Chrome a while ago.
904630	909950	And it's also open source for machine learning purposes on TensorFlow privacy.
910250	915650	Now, Brian will talk to you a little bit more about building the infrastructure.
915830	922290	So getting to the point where we could build a new product from the ground up with differential privacy was a long journey.
922290	928150	But I want to take you through three really critically important pieces of that journey, which is building strong foundations,
928150	931190	building frameworks that are familiar to potential users,
931190	935190	and helping users to assess the privacy utility frontier.
935490	941270	So the infrastructure that we build starts with the differentially private foundational libraries.
941290	950290	We started these about three years ago, and our vision was to power differentially private use cases that could primarily accelerate developer velocity.
950290	957490	A lot of times users are trying to do experiments or prototypes before actually building a project,
957490	961170	and this ideation stage is perfect for differential privacy,
961170	965870	because it can provide a very good way of getting trends or overall statistics,
965870	972510	and allows people to assess whether it's even worth continuing development on a particular project.
973470	980970	So the foundational library that we use here at Google is actually the same as the one that we open sourced and published,
980970	983150	so that others could reuse our work.
983150	987310	For us, it's always been important to provide transparency around our implementation.
988070	991710	Hopefully folks have had a chance to poke around the library.
992690	999710	It was launched in C++ several months ago, and a couple of weeks ago we also launched a Java version of this.
999730	1009410	I think the analogy we oftentimes use with differential privacy is best compared to cryptography, right?
1009450	1014570	There you say, don't roll your own crypto, same thing goes for differential privacy.
1014570	1019230	There are a lot of gotchas, there's a lot of difficulty involved in rolling it out correctly,
1019230	1025050	and an incorrect or partial implementation of DP is in some cases worse than none at all,
1025050	1028830	because you might have a gaping hole in your system that you're not aware of.
1028830	1031990	So reusing a hardened implementation is just a good idea.
1033370	1037290	So the second step we did was to build a SQL engine,
1037290	1041810	and there were lots of languages we could have used, lots of ways we could have gotten about this,
1041810	1044750	but SQL really made the most sense, right?
1044750	1049030	It's the language that most developers are familiar with,
1049030	1054390	you don't have to be an expert to get in and start doing an analysis on data sets,
1054930	1057670	and so it just felt like the right choice.
1057670	1063470	So I'm just going to give a little bit of an overview of some of the challenges involved in this.
1063810	1068170	So one of the big ones is that SQL allows pretty much anything.
1068170	1070710	A lot of implementations are actually Turing-complete,
1070710	1072950	so enforcing differential privacy,
1073250	1079590	while at the same time allowing non-experts to write SQL that they're basically familiar with is a big challenge.
1079710	1084650	A lot of challenges come up around things like joins between different data sets,
1084650	1088570	or cardinality changes, if you're allowed to just multiply rows endlessly,
1088570	1092350	lots of privacy implications come out of that.
1093130	1100210	So the basis of differential privacy is really about quantifying the effect that a single user can have on output.
1100710	1104590	And part of a differentially private SQL engine really has to have the same property.
1104590	1111070	So what we do is we actually split SQL statements into per-user transforms and cross-user aggregations.
1111070	1116750	The per-user transforms are very generous, you can do essentially arbitrary, even Turing-complete SQL.
1116750	1125530	The cross-user transforms are a restricted set of aggregation functions that are much more protected from a privacy standpoint.
1132100	1145240	Finally, you really want to ensure correctness when you're talking about any type of privacy-sensitive infrastructure.
1145540	1151400	And unfortunately, proving that something you've built is differentially private is really hard.
1151400	1154100	It's so hard, it's actually provably undecidable.
1154820	1162840	That said, as part of the foundational libraries, we employ stochastic model checking as part of regression testing.
1162840	1167460	Now obviously this isn't a sufficient guarantee for differential privacy by itself,
1167460	1177900	but it really quickly catches violations, both obvious and subtle, and provides a stable backdrop to more traditional privacy verification work and expert analysis.
1178160	1184080	So this is just an example of a differentially private query, we'll get into this a little bit more later.
1184100	1191280	It's meant to be user-centric, it highlights the fact that you're doing an anonymous query right up front.
1191300	1202320	And in general, it basically looks like a normal SQL query with just a few more keywords and syntactic sugar.
1202620	1213000	You see that this query includes things like count and average, and is selecting from some table in exactly the same way that you would perform normal SQL.
1215280	1220020	So that was a really cursory overview of this particular implementation.
1220200	1226400	You can check out the details of our SQL implementation in full in the paper that we published at PETS this year.
1228280	1232600	And we'll be getting into some of the more detailed math later in the talk.
1232880	1237620	So we're going to spend some time to return to the math behind differential privacy a little bit.
1237620	1243520	We talked kind of about how our SQL engine was structured and a little bit about differential privacy.
1243560	1252720	But going into a bit more of the details, you know, usually when a DP is introduced, it looks something like this.
1253320	1259100	Of course, what we're actually saying when we say some sort of probabilistic statement about any differentially private system
1259440	1264980	is that adding or removing a user from a data set doesn't change the results that much.
1264980	1269220	That much actually has a really rigorous technical definition.
1270040	1272800	There's two parameters, there's epsilon and delta.
1272880	1278380	Epsilon, we generally think about in a sort of a Bayesian context,
1278380	1286020	which is to say that supposing I have some notion of a detail of a user that you're trying to target, right?
1286020	1289900	Supposing you're about 60% sure that they live in Wisconsin,
1289900	1296380	and the attacker is attempting to improve that confidence from their perspective, ideally to 100%.
1296880	1304220	Well, it turns out that there's a really nice translation between how much an attacker can improve their confidence
1304220	1310080	in knowing something and the epsilon that you choose.
1310080	1318200	So for a particular choice of epsilon, perhaps they might be able to improve only 5% their confidence if they're starting at 60%.
1318200	1325920	If you really, really want to be rigorous, perhaps they're not able to really improve it, you know, in any sort of measurable way at all.
1325920	1330740	And so you have this nice principled trade-off between risk and accuracy,
1330740	1339160	since the epsilon has the strongest influence on sort of the quality, the accuracy, the utility of the output results.
1340480	1344280	Delta is a much easier concept to understand and explain.
1344280	1351580	It's basically our out to say that we basically made a differentially private engine,
1351580	1356020	but occasionally, very occasionally, it's not differentially private.
1356020	1358940	So delta is the probability of catastrophic failure.
1359160	1365320	There are lots of ways to think about it, but the way we generally talk about it is perhaps the number of output rows
1365320	1368740	that are not strictly differentially private.
1368740	1372740	Now, of course, you don't know which rows those are, so there's always sort of a reasonable doubt.
1372740	1381540	But this allows you to really quantify a baseline of failure modes that your output data set can have.
1381820	1388880	Delta tends to be very, very small, typically on the order of 1 over n, where n is the number of users, or rows, rather.
1390880	1398340	And because the nature of the noise that we use is a decaying exponential,
1398340	1402340	the delta that is sort of induced by some of the math,
1403420	1411120	actually can be sort of smaller than is machine representable in many cases, as we'll get into in a few minutes.
1411220	1414020	So oftentimes, you don't actually have to worry about a delta,
1414020	1417820	but it's sort of important enough that you should be able to reason about it.
1418420	1422780	So just returning earlier, we talked about the privacy model,
1422780	1427580	per-user transformation that's very generous, more restricted cross-user aggregation.
1428340	1429540	And that's why we apply to SQL.
1429960	1437720	Let's take kind of an example of a typical privacy-sensitive query that seems pretty innocuous.
1437720	1443580	We're counting up the number of visits to a website based on browser agent.
1443800	1449620	So a quick look through the logs reveals that a single user used a particular agent.
1449900	1455120	And already, we've kind of re-identified an individual without doing anything more than counting statistics.
1455120	1460000	So we can see the problem with kind of a non-differentially private environment already.
1460440	1466780	So classically, sort of naively, one thing that one does in differential privacy
1466780	1472000	is they add some kind of noise that satisfies differential privacy to counts.
1472000	1475020	And so in this case, we go through all the browser agents,
1475020	1480820	we adjust the counts, plus or minus, a small amount of noise dependent on epsilon.
1481540	1483880	And in general, you might think, oh, we're good.
1483880	1489500	Except that a browser agent can be an arbitrary string, right?
1489500	1492000	So this could be a made-up thing.
1492000	1495120	Essentially, from the position of an attacker,
1495120	1501560	you have to assume that browser agent is an unbounded space of potentially arbitrarily linked strings.
1501560	1506980	Which means that if you see a value there in your logs,
1507580	1511200	then at least one user must have registered that.
1511200	1513060	There are lots of ways of dealing with this.
1513060	1515260	You could generate a bunch of random strings,
1515260	1520240	and sort of give essentially zero noise to each of those.
1520420	1525600	But you, for an unbounded space, would have to generate unboundedly many browser agents.
1525600	1528920	So the way that we do it is we basically just filter
1528920	1535320	and say that at least tau users need to contribute to any particular partition.
1535500	1539160	It turns out that there's a direct line between tau and delta.
1539160	1543600	So it's really easy to reason about what sort of the correct tau or the correct delta
1544160	1546860	needs to be in terms of the existing parameters.
1547940	1552340	As I said earlier, the nature of the noise is a decaying exponential.
1552340	1555180	So even for 50 users or 100 users,
1555180	1559340	you might be at a vanishingly small, perhaps unmeasurably small,
1559340	1564760	probability that a particular partition is surfaced incorrectly.
1565780	1567780	So you think, we're done, right?
1567780	1571360	Well, almost. There's a little bit more nuance to this.
1571360	1576580	And this is kind of one of the big strengths of this type of a privacy model in SQL
1576580	1578180	is that you can enforce it.
1578900	1583800	So an individual user in any particular session might be using a single browser.
1583800	1586600	But across all their sessions, they may have used many browsers,
1586600	1588300	unboundedly many, in fact.
1588360	1590480	And from the perspective of differential privacy,
1590480	1593760	we're trying to protect a user, not a browser.
1593760	1601260	So if a user has used 10 different browsers and everybody else has just used one,
1601260	1607040	potentially you violated kind of the epsilon guarantees of your system,
1607040	1608280	if you allow that.
1608280	1614620	So the way that we deal with that is with reservoir sampling of a user's contributions
1614620	1618160	and then limiting those contributions.
1618160	1621720	So a user might be able to contribute to 10 separate partitions
1621720	1624380	or three or one or whatever seems appropriate.
1624380	1629360	And again, it's another parameter that can be chosen by the user.
1629480	1634520	But you can also sort of choose one based on an appropriate epsilon and delta.
1635020	1640160	So again, it's just about giving an engine the flexibility to represent
1640160	1643580	sort of both utility and privacy needs at the same time.
1644040	1646780	And this is basically the model in a nutshell.
1646780	1650960	Now, of course, if you were requiring users to write exactly the syntax every time,
1651720	1652120	that's not possible.
1652120	1658240	And so the new syntax is introduced with anonymization is basically just a clever rewriter
1658240	1662700	that is doing all of that in the background to the AST of the SQL
1663280	1668540	and allowing people to write basically SQL that they're familiar with.
1668540	1672580	You see a few more keywords in here, obviously, the privacy parameters,
1672580	1678300	the anonymization keyword in the aggregation calls.
1679540	1684500	And of course, sort of importantly, it's really critical to differential privacy
1684500	1687960	to be able to bound what a single user is able to contribute.
1688420	1694720	So, you know, from the perspective of a particular partition,
1694720	1701240	perhaps, you know, you only want a user to be able to contribute a single visit to a website.
1701500	1708180	But perhaps for a salary, you'd want to allow them to contribute up to,
1708180	1711020	you know, $1 million, $1 million, whatever seems reasonable there.
1711280	1714920	Perhaps if you were contributing an average age or something,
1714920	1717700	you'd want to bound people between 0 and 120.
1718320	1722740	The more that you can definitely say about aggregation bounds,
1722740	1724560	the more accuracy you can do.
1724560	1727620	But of course, a lot of times when you're dealing with arbitrary data,
1727620	1733080	machine learning, that sort of thing, you have no idea what bounds that are in the data.
1733080	1735900	In fact, that's maybe the reason you're doing the analysis in the first place
1735900	1737660	is to learn a bit more about the data.
1738480	1744430	And so one of the other contributions of this work is automatic bounds determination.
1745840	1750800	You want to be able to sort of get the best possible bounding on your data
1750800	1753580	without paying for it very much.
1753580	1757120	Once again, finding out global bounds that are differentially private
1757120	1760660	on arbitrary functions is undecidable.
1761220	1763980	So, you know, you can just sort of give up out of the gate
1763980	1766380	and just sort of go to the next best thing,
1766380	1768200	which is a good approximation.
1768240	1773440	And so what we use when bounds are not provided is logarithmic histograms.
1773440	1776940	Basically, you have partitions of increasing size.
1777260	1782620	Doubling, for instance, allows you to get a reasonably good upper bound
1782620	1784920	that might be off by a factor of two,
1784920	1787840	which in terms of differential privacy actually doesn't cost you very much
1787840	1792300	in terms of utility, but still gives you a really solid understanding
1792300	1796760	of, you know, sort of the worst case upper bound.
1796760	1801380	And what's really amazing about this is that, you know,
1801380	1805240	even with our epsilon having to be split between the exploration bounds
1805240	1809060	of the histogram and the actual computation,
1809060	1812900	the accuracy is often still better than manually determined bounds.
1813720	1818460	Again, just because you're oftentimes talking about worst case scenarios
1818460	1821340	when you're determining bounds in a computation.
1822300	1826180	And this is finding them sort of data dependent bounds.
1826520	1830160	And typically, you know, you might require maybe 100 elements
1830160	1834580	or fewer, depending on the nature of the calculation.
1836000	1839860	Now, let me go back to our open sourcing availability
1839860	1843280	and some action items for you.
1845020	1849120	First, we would like to take part of our differentially private training
1849120	1851360	that we're piloting with this audience.
1851360	1853840	As a first step, please take our survey.
1853840	1855760	And it only takes five minutes.
1855760	1858780	It tries to figure out how much have you understood
1858780	1862620	about differential privacy before you jump into the next topics.
1863120	1865460	Then you can check another introductory video
1865460	1868760	on differential privacy in that link.
1870700	1873860	Third, you can try out some of our code labs.
1873860	1879200	The link that's pointing out there points to a Go pipeline
1879200	1883140	that you can build to produce differentially private metrics
1883140	1887140	for a restaurant and its busy times.
1887640	1890380	Fourth, if you're curious about differential privacy
1890380	1894020	and learning more, you can check out the blog that we're linking.
1895340	1900900	Finally, provide us some feedback on the overall training course
1900900	1903160	and the materials that we've created for you.
1903160	1907500	We're really looking forward to hearing more from your end.
1907500	1911920	And with that, we'll open it up for questions.
1918990	1922670	All right. Thank you for this terrific talk.
1922670	1925030	This was differential privacy, more important than ever
1925030	1927290	in the world of COVID-19.
1927670	1931230	We have Aditi Joshi, Miguel Guevara, and Royce Wilson here
1931230	1935170	to chat with us and answer some questions.
1935170	1938390	First, I think, Aditi, you wanted to introduce the team.
1972290	1975750	Hold on one moment. I think we're having an audio problem.
1978670	1981030	We'll wait for a cue.
1981070	1984030	In the meantime, thanks, everybody, for your patience.
1984030	1986210	This is our glitched episode.
2001640	2004260	And just as a reminder, you can submit any questions you have
2004260	2007280	for the team into the Q&A channel on the Discord.
2007480	2012180	You can find the link there at the link from the Twitch stream.
2012180	2015160	You can also go to our website. It contains links to Twitch.
2015800	2017460	Excuse me, to our Discord.
2017460	2045490	I have her try speaking.
2085730	2089790	So that it was as useful and practical to the audience as possible.
2089870	2092210	And with that, I'm going to turn it over to Miguel and Royce
2092210	2093510	to introduce themselves.
2093990	2097430	All right. One moment. Let me check with our team behind the scenes.
2097430	2099470	Do we have audio right now for our speakers?
2102860	2105100	Okay, we do. Great. Fantastic. Wonderful.
2105100	2107560	Thanks for that intro, Aditi. Miguel, take it away.
2108180	2111780	Great. Thank you, everyone, for coming today to our talk.
2111780	2116160	I'm Miguel. I'm a product manager working on Google's privacy team.
2116300	2119500	Basically, I work with Royce and other engineers on the team
2119500	2123240	on developing our internal differential private infrastructure
2123540	2126040	that we recently open sourced.
2129540	2133000	And then I am Royce. I'm a sophomore engineer at Google.
2133000	2135640	I work on privacy infrastructure.
2135960	2140360	I was also the lead author behind the paper that the talk is based on.
2145110	2148170	Terrific. So we have a couple of questions.
2148770	2150970	I'll start off with the question that I have,
2150970	2153770	just taking the liberty and privilege to do so.
2154190	2156790	Have you encountered any misunderstandings in your work
2156790	2158810	about how Google uses differential privacy?
2158810	2160270	And what have you found to be most effective
2160270	2162910	in combating any of those misunderstandings?
2164330	2165730	That's an interesting question.
2165910	2169810	I wouldn't say it's misunderstandings on how Google uses differential privacy.
2169810	2171550	I would say it's just your own misunderstanding
2171550	2174730	about what differential privacy is.
2174730	2178630	A lot of the times when we launch new features
2178630	2181110	and when we open source things, people approach to us
2181110	2183430	trying to use it. They're very excited
2183430	2186410	because they sort of think that differential privacy
2186410	2190470	is this magic wand that when it touches any sort of data,
2190470	2193430	it suddenly transforms it into private data.
2193610	2195870	And I think that the biggest misunderstanding is
2195870	2198990	people confuse differential privacy a lot with synthetic data.
2198990	2201610	So they believe that differential privacy can automatically
2201610	2203790	turn things into synthetic data.
2204110	2206790	And I think that's the biggest misunderstanding.
2206790	2209630	And one thing that's, I think, important to highlight
2209630	2212650	is that our open source offering helps people
2212650	2215990	produce statistics in a differential private manner.
2216010	2219750	So if any of you has one of those use cases,
2219750	2222450	then I think that the open sourcing offering
2222450	2224230	is a good place to start.
2224230	2226190	And that's the way that we use it at Google
2226190	2229310	to produce a lot of these statistical information
2229310	2231130	in a private manner.
2233190	2237450	Great. Any input from Aditi or Royce on that particular issue?
2237450	2239410	No, I think Miguel captured it.
2239410	2243530	Great. And so this is kind of a related question, I guess.
2243530	2246010	Can you comment on what Google services,
2246010	2248230	besides location visit and busy time reporting,
2248230	2250810	are implementing differential privacy today?
2250810	2252630	Can you give us a sense?
2253570	2258110	Yeah. So the mobility reports also use differential privacy
2258110	2259810	for all of their reporting.
2260150	2263810	We have a lot of features on the backend
2263810	2265650	that use differential privacy.
2265650	2270050	So for instance, Maps has this way of trying to cache tiles
2270470	2274630	that you'll probably use when you have no data.
2274630	2278430	And the way that Maps does the ranking on the backend
2278430	2280910	is by using differential privacy.
2280910	2283850	Popular Dishes on Maps also uses differential privacy
2283850	2286230	to do the ranking of the dishes.
2286530	2289570	And those are some of the user-facing features that we have.
2289570	2294770	However, we use differential privacy a lot for internal analysis.
2294770	2298390	And that internal analysis informs new features.
2301190	2301950	Great.
2301950	2303910	I'm going to use the link to the COVID mobility reports
2303910	2307410	in the Discord channel so people can reference it.
2309010	2312390	All the short links from your slide deck into Discord as well.
2312390	2313110	Fantastic.
2315110	2317150	And then just more broadly,
2317150	2318750	and this is sort of a big long-term question,
2318750	2321010	what do you think is next for differential privacy?
2321010	2323790	That could be sort of in terms of the academic work
2323790	2327510	or in terms of the applied work that you see your team going with
2327510	2328910	in the next several years?
2329470	2331110	Royce, do you want to take that question?
2331610	2335150	Yeah.
2337270	2339910	Miguel might be more suited for this one, I think.
2342230	2343650	Do you want to...?
2343650	2344430	Okay.
2344570	2349490	I think that the biggest academic frontier
2349490	2355990	is improving the performance of differential privacy.
2355990	2357730	At the moment, for certain use cases,
2357730	2360630	you get very big hits in terms of utility.
2360630	2363770	And therefore, it's not good for certain use cases.
2363770	2367010	So I really hope that within the academic community,
2367010	2370370	they can come up with new ways of doing differential privacy
2370510	2373390	with less utility trade-offs.
2373390	2376190	I think that's the other thing that I was thinking about
2376190	2380570	is just make it generally more usable.
2380570	2384130	At the moment, it requires a significant level of expertise
2384130	2386850	to understand it and to implement it.
2386850	2389410	And I hope that in 10 years or less,
2389410	2391490	we get to the point where crypto is nowadays,
2391490	2394530	where there are standard libraries
2394530	2398150	and you can do crypto without being an expert.
2398150	2401510	And you can rely on some other experts to do the right work
2401510	2403830	so that you can be safe.
2403830	2406050	So I hope that differential privacy reaches that point
2406050	2408290	where it becomes so commonplace
2408290	2411430	and a new default way of doing data analysis
2411430	2415470	that is backed up by a lot of expert code
2415470	2417670	that has been reviewed and vetted.
2418330	2419850	And just like you don't roll your own crypto,
2419850	2422070	you don't do the same thing with differential privacy.
2422070	2424690	So the goal of our training is to really make it
2424690	2427370	as accessible to everyone as possible.
2427370	2429350	And so when I post the short links,
2429350	2430450	please go through the training.
2430450	2432250	If you feel like there's something missing,
2432250	2434870	something's not cleared, give us that feedback.
2434910	2436910	Our goal is to keep iterating on this training
2436910	2440730	and to make it as easy to implement as possible
2440730	2442790	for everyone across the board.
2444530	2445770	Great. All right.
2445770	2448150	Well, I think that's all the questions that we've got right now
2448150	2449670	for the time being.
2449670	2451430	And I will unleash you on the Discord
2451430	2453990	to answer any further questions anybody may have.
2453990	2456590	Thank you all so much for providing this valuable resource
2457370	2458790	and for having this terrific talk.
2459290	2461130	Have a terrific rest of your DEF CON.
2461210	2463210	And our next talk will start at 1 p.m.
2463450	2463970	Thank you so much.