start	end	text
19400	24720	Hello, everybody. Good morning. My name is Julia Reinhardt. I'm a fellow in residence
24720	30560	with the Mozilla Foundation and a privacy and data protection professional based in San Francisco.
30720	36380	Previously, for almost 15 years, I used to work as a diplomat for my native Germany.
36520	41000	And I'm really happy to present at DEF CON this year, even if it's safe mode.
42440	47160	I'm here today to speak about regulatory trends for artificial intelligence,
47160	53280	particularly those coming from Europe. I know this is not a topic otherwise featured at the
53280	58940	Crypto & Privacy Village. And I hope you will bear with me when we venture out into a topic
58940	66820	that is not top of mind for most of you. But I expect it to be at least as important as GDPR,
66820	70280	the European data protection regime that took effect two years ago,
70280	74620	and that hit many in the technology space in the US unprepared.
75480	82720	As with any new technology, the use of AI brings both opportunities and risks. Citizens fear being
82720	89080	left powerless in defending their rights and safety when facing informational asymmetries
89080	94140	of algorithmic decision making. And companies are concerned by legal uncertainty.
95020	102700	I'm usually not one to quote CEO of Tesla and SpaceX Elon Musk, but no doubt he's among those
102700	109920	who said it the pointiest way. Mark my words, he said, AI is far more dangerous than nukes.
109920	116100	Why do we have no regulatory oversight? So of course, he's not the only one to say that.
116100	122060	Other corporate leaders have called for increased regulation and so have politicians and respected
122060	129160	scholars and academic research institutes and think tanks. Reasons for this call to action
129160	135540	is the understanding that for all its many benefits, AI also presents many risks and they
135540	142520	include biased algorithms, privacy violations, and the potential for injuries caused by defective
142520	148780	software. So with the increasing use of AI based solutions in areas like criminal justice,
148780	155220	healthcare, robotics, financial services, and education, corporate interests will inevitably
155220	162740	conflict with societal benefits. So that conflict raises the question of what systems should be put
162740	169860	in place to mitigate potential harms. My research is focused on upcoming regulation in Europe
169860	177300	for applications of artificial intelligence and how they could affect US companies. Not the big
177300	182740	tech giants that we all think of that all have their offices in Brussels observing and influencing
182740	190460	these discussions, but the small ones and that don't operate in Europe or globally yet. If you
190460	195800	have thoughts about this as an individual, but also if you work for a company that fits into what
195800	201840	I'm looking at, I'd love if you reached out and gave me feedback. So on my last slide, I share my
201840	207140	website and my Twitter handle again, and I'd be really grateful to receive your messages.
207850	215080	So you could ask, why would a US organization be affected by European regulation? They only have to
215080	222020	follow US law. But actually, what we have seen with the European Union's General Data Protection
222020	232120	Regulation, GDPR, is that lots of businesses based solely in the US are affected, mainly because they
232120	237740	have European users. So they process personal data of European residents. And these residents
237740	244020	of Europe are protected by GDPR wherever the processing of their data takes place. So I could
244020	250700	go on forever on GDPR. It's a topic that I know well and that I've worked on for years, including
250700	257600	when it was still being negotiated among EU member states, and I worked for one of them. But
257600	264340	an important experience I wanted to share here is that many US-based organizations
264920	272300	that process personal data of people around the world have decided to apply GDPR and extend all
272300	279260	rights that go with it to their customers who are not European residents but live outside of Europe.
281780	287580	So I've made a little slide on the advantage of GDPR compliance on a global level.
288340	294100	To these companies, it gives them an edge in global compliance. It's easier for them
294100	300360	in terms of handling complaints and requests. So they say, just give all of our customers
300360	306780	all the rights that Europeans have, a very high bar. And it's still easier for the organization
306780	311800	than to sort out the customer's location and attribute different rights according to their
311800	320740	location. GDPR offers them a legal framework and a set of standards that is at least compared to
320740	327600	other less spelled out legislation or the lack thereof, relatively clearly adoptable.
328860	335800	So while my clients are mainly small and medium enterprises based in the US,
335800	342500	with only some clients in Europe, or with the mere intention of soon expanding to Europe,
342500	349440	this privacy management strategy has been found with bigger tech firms as well. So the image on
349440	354880	the slide that you see shows the view from Salesforce's legal team space in San Francisco.
355460	361460	And they are certainly part of these companies I'm talking about. Organizations appreciate that
361460	368840	there is a standard now that is law in one part of the world that can serve as a guideline
368840	374460	also for other parts of the world. And even if this guideline is more demanding than legislation
374460	380540	in their markets outside of Europe, it makes life so much easier for them to have one high profile
380540	387820	standard than have many different ones, what I call the global privacy patchwork, or none at all.
387820	396560	Now, this was definitely a huge finding in the past two years since May 2018, when GDPR became
396560	404120	enforceable. Also, with the US administration being relatively silent about data protection,
404120	411740	and other countries like Japan or Israel following a rather GDPR-like model, that made the EU the
411800	418220	de facto rule setter in the technology policy worldwide, in the important field that is data
418220	426040	protection, as data is integral to all technology. So what I want to find out now is whether we
426040	431840	could expect to see the same trend with upcoming EU regulation in other fields of tech policy,
431840	440020	for example, artificial intelligence. So if we look at the timeline of policymaking on AI in
440020	446480	Europe, and that's the top line, the dark blue one, we're pretty much in the middle of concepts
446480	453840	gearing up to become concrete policy proposals. As you see on that timeline, after sketching first
453840	460140	ideas and establishing a high level expert group that has issued its proposals, the European
460140	465680	Commission issued a white paper early this year and started a consultation process in which more
465680	473940	than 1,200 individuals, advocacy groups, researchers and companies participated, and I
473940	485060	helped to formulate the Mozilla contribution. Most recently, in July, the experts last July,
485060	491780	the experts invited by the European Commission have issued an assessment list for trustworthy AI
491780	499600	that companies and developers can use to assess their algorithm use. On the international level,
499600	505200	that's the bottom part of the timeline, many of the concepts that form part of the European draft
505200	511180	have been formulated as principles within the OECD, the Organization for Economic Development
511180	517460	and Cooperation and Development, that most European countries and the US are members of,
517940	524880	adopted in May 2019. They promote uses of AI that are innovative and trustworthy,
524880	531460	and that respect human rights and democratic values. A month later, the G20, the world's
531460	538840	biggest economies, have drawn from them heavily when they adopted their human-centered AI principles.
538840	546800	Of course, these are all non-binding, but still highly influential. In the US, the White House
546800	552760	announced the American AI Initiative, which focuses on driving technological innovation and
552760	560000	standards to protect a competitive edge on AI. So the White House last January released 10 principles
560000	565540	for federal agencies to follow when proposing rules governing the private sector's deployment
565540	572540	of AI technology. These are binding, and they follow many of the same ideas, but they are very
572540	579040	light touch, with lots of exceptions, and a strong emphasis on leaving space for corporations
579040	588760	in whatever way to innovate and compete globally. It's a more hands-off free market view, and a
588760	593660	pretty stark renunciation of the careful approach that was initiated under the Obama administration
593660	599900	in 2016, but that hadn't yet led to any actual US comprehensive legislation.
600940	607020	So, back to Europe. Right now, Brussels is working on a draft policy that is expected
607020	614060	to be tabled to member states for negotiation around the beginning of 2021, and that could,
614060	621100	once those negotiations have been successful, become the world's first general regulation on AI.
623660	631480	So what will European AI regulation look like, most likely? It's a three-pronged approach,
631480	637880	with a substantial increase in public and private investments in AI to boost its uptake,
637880	643480	but also funding for research, more cooperation between researchers across Europe,
643480	649100	and better know-how to prepare for socio-economic changes. This is called the ecosystem of
649100	655160	excellence on top. And I would say that the numbers floating here don't add up in
655160	663600	any way to what is seen in the US or China, but it's a start. Combined with a European strategy
663600	671760	for data, which aims to improve the use of data by creating an EU single market for data, because,
671760	678220	of course, as you know, it's 27 different member states, so we're aiming for a single market in
678220	684200	that field to facilitate access to data and computing infrastructures, which is, of course,
684200	690060	an essential requirement for the development and use of AI applications. And last, and I've put
690060	698660	that in the middle, with the aim of an ecosystem of trust. The Commission uses a human rights first
698660	704560	approach and details a number of requirements that AI applications would have to fulfill
704560	711400	in order to be considered trustworthy. This is the new framework that I will present further on.
711640	717080	And, of course, not forget existing EU laws and regulations, whether on EU level or in the member
717080	723920	states, that already apply to AI solutions, including rules on data protection, GDPR,
723920	730640	non-discrimination, consumer protection, product safety, and liability. Consumers, of course,
730640	736820	expect the same level of safety and respect of their rights, whether or not a product or a system
736820	743520	relies on AI. So, in some cases, the AI-related aspects of these existing rules may be difficult
743520	750340	to enforce because AI systems are opaque, unpredictable, complex, and autonomous.
750780	758820	So, in some cases, they might need to be amended or updated. So, I'll come to the requirements on
758820	763820	my next slide, but I need to first underline that this approach singles out high-risk
763820	771240	applications, which would become subject to strict requirements and not those that are considered
771240	777180	low-risk, for which no additional legal requirements would be imposed. And for these
777180	784080	low-risk ones, the Commission is considering a voluntary labeling system that would certify
784080	790400	compliance with parts of the requirements and allow companies to market their AI products
790400	798540	as trustworthy. So, rather than introducing a generic AI application, this is a regulation,
798540	804800	sorry, this is a nuanced risk-based approach, possibly one that is application and technology
804800	813240	specific. However, this approach may also lead to uncertainty, and I personally worry that although
813240	820580	the text includes a definition of what is meant by high-risk sectors, specifically mentioned are
820580	827360	healthcare, transport, energy, judicial decision-making, and mass citizen surveillance,
827880	834280	it still won't always be easy to determine. It is very subjective, and an algorithm use
834280	841760	could be considered low-risk for someone, but actually has high-risk consequences for others.
841760	850040	Plus, risk here is defined as a risk to an individual, which excludes a whole
850040	854540	category of AI applications that pose major collective risks.
857320	865060	So, developers of high-risk AI applications would be obliged to follow rules in these key aspects.
865320	872760	First, human oversight. Some degrees of human oversight would be required, ranging from
872760	879260	requiring human review before a decision is implemented to the possibility of human
879260	885440	intervention in real-time or afterwards. Then there could be requirements on robustness
885440	891540	and accuracy, the ability to react to inconsistencies, and that the application
891540	899720	is resilient against attacks and manipulation. Training data sets need to be representative and
899720	904100	comprehensive and comply with privacy and data protection rules.
905160	911400	Information to be provided. People would need to be informed when they interact with an AI system
911400	916400	and not a human, and what its capabilities and limitations are.
917940	923100	Developers would also need to keep records of how they selected what kind of training and
923100	929380	testing data, and how they programmed and trained the system to enable AI decisions to be traced
929380	936980	back. And then there will probably be specific requirements for remote biometric
936980	946120	identification, also known as facial recognition. So, no ban inevitably, but the EU insists that
946120	951920	data protection rules and the Charter of Fundamental Rights apply, which already allow
951920	957820	the uses of facial recognition only in cases where this action is justified and proportionate
958300	964460	and is subject to adequate safeguards. So, there will probably be an open discussion on whether
964460	971520	exceptions could be justified. So, as a developer, and I know that many of you are developers,
971840	978840	think of this like some kind of FDA-style clinical testing, not for drugs, but for algorithms.
979900	986120	So, how do I expect US companies to be affected by future regulation on AI?
987820	992020	The EU guidelines will likely have a ripple effect in the US,
992020	997700	since many American technology companies provide AI solutions and services to the EU.
997700	1004980	It will also impact US companies that EU investors are looking to buy,
1004980	1012040	and companies that plan to expand into European markets. Of course, the additional policy and
1012040	1016980	regulatory measures the EU considers will increase the cost of compliance and put the
1016980	1024260	administrative burden and possible IP-related difficulties on companies that develop or
1024840	1032540	deploy AI systems. And, of course, on a less business and more political level,
1032540	1038540	citizens will see what rights are granted in the EU, and at least some of them could demand
1038540	1049260	those rights as well. So, in the US, it's private businesses that are currently pushing harder for
1049260	1055200	AI regulation than government agencies. Some of the big players, I mentioned them here, but it's
1055200	1061480	also Salesforce that I mentioned earlier, have established internal principles on which they
1061480	1069680	base their AI development in the future. This is not ideal to my mind, but it's a first step,
1069680	1076100	because internally established principles of AI ethics are hardly enforceable,
1076100	1082120	competitors are not bound by them. And also, there are so many guidelines around from a consumer
1082120	1089200	point of view that I take, it's hard to really know where a company stands in terms of compliance,
1089200	1095860	because everybody has their own rules. But at least it obliges the company that decides to go
1095860	1102440	this way to face scrutiny about its products, but it isn't designed to really calm citizens' concerns
1102440	1107260	on a broad base, because there are still players around who don't feel bound by any of those
1107260	1116220	principles. And then there's self-regulation of industry. And by that, I mean that a whole
1116220	1121540	industry designs and enforces new rules and standards for themselves when government rules
1121540	1129480	are lacking. Usually, these would be codes of conduct, or binding rules that industry establishes
1129480	1136700	for itself, for example, within federations. The hope would be that industry collective action
1136700	1143940	can change the incentives in a situation where the competitive dynamic of AI development leads
1143940	1151820	to a prisoner dilemma for both companies and state, wherein both are incentivized to
1152420	1158500	prioritize the fast development of AI instead of the safe development of AI.
1158500	1163740	But unfortunately, this hasn't been successful, or at least not impactful for now.
1165120	1171420	And that's why some companies actively push government for more committed steps in the US.
1171420	1176360	I've often heard people in companies say that they look to Europe for these obligations to come
1176360	1183740	up. They are mostly the ones who've seen the success of GDPR and are convinced the US needed
1183740	1190260	similar groundbreaking law regulating technological development, but they don't see that the current
1190260	1197080	US administration is in any way interested in pushing that. The dominant view in DC for the
1197080	1204920	moment is that the US needs to lead in AI development to compete with China, no matter
1204920	1214940	what. But of course, we have elections in November. So please vote. At last, and last, there's the way
1214940	1223180	through local and state laws. We've seen that with data protection, that when the federal level in
1223180	1230020	the US doesn't move enough, the states engage more. There are some examples that state and
1230020	1236180	local level have moved that are pretty promising. They don't have the reach to create general
1236180	1243840	nationwide regulation, of course, but they're creating smaller scale uses, use cases for
1243840	1252280	regulation, like regulatory sandboxes, but in real life, that could be useful once the federal level
1252280	1260720	is ready to engage. So for instance, the California Consumer Privacy Act, CCPA, that took effect last
1260720	1269620	year, showed that a US state can issue impactful legislation that addresses concerns about data
1269620	1274600	protection and that significant fines can be levied against misuse of personal data.
1274900	1283300	So the CCPA is far from perfect. And it started to be enforced only now, but I don't really see
1283300	1290120	how it will be effectively enforced, given so many imperfections. But still, it was quite a step.
1290120	1295900	And many US states have meanwhile issued their own privacy legislation. However,
1296600	1302020	none of them is as far reaching as the California law.
1303020	1308640	A more recent example, so more specifically from the AI field, again, from California,
1308640	1316400	is the BOT, the Bolstering Online Transparency Act, which makes it unlawful for any person to
1316400	1321460	use a bot to communicate or interact with another person in California online,
1321460	1325710	with the intent to mislead the other person about its artificial identity.
1326260	1333060	This legislation specifically applies to bots that intend to influence voters,
1333060	1339940	as well as intentionally deceptive bots used to sell goods and services. It doesn't make bots
1339940	1347220	illegal, but it requires them to identify themselves as non-human. So the law took effect
1347260	1353360	a month ago, July 1st, only in California, and it doesn't clearly address how it applies to bots
1353360	1361220	that don't originate in the Golden State, and how enforcement would work then. So one last example
1361220	1367260	for local action on municipal level, this time regarding facial recognition technology, which
1367260	1377120	is the most controversial area of AI use today. In this case, when so far no federal guidelines
1377120	1383820	exist to limit or standardize its use, and few state rules are in place, cities feel left to
1383820	1391960	decide for themselves what, if anything, to do. So the city and county of San Francisco last year
1391960	1398600	passed a bill to ban facial recognition technologies for police and city agency use.
1398740	1405780	So other US cities have followed suit. In this case, it is only one sector, government,
1405780	1412280	that is required to refrain from the use of this technology, while the private sector and
1412280	1419500	individual can still use it. But it shows an awareness that with the current state of technology
1419500	1426820	development, organizations clearly aren't ready to use facial recognition in a safe, secure, and
1426820	1433260	responsible manner. And that there is a need for clear legislation that, on a federal level, so far
1433260	1441920	no one has managed to pass. As I mentioned before, what most technology companies are saying is that
1441920	1447340	they look not to DC, but to Europe when it comes to effective regulation in that field.
1448480	1455180	So what I've been doing recently is to interview companies in the Bay Area working on AI systems
1455180	1462480	about the expected impact of European regulation on them, and about their preparedness and their
1462480	1469900	sense of agreement with these principles. My focus, I mentioned that, is on small and medium-sized
1469900	1476240	companies that are less in the focus of the media and that face the additional challenges competing
1476240	1483100	with the tech giants in a field where size clearly matters. Because the more data you can gather,
1483100	1488740	the better your AI works. So I'm also interested in this type of company because the general
1488740	1495500	impression with GDPR was that the small players have been losing ground due to compliance costs,
1495500	1503860	and that regulation blocks them more than the big ones. So far, my interviews show that most,
1503860	1511520	if not all, contacts have an understanding of the importance of rule-setting around AI-based systems,
1511520	1516660	and that they report having thought about limiting their own developments
1516660	1524080	until political guidance is given what is admissible and what is not. All of them had
1524080	1530380	experienced some kind of adaptation process to GDPR two years ago. Some of them were my clients,
1530380	1538320	so I helped them with that. And they reported that having spent considerable money and time
1538320	1544380	on compliance with rules that originated in Europe, but that several of them applied to
1544380	1551620	all of their products and all users wherever they are located. Now, with AI regulation coming up,
1551620	1558440	they expect the same to happen, but there is disagreement on whether this kind of general
1558440	1565680	regulation is even possible in a broad field like AI. So they consider it too broad to fall under
1565680	1572340	one rule. That includes small applications that are considered to simply make life easier,
1572900	1581560	as well as large systems of mass surveillance. So importantly, many of them also said that they
1581560	1587900	expected European legislation to be faster, more comprehensive, and more enforceable than any
1587900	1595420	US attempt for regulation. Maybe that's a lesson learned from GDPR and doesn't mean that
1595420	1602760	this would be the same with the upcoming policies in the AI sector, but it's a good indicator of the
1602760	1609300	urge many players feel to prepare for rules and not to leave the space entirely to the big players
1609300	1617100	and their lobbying power. So in my mind, although the EU's upcoming rules are geared primarily
1617100	1624260	to European firms and will become law only potentially sometime in the future,
1624260	1629580	US companies need to look at them already now. So it won't be enough to just wait and
1629580	1635220	see, and it's better to prepare for the probability of AI regulation in Europe now.
1637600	1643540	So again, if you have thoughts about this as an individual, but also if you work for a company
1643540	1648280	that fits into what I'm looking at, I'd love if you reached out and gave me a feedback.
1648280	1652920	Here's my website and Twitter handle at the bottom of the page. And if you're interested
1652920	1658120	in reading more about this field and add your comments, have a look at the white paper that
1658120	1663140	we're working on with the Mozilla Foundation at the link in the center of the slide.
1663380	1668440	So thank you for attending. I wish I could have seen you all in person.
1668500	1673880	I hope you're all safe and stay healthy. And I'm looking forward to your questions.
1680800	1688600	And welcome back. We have Julia here with us. If anybody has any questions,
1688600	1696000	drop them in the Q&A channel and Discord. We've got a few here. Julia, I enjoyed your talk. Is
1696000	1701080	there a document with more in-depth info about the requirements of trustworthy AI in high-risk
1701080	1715060	applications? Yeah, so I directed the person who asked to the assessment list that has been
1715060	1721320	published by the High-Level Expert Group this July, a couple weeks ago. And I think that shows
1721580	1728100	a bit of what they expect, you know, in terms of actual requirements, very concretely. But of
1728100	1732940	course, I mean, remember that this hasn't been finalized yet. This is just a proposal. Well,
1732940	1738200	it's actually not even a concrete proposal yet. It will become a proposal probably around the
1738200	1747200	beginning of 2021. So it's all in the making. Cool. Thank you. And then here's another question.
1747460	1752620	What would you like to see happen with CCPA in regards to clarifying bots, whether from
1752620	1758740	California or not, on how bots identify as non-humans prior to contact? What type of
1758740	1766340	enforcement or not would you like to see? Yeah, so I think there are two aspects here,
1766340	1773040	bots specifically, and then CCPA. For CCPA, I think my main doubt is that the Attorney General
1773040	1780920	in California won't or doesn't have the bandwidth, really, the capacity to really enforce all the
1780920	1790200	cases, because it really depends on him and his team to find and to inquire and then to enforce.
1790340	1797860	And I don't think this will really work, especially because the text is not that clear.
1798640	1804640	And so that is my main doubt. I mean, there are lots of others. You know, CCPA is really kind of
1805420	1814420	not very well designed in terms of wording and terminology. But it's been enforceable only since
1814420	1820140	1st of July this year. So we'll see how that goes. In terms of bots, I think it's most useful
1820140	1834620	to look at the bot bill that's been in effect also since July 1st here in California. It's a
1834620	1844440	I think, I mean, also there, the application, the actual what it applies to is a bit narrow,
1844440	1851340	but it shows how this could work. You know, you really, when you're contacted by a bot,
1851340	1856560	you know exactly whether you're talking to you or not. And that would be the same also in the
1856560	1863580	European proposal. I mean, not exactly the same that we don't know yet, but the idea is the same.
1863580	1869360	That'd be interesting. All right, we have one last question here. Do you think that AI will
1869360	1874920	develop that much to actually take us over, take us over in the sense that they might steal our
1874920	1881640	jobs, our houses, our budget, etc? Well, I mean, there, of course,
1881640	1888620	the eternal question is, what do you define as AI? And if we take a broad look at it,
1888620	1896340	your wording of the question is, sounds a bit dramatic, but I mean, in a way,
1896340	1902780	in some sectors, you could be right, because in the end, it's really important for humans
1903580	1912740	to control what they're creating and to actually have a last say in what the technology is supposed
1912740	1919920	to do. And so I think if we, I think the awareness is there to find rules, to make that,
1919920	1929040	to make sure that this happens. So I do think that in some sectors, there's a potential for
1929040	1935500	actual harm, yes. And for dramatic harm. I mean, this is in the very high risk sectors, of course.
1935500	1940640	But even in the less high risk sectors, I think that, you know, democracy, fundamental rights,
1940640	1943580	are already at stake. Yes.
1944500	1952120	Wow. Well, thank you for taking the time to join us today. If you have any more questions,
1952120	1958100	put them in the Discord and Julia will be able to answer them there. Thank you.
1958460	1959000	Thank you.
1959000	1961100	Our next talk up will be...