Hello, everybody. Good morning. My name is Julia Reinhardt. I'm a fellow in residence
with the Mozilla Foundation and a privacy and data protection professional based in San Francisco.
Previously, for almost 15 years, I used to work as a diplomat for my native Germany.
And I'm really happy to present at DEF CON this year, even if it's safe mode.
I'm here today to speak about regulatory trends for artificial intelligence,
particularly those coming from Europe. I know this is not a topic otherwise featured at the
Crypto & Privacy Village. And I hope you will bear with me when we venture out into a topic
that is not top of mind for most of you. But I expect it to be at least as important as GDPR,
the European data protection regime that took effect two years ago,
and that hit many in the technology space in the US unprepared.
As with any new technology, the use of AI brings both opportunities and risks. Citizens fear being
left powerless in defending their rights and safety when facing informational asymmetries
of algorithmic decision making. And companies are concerned by legal uncertainty.
I'm usually not one to quote CEO of Tesla and SpaceX Elon Musk, but no doubt he's among those
who said it the pointiest way. Mark my words, he said, AI is far more dangerous than nukes.
Why do we have no regulatory oversight? So of course, he's not the only one to say that.
Other corporate leaders have called for increased regulation and so have politicians and respected
scholars and academic research institutes and think tanks. Reasons for this call to action
is the understanding that for all its many benefits, AI also presents many risks and they
include biased algorithms, privacy violations, and the potential for injuries caused by defective
software. So with the increasing use of AI based solutions in areas like criminal justice,
healthcare, robotics, financial services, and education, corporate interests will inevitably
conflict with societal benefits. So that conflict raises the question of what systems should be put
in place to mitigate potential harms. My research is focused on upcoming regulation in Europe
for applications of artificial intelligence and how they could affect US companies. Not the big
tech giants that we all think of that all have their offices in Brussels observing and influencing
these discussions, but the small ones and that don't operate in Europe or globally yet. If you
have thoughts about this as an individual, but also if you work for a company that fits into what
I'm looking at, I'd love if you reached out and gave me feedback. So on my last slide, I share my
website and my Twitter handle again, and I'd be really grateful to receive your messages.
So you could ask, why would a US organization be affected by European regulation? They only have to
follow US law. But actually, what we have seen with the European Union's General Data Protection
Regulation, GDPR, is that lots of businesses based solely in the US are affected, mainly because they
have European users. So they process personal data of European residents. And these residents
of Europe are protected by GDPR wherever the processing of their data takes place. So I could
go on forever on GDPR. It's a topic that I know well and that I've worked on for years, including
when it was still being negotiated among EU member states, and I worked for one of them. But
an important experience I wanted to share here is that many US-based organizations
that process personal data of people around the world have decided to apply GDPR and extend all
rights that go with it to their customers who are not European residents but live outside of Europe.
So I've made a little slide on the advantage of GDPR compliance on a global level.
To these companies, it gives them an edge in global compliance. It's easier for them
in terms of handling complaints and requests. So they say, just give all of our customers
all the rights that Europeans have, a very high bar. And it's still easier for the organization
than to sort out the customer's location and attribute different rights according to their
location. GDPR offers them a legal framework and a set of standards that is at least compared to
other less spelled out legislation or the lack thereof, relatively clearly adoptable.
So while my clients are mainly small and medium enterprises based in the US,
with only some clients in Europe, or with the mere intention of soon expanding to Europe,
this privacy management strategy has been found with bigger tech firms as well. So the image on
the slide that you see shows the view from Salesforce's legal team space in San Francisco.
And they are certainly part of these companies I'm talking about. Organizations appreciate that
there is a standard now that is law in one part of the world that can serve as a guideline
also for other parts of the world. And even if this guideline is more demanding than legislation
in their markets outside of Europe, it makes life so much easier for them to have one high profile
standard than have many different ones, what I call the global privacy patchwork, or none at all.
Now, this was definitely a huge finding in the past two years since May 2018, when GDPR became
enforceable. Also, with the US administration being relatively silent about data protection,
and other countries like Japan or Israel following a rather GDPR-like model, that made the EU the
de facto rule setter in the technology policy worldwide, in the important field that is data
protection, as data is integral to all technology. So what I want to find out now is whether we
could expect to see the same trend with upcoming EU regulation in other fields of tech policy,
for example, artificial intelligence. So if we look at the timeline of policymaking on AI in
Europe, and that's the top line, the dark blue one, we're pretty much in the middle of concepts
gearing up to become concrete policy proposals. As you see on that timeline, after sketching first
ideas and establishing a high level expert group that has issued its proposals, the European
Commission issued a white paper early this year and started a consultation process in which more
than 1,200 individuals, advocacy groups, researchers and companies participated, and I
helped to formulate the Mozilla contribution. Most recently, in July, the experts last July,
the experts invited by the European Commission have issued an assessment list for trustworthy AI
that companies and developers can use to assess their algorithm use. On the international level,
that's the bottom part of the timeline, many of the concepts that form part of the European draft
have been formulated as principles within the OECD, the Organization for Economic Development
and Cooperation and Development, that most European countries and the US are members of,
adopted in May 2019. They promote uses of AI that are innovative and trustworthy,
and that respect human rights and democratic values. A month later, the G20, the world's
biggest economies, have drawn from them heavily when they adopted their human-centered AI principles.
Of course, these are all non-binding, but still highly influential. In the US, the White House
announced the American AI Initiative, which focuses on driving technological innovation and
standards to protect a competitive edge on AI. So the White House last January released 10 principles
for federal agencies to follow when proposing rules governing the private sector's deployment
of AI technology. These are binding, and they follow many of the same ideas, but they are very
light touch, with lots of exceptions, and a strong emphasis on leaving space for corporations
in whatever way to innovate and compete globally. It's a more hands-off free market view, and a
pretty stark renunciation of the careful approach that was initiated under the Obama administration
in 2016, but that hadn't yet led to any actual US comprehensive legislation.
So, back to Europe. Right now, Brussels is working on a draft policy that is expected
to be tabled to member states for negotiation around the beginning of 2021, and that could,
once those negotiations have been successful, become the world's first general regulation on AI.
So what will European AI regulation look like, most likely? It's a three-pronged approach,
with a substantial increase in public and private investments in AI to boost its uptake,
but also funding for research, more cooperation between researchers across Europe,
and better know-how to prepare for socio-economic changes. This is called the ecosystem of
excellence on top. And I would say that the numbers floating here don't add up in
any way to what is seen in the US or China, but it's a start. Combined with a European strategy
for data, which aims to improve the use of data by creating an EU single market for data, because,
of course, as you know, it's 27 different member states, so we're aiming for a single market in
that field to facilitate access to data and computing infrastructures, which is, of course,
an essential requirement for the development and use of AI applications. And last, and I've put
that in the middle, with the aim of an ecosystem of trust. The Commission uses a human rights first
approach and details a number of requirements that AI applications would have to fulfill
in order to be considered trustworthy. This is the new framework that I will present further on.
And, of course, not forget existing EU laws and regulations, whether on EU level or in the member
states, that already apply to AI solutions, including rules on data protection, GDPR,
non-discrimination, consumer protection, product safety, and liability. Consumers, of course,
expect the same level of safety and respect of their rights, whether or not a product or a system
relies on AI. So, in some cases, the AI-related aspects of these existing rules may be difficult
to enforce because AI systems are opaque, unpredictable, complex, and autonomous.
So, in some cases, they might need to be amended or updated. So, I'll come to the requirements on
my next slide, but I need to first underline that this approach singles out high-risk
applications, which would become subject to strict requirements and not those that are considered
low-risk, for which no additional legal requirements would be imposed. And for these
low-risk ones, the Commission is considering a voluntary labeling system that would certify
compliance with parts of the requirements and allow companies to market their AI products
as trustworthy. So, rather than introducing a generic AI application, this is a regulation,
sorry, this is a nuanced risk-based approach, possibly one that is application and technology
specific. However, this approach may also lead to uncertainty, and I personally worry that although
the text includes a definition of what is meant by high-risk sectors, specifically mentioned are
healthcare, transport, energy, judicial decision-making, and mass citizen surveillance,
it still won't always be easy to determine. It is very subjective, and an algorithm use
could be considered low-risk for someone, but actually has high-risk consequences for others.
Plus, risk here is defined as a risk to an individual, which excludes a whole
category of AI applications that pose major collective risks.
So, developers of high-risk AI applications would be obliged to follow rules in these key aspects.
First, human oversight. Some degrees of human oversight would be required, ranging from
requiring human review before a decision is implemented to the possibility of human
intervention in real-time or afterwards. Then there could be requirements on robustness
and accuracy, the ability to react to inconsistencies, and that the application
is resilient against attacks and manipulation. Training data sets need to be representative and
comprehensive and comply with privacy and data protection rules.
Information to be provided. People would need to be informed when they interact with an AI system
and not a human, and what its capabilities and limitations are.
Developers would also need to keep records of how they selected what kind of training and
testing data, and how they programmed and trained the system to enable AI decisions to be traced
back. And then there will probably be specific requirements for remote biometric
identification, also known as facial recognition. So, no ban inevitably, but the EU insists that
data protection rules and the Charter of Fundamental Rights apply, which already allow
the uses of facial recognition only in cases where this action is justified and proportionate
and is subject to adequate safeguards. So, there will probably be an open discussion on whether
exceptions could be justified. So, as a developer, and I know that many of you are developers,
think of this like some kind of FDA-style clinical testing, not for drugs, but for algorithms.
So, how do I expect US companies to be affected by future regulation on AI?
The EU guidelines will likely have a ripple effect in the US,
since many American technology companies provide AI solutions and services to the EU.
It will also impact US companies that EU investors are looking to buy,
and companies that plan to expand into European markets. Of course, the additional policy and
regulatory measures the EU considers will increase the cost of compliance and put the
administrative burden and possible IP-related difficulties on companies that develop or
deploy AI systems. And, of course, on a less business and more political level,
citizens will see what rights are granted in the EU, and at least some of them could demand
those rights as well. So, in the US, it's private businesses that are currently pushing harder for
AI regulation than government agencies. Some of the big players, I mentioned them here, but it's
also Salesforce that I mentioned earlier, have established internal principles on which they
base their AI development in the future. This is not ideal to my mind, but it's a first step,
because internally established principles of AI ethics are hardly enforceable,
competitors are not bound by them. And also, there are so many guidelines around from a consumer
point of view that I take, it's hard to really know where a company stands in terms of compliance,
because everybody has their own rules. But at least it obliges the company that decides to go
this way to face scrutiny about its products, but it isn't designed to really calm citizens' concerns
on a broad base, because there are still players around who don't feel bound by any of those
principles. And then there's self-regulation of industry. And by that, I mean that a whole
industry designs and enforces new rules and standards for themselves when government rules
are lacking. Usually, these would be codes of conduct, or binding rules that industry establishes
for itself, for example, within federations. The hope would be that industry collective action
can change the incentives in a situation where the competitive dynamic of AI development leads
to a prisoner dilemma for both companies and state, wherein both are incentivized to
prioritize the fast development of AI instead of the safe development of AI.
But unfortunately, this hasn't been successful, or at least not impactful for now.
And that's why some companies actively push government for more committed steps in the US.
I've often heard people in companies say that they look to Europe for these obligations to come
up. They are mostly the ones who've seen the success of GDPR and are convinced the US needed
similar groundbreaking law regulating technological development, but they don't see that the current
US administration is in any way interested in pushing that. The dominant view in DC for the
moment is that the US needs to lead in AI development to compete with China, no matter
what. But of course, we have elections in November. So please vote. At last, and last, there's the way
through local and state laws. We've seen that with data protection, that when the federal level in
the US doesn't move enough, the states engage more. There are some examples that state and
local level have moved that are pretty promising. They don't have the reach to create general
nationwide regulation, of course, but they're creating smaller scale uses, use cases for
regulation, like regulatory sandboxes, but in real life, that could be useful once the federal level
is ready to engage. So for instance, the California Consumer Privacy Act, CCPA, that took effect last
year, showed that a US state can issue impactful legislation that addresses concerns about data
protection and that significant fines can be levied against misuse of personal data.
So the CCPA is far from perfect. And it started to be enforced only now, but I don't really see
how it will be effectively enforced, given so many imperfections. But still, it was quite a step.
And many US states have meanwhile issued their own privacy legislation. However,
none of them is as far reaching as the California law.
A more recent example, so more specifically from the AI field, again, from California,
is the BOT, the Bolstering Online Transparency Act, which makes it unlawful for any person to
use a bot to communicate or interact with another person in California online,
with the intent to mislead the other person about its artificial identity.
This legislation specifically applies to bots that intend to influence voters,
as well as intentionally deceptive bots used to sell goods and services. It doesn't make bots
illegal, but it requires them to identify themselves as non-human. So the law took effect
a month ago, July 1st, only in California, and it doesn't clearly address how it applies to bots
that don't originate in the Golden State, and how enforcement would work then. So one last example
for local action on municipal level, this time regarding facial recognition technology, which
is the most controversial area of AI use today. In this case, when so far no federal guidelines
exist to limit or standardize its use, and few state rules are in place, cities feel left to
decide for themselves what, if anything, to do. So the city and county of San Francisco last year
passed a bill to ban facial recognition technologies for police and city agency use.
So other US cities have followed suit. In this case, it is only one sector, government,
that is required to refrain from the use of this technology, while the private sector and
individual can still use it. But it shows an awareness that with the current state of technology
development, organizations clearly aren't ready to use facial recognition in a safe, secure, and
responsible manner. And that there is a need for clear legislation that, on a federal level, so far
no one has managed to pass. As I mentioned before, what most technology companies are saying is that
they look not to DC, but to Europe when it comes to effective regulation in that field.
So what I've been doing recently is to interview companies in the Bay Area working on AI systems
about the expected impact of European regulation on them, and about their preparedness and their
sense of agreement with these principles. My focus, I mentioned that, is on small and medium-sized
companies that are less in the focus of the media and that face the additional challenges competing
with the tech giants in a field where size clearly matters. Because the more data you can gather,
the better your AI works. So I'm also interested in this type of company because the general
impression with GDPR was that the small players have been losing ground due to compliance costs,
and that regulation blocks them more than the big ones. So far, my interviews show that most,
if not all, contacts have an understanding of the importance of rule-setting around AI-based systems,
and that they report having thought about limiting their own developments
until political guidance is given what is admissible and what is not. All of them had
experienced some kind of adaptation process to GDPR two years ago. Some of them were my clients,
so I helped them with that. And they reported that having spent considerable money and time
on compliance with rules that originated in Europe, but that several of them applied to
all of their products and all users wherever they are located. Now, with AI regulation coming up,
they expect the same to happen, but there is disagreement on whether this kind of general
regulation is even possible in a broad field like AI. So they consider it too broad to fall under
one rule. That includes small applications that are considered to simply make life easier,
as well as large systems of mass surveillance. So importantly, many of them also said that they
expected European legislation to be faster, more comprehensive, and more enforceable than any
US attempt for regulation. Maybe that's a lesson learned from GDPR and doesn't mean that
this would be the same with the upcoming policies in the AI sector, but it's a good indicator of the
urge many players feel to prepare for rules and not to leave the space entirely to the big players
and their lobbying power. So in my mind, although the EU's upcoming rules are geared primarily
to European firms and will become law only potentially sometime in the future,
US companies need to look at them already now. So it won't be enough to just wait and
see, and it's better to prepare for the probability of AI regulation in Europe now.
So again, if you have thoughts about this as an individual, but also if you work for a company
that fits into what I'm looking at, I'd love if you reached out and gave me a feedback.
Here's my website and Twitter handle at the bottom of the page. And if you're interested
in reading more about this field and add your comments, have a look at the white paper that
we're working on with the Mozilla Foundation at the link in the center of the slide.
So thank you for attending. I wish I could have seen you all in person.
I hope you're all safe and stay healthy. And I'm looking forward to your questions.
And welcome back. We have Julia here with us. If anybody has any questions,
drop them in the Q&A channel and Discord. We've got a few here. Julia, I enjoyed your talk. Is
there a document with more in-depth info about the requirements of trustworthy AI in high-risk
applications? Yeah, so I directed the person who asked to the assessment list that has been
published by the High-Level Expert Group this July, a couple weeks ago. And I think that shows
a bit of what they expect, you know, in terms of actual requirements, very concretely. But of
course, I mean, remember that this hasn't been finalized yet. This is just a proposal. Well,
it's actually not even a concrete proposal yet. It will become a proposal probably around the
beginning of 2021. So it's all in the making. Cool. Thank you. And then here's another question.
What would you like to see happen with CCPA in regards to clarifying bots, whether from
California or not, on how bots identify as non-humans prior to contact? What type of
enforcement or not would you like to see? Yeah, so I think there are two aspects here,
bots specifically, and then CCPA. For CCPA, I think my main doubt is that the Attorney General
in California won't or doesn't have the bandwidth, really, the capacity to really enforce all the
cases, because it really depends on him and his team to find and to inquire and then to enforce.
And I don't think this will really work, especially because the text is not that clear.
And so that is my main doubt. I mean, there are lots of others. You know, CCPA is really kind of
not very well designed in terms of wording and terminology. But it's been enforceable only since
1st of July this year. So we'll see how that goes. In terms of bots, I think it's most useful
to look at the bot bill that's been in effect also since July 1st here in California. It's a
I think, I mean, also there, the application, the actual what it applies to is a bit narrow,
but it shows how this could work. You know, you really, when you're contacted by a bot,
you know exactly whether you're talking to you or not. And that would be the same also in the
European proposal. I mean, not exactly the same that we don't know yet, but the idea is the same.
That'd be interesting. All right, we have one last question here. Do you think that AI will
develop that much to actually take us over, take us over in the sense that they might steal our
jobs, our houses, our budget, etc? Well, I mean, there, of course,
the eternal question is, what do you define as AI? And if we take a broad look at it,
your wording of the question is, sounds a bit dramatic, but I mean, in a way,
in some sectors, you could be right, because in the end, it's really important for humans
to control what they're creating and to actually have a last say in what the technology is supposed
to do. And so I think if we, I think the awareness is there to find rules, to make that,
to make sure that this happens. So I do think that in some sectors, there's a potential for
actual harm, yes. And for dramatic harm. I mean, this is in the very high risk sectors, of course.
But even in the less high risk sectors, I think that, you know, democracy, fundamental rights,
are already at stake. Yes.
Wow. Well, thank you for taking the time to join us today. If you have any more questions,
put them in the Discord and Julia will be able to answer them there. Thank you.
Thank you.
Our next talk up will be...