WEBVTT 00:19.400 --> 00:20.680 Hello, everybody. 00:20.680 --> 00:21.940 Good morning. 00:21.940 --> 00:23.700 My name is Julia Reinhardt. 00:23.700 --> 00:30.560 I'm a fellow in residence with the Mozilla Foundation and a privacy and data protection professional based in San Francisco. 00:30.720 --> 00:36.380 Previously, for almost 15 years, I used to work as a diplomat for my native Germany. 00:36.520 --> 00:41.000 And I'm really happy to present at DEF CON this year, even if it's safe mode. 00:42.440 --> 00:50.200 I'm here today to speak about regulatory trends for artificial intelligence, particularly those coming from Europe. 00:50.200 --> 00:54.740 I know this is not a topic otherwise featured at the Crypto & Privacy Village. 00:54.740 --> 01:02.280 And I hope you will bear with me when we venture out into a topic that is not top of mind for most of you. 01:02.280 --> 01:14.620 But I expect it to be at least as important as GDPR, the European data protection regime that took effect two years ago, and that hit many in the technology space in the US unprepared. 01:15.480 --> 01:21.360 As with any new technology, the use of AI brings both opportunities and risks. 01:21.360 --> 01:31.480 Citizens fear being left powerless in defending their rights and safety when facing informational asymmetries of algorithmic decision making. 01:31.480 --> 01:34.140 And companies are concerned by legal uncertainty. 01:35.020 --> 01:44.740 I'm usually not one to quote CEO of Tesla and SpaceX Elon Musk, but no doubt he's among those who said it the pointiest way. 01:44.740 --> 01:49.920 Mark my words, he said, AI is far more dangerous than nukes. 01:49.920 --> 01:52.760 Why do we have no regulatory oversight? 01:53.200 --> 01:56.100 So of course, he's not the only one to say that. 01:56.100 --> 02:06.060 Other corporate leaders have called for increased regulation and so have politicians and respected scholars and academic research institutes and think tanks. 02:06.280 --> 02:23.680 Reasons for this call to action is the understanding that for all its many benefits, AI also presents many risks and they include biased algorithms, privacy violations, and the potential for injuries caused by defective software. 02:23.680 --> 02:38.560 So with the increasing use of AI based solutions in areas like criminal justice, healthcare, robotics, financial services, and education, corporate interests will inevitably conflict with societal benefits. 02:38.560 --> 02:45.420 So that conflict raises the question of what systems should be put in place to mitigate potential harms. 02:45.780 --> 02:56.140 My research is focused on upcoming regulation in Europe for applications of artificial intelligence and how they could affect US companies. 02:56.140 --> 03:09.080 Not the big tech giants that we all think of that all have their offices in Brussels observing and influencing these discussions, but the small ones and that don't operate in Europe or globally yet. 03:09.480 --> 03:20.000 If you have thoughts about this as an individual, but also if you work for a company that fits into what I'm looking at, I'd love if you reached out and gave me feedback. 03:20.060 --> 03:27.140 So on my last slide, I share my website and my Twitter handle again, and I'd be really grateful to receive your messages. 03:27.850 --> 03:34.000 So you could ask, why would a US organization be affected by European regulation? 03:34.000 --> 03:36.840 They only have to follow US law. 03:36.940 --> 03:53.280 But actually, what we have seen with the European Union's General Data Protection Regulation, GDPR, is that lots of businesses based solely in the US are affected, mainly because they have European users. 03:53.280 --> 03:56.900 So they process personal data of European residents. 03:56.940 --> 04:03.220 And these residents of Europe are protected by GDPR wherever the processing of their data takes place. 04:03.220 --> 04:06.800 So I could go on forever on GDPR. 04:06.800 --> 04:17.140 It's a topic that I know well and that I've worked on for years, including when it was still being negotiated among EU member states, and I worked for one of them. 04:17.180 --> 04:39.260 But an important experience I wanted to share here is that many US-based organizations that process personal data of people around the world have decided to apply GDPR and extend all rights that go with it to their customers who are not European residents but live outside of Europe. 04:41.780 --> 04:47.580 So I've made a little slide on the advantage of GDPR compliance on a global level. 04:48.340 --> 04:53.120 To these companies, it gives them an edge in global compliance. 04:53.120 --> 04:57.280 It's easier for them in terms of handling complaints and requests. 04:57.560 --> 05:04.560 So they say, just give all of our customers all the rights that Europeans have, a very high bar. 05:04.580 --> 05:12.940 And it's still easier for the organization than to sort out the customer's location and attribute different rights according to their location. 05:13.340 --> 05:27.600 GDPR offers them a legal framework and a set of standards that is at least compared to other less spelled out legislation or the lack thereof, relatively clearly adoptable. 05:28.860 --> 05:48.700 So while my clients are mainly small and medium enterprises based in the US, with only some clients in Europe, or with the mere intention of soon expanding to Europe, this privacy management strategy has been found with bigger tech firms as well. 05:48.700 --> 05:54.880 So the image on the slide that you see shows the view from Salesforce's legal team space in San Francisco. 05:55.460 --> 05:58.820 And they are certainly part of these companies I'm talking about. 05:59.600 --> 06:10.580 Organizations appreciate that there is a standard now that is law in one part of the world that can serve as a guideline also for other parts of the world. 06:10.580 --> 06:27.820 And even if this guideline is more demanding than legislation in their markets outside of Europe, it makes life so much easier for them to have one high profile standard than have many different ones, what I call the global privacy patchwork, or none at all. 06:27.820 --> 06:38.040 Now, this was definitely a huge finding in the past two years since May 2018, when GDPR became enforceable. 06:38.080 --> 06:59.160 Also, with the US administration being relatively silent about data protection, and other countries like Japan or Israel following a rather GDPR-like model, that made the EU the de facto rule setter in the technology policy worldwide, in the important field that is data protection, 06:59.160 --> 07:02.700 as data is integral to all technology. 07:03.240 --> 07:14.620 So what I want to find out now is whether we could expect to see the same trend with upcoming EU regulation in other fields of tech policy, for example, artificial intelligence. 07:16.520 --> 07:29.700 So if we look at the timeline of policymaking on AI in Europe, and that's the top line, the dark blue one, we're pretty much in the middle of concepts gearing up to become concrete policy proposals. 07:29.860 --> 07:46.780 As you see on that timeline, after sketching first ideas and establishing a high level expert group that has issued its proposals, the European Commission issued a white paper early this year and started a consultation process in which more than 1,200 07:46.780 --> 08:00.640 individuals, advocacy groups, researchers and companies participated, and I helped to formulate the Mozilla contribution. 08:00.640 --> 08:17.380 Most recently, in July, the experts last July, the experts invited by the European Commission have issued an assessment list for trustworthy AI that companies and developers can use to assess their algorithm use. 08:17.620 --> 08:33.660 On the international level, that's the bottom part of the timeline, many of the concepts that form part of the European draft have been formulated as principles within the OECD, the Organization for Economic Development and Cooperation and Development, 08:33.660 --> 08:40.540 that most European countries and the US are members of, adopted in May 2019. 08:40.720 --> 08:48.460 They promote uses of AI that are innovative and trustworthy, and that respect human rights and democratic values. 08:48.460 --> 08:58.840 A month later, the G20, the world's biggest economies, have drawn from them heavily when they adopted their human-centered AI principles. 08:58.840 --> 09:03.480 Of course, these are all non-binding, but still highly influential. 09:04.000 --> 09:16.740 In the US, the White House announced the American AI Initiative, which focuses on driving technological innovation and standards to protect a competitive edge on AI. 09:16.740 --> 09:26.920 So the White House last January released 10 principles for federal agencies to follow when proposing rules governing the private sector's deployment of AI technology. 09:26.920 --> 09:43.100 These are binding, and they follow many of the same ideas, but they are very light touch, with lots of exceptions, and a strong emphasis on leaving space for corporations in whatever way to innovate and compete globally. 09:44.060 --> 09:59.900 It's a more hands-off free market view, and a pretty stark renunciation of the careful approach that was initiated under the Obama administration in 2016, but that hadn't yet led to any actual US comprehensive legislation. 10:00.940 --> 10:03.080 So, back to Europe. 10:03.080 --> 10:21.100 Right now, Brussels is working on a draft policy that is expected to be tabled to member states for negotiation around the beginning of 2021, and that could, once those negotiations have been successful, become the world's first general regulation on AI. 10:23.660 --> 10:28.780 So what will European AI regulation look like, most likely? 10:29.240 --> 10:47.000 It's a three-pronged approach, with a substantial increase in public and private investments in AI to boost its uptake, but also funding for research, more cooperation between researchers across Europe, and better know-how to prepare for socio-economic changes. 10:47.000 --> 10:51.660 This is called the ecosystem of excellence on top. 10:51.660 --> 11:00.160 And I would say that the numbers floating here don't add up in any way to what is seen in the US or China, but it's a start. 11:01.220 --> 11:22.700 Combined with a European strategy for data, which aims to improve the use of data by creating an EU single market for data, because, of course, as you know, it's 27 different member states, so we're aiming for a single market in that field to facilitate access to data and computing infrastructures, 11:22.700 --> 11:28.820 which is, of course, an essential requirement for the development and use of AI applications. 11:29.220 --> 11:34.720 And last, and I've put that in the middle, with the aim of an ecosystem of trust. 11:35.160 --> 11:47.780 The Commission uses a human rights first approach and details a number of requirements that AI applications would have to fulfill in order to be considered trustworthy. 11:47.840 --> 11:51.400 This is the new framework that I will present further on. 11:51.640 --> 12:09.080 And, of course, not forget existing EU laws and regulations, whether on EU level or in the member states, that already apply to AI solutions, including rules on data protection, GDPR, non-discrimination, consumer protection, product safety, and liability. 12:09.620 --> 12:18.380 Consumers, of course, expect the same level of safety and respect of their rights, whether or not a product or a system relies on AI. 12:18.700 --> 12:30.340 So, in some cases, the AI-related aspects of these existing rules may be difficult to enforce because AI systems are opaque, unpredictable, complex, and autonomous. 12:30.780 --> 12:35.780 So, in some cases, they might need to be amended or updated. 12:36.260 --> 12:56.600 So, I'll come to the requirements on my next slide, but I need to first underline that this approach singles out high-risk applications, which would become subject to strict requirements and not those that are considered low-risk, for which no additional legal requirements would be imposed. 12:56.600 --> 13:12.100 And for these low-risk ones, the Commission is considering a voluntary labeling system that would certify compliance with parts of the requirements and allow companies to market their AI products as trustworthy. 13:12.640 --> 13:26.200 So, rather than introducing a generic AI application, this is a regulation, sorry, this is a nuanced risk-based approach, possibly one that is application and technology specific. 13:26.260 --> 13:47.360 However, this approach may also lead to uncertainty, and I personally worry that although the text includes a definition of what is meant by high-risk sectors, specifically mentioned are healthcare, transport, energy, judicial decision-making, and mass citizen surveillance, 13:47.880 --> 13:51.480 it still won't always be easy to determine. 13:51.480 --> 14:01.760 It is very subjective, and an algorithm use could be considered low-risk for someone, but actually has high-risk consequences for others. 14:01.760 --> 14:14.540 Plus, risk here is defined as a risk to an individual, which excludes a whole category of AI applications that pose major collective risks. 14:17.320 --> 14:25.060 So, developers of high-risk AI applications would be obliged to follow rules in these key aspects. 14:25.320 --> 14:28.100 First, human oversight. 14:28.100 --> 14:42.180 Some degrees of human oversight would be required, ranging from requiring human review before a decision is implemented to the possibility of human intervention in real-time or afterwards. 14:42.600 --> 14:55.440 Then there could be requirements on robustness and accuracy, the ability to react to inconsistencies, and that the application is resilient against attacks and manipulation. 14:56.400 --> 15:04.100 Training data sets need to be representative and comprehensive and comply with privacy and data protection rules. 15:05.160 --> 15:06.760 Information to be provided. 15:06.760 --> 15:16.400 People would need to be informed when they interact with an AI system and not a human, and what its capabilities and limitations are. 15:17.940 --> 15:30.420 Developers would also need to keep records of how they selected what kind of training and testing data, and how they programmed and trained the system to enable AI decisions to be traced back. 15:31.040 --> 15:40.900 And then there will probably be specific requirements for remote biometric identification, also known as facial recognition. 15:40.980 --> 16:00.980 So, no ban inevitably, but the EU insists that data protection rules and the Charter of Fundamental Rights apply, which already allow the uses of facial recognition only in cases where this action is justified and proportionate and is subject to adequate safeguards. 16:01.480 --> 16:07.040 So, there will probably be an open discussion on whether exceptions could be justified. 16:07.620 --> 16:18.840 So, as a developer, and I know that many of you are developers, think of this like some kind of FDA-style clinical testing, not for drugs, but for algorithms. 16:19.900 --> 16:26.120 So, how do I expect US companies to be affected by future regulation on AI? 16:27.820 --> 16:37.700 The EU guidelines will likely have a ripple effect in the US, since many American technology companies provide AI solutions and services to the EU. 16:37.700 --> 16:49.160 It will also impact US companies that EU investors are looking to buy, and companies that plan to expand into European markets. 16:49.160 --> 17:06.940 Of course, the additional policy and regulatory measures the EU considers will increase the cost of compliance and put the administrative burden and possible IP-related difficulties on companies that develop or deploy AI systems. 17:07.560 --> 17:20.160 And, of course, on a less business and more political level, citizens will see what rights are granted in the EU, and at least some of them could demand those rights as well. 17:21.240 --> 17:32.200 So, in the US, it's private businesses that are currently pushing harder for AI regulation than government agencies. 17:32.240 --> 17:44.320 Some of the big players, I mentioned them here, but it's also Salesforce that I mentioned earlier, have established internal principles on which they base their AI development in the future. 17:44.860 --> 17:58.300 This is not ideal to my mind, but it's a first step, because internally established principles of AI ethics are hardly enforceable, competitors are not bound by them. 17:58.300 --> 18:11.560 And also, there are so many guidelines around from a consumer point of view that I take, it's hard to really know where a company stands in terms of compliance, because everybody has their own rules. 18:11.960 --> 18:28.280 But at least it obliges the company that decides to go this way to face scrutiny about its products, but it isn't designed to really calm citizens' concerns on a broad base, because there are still players around who don't feel bound by any of those principles. 18:29.680 --> 18:33.260 And then there's self-regulation of industry. 18:33.260 --> 18:42.560 And by that, I mean that a whole industry designs and enforces new rules and standards for themselves when government rules are lacking. 18:42.560 --> 18:52.660 Usually, these would be codes of conduct, or binding rules that industry establishes for itself, for example, within federations. 18:52.700 --> 19:18.500 The hope would be that industry collective action can change the incentives in a situation where the competitive dynamic of AI development leads to a prisoner dilemma for both companies and state, wherein both are incentivized to prioritize the fast development of AI instead of the safe development of AI. 19:18.500 --> 19:23.740 But unfortunately, this hasn't been successful, or at least not impactful for now. 19:25.120 --> 19:31.420 And that's why some companies actively push government for more committed steps in the US. 19:31.420 --> 19:37.160 I've often heard people in companies say that they look to Europe for these obligations to come up. 19:37.160 --> 19:54.600 They are mostly the ones who've seen the success of GDPR and are convinced the US needed similar groundbreaking law regulating technological development, but they don't see that the current US administration is in any way interested in pushing that. 19:54.620 --> 20:05.840 The dominant view in DC for the moment is that the US needs to lead in AI development to compete with China, no matter what. 20:05.840 --> 20:08.560 But of course, we have elections in November. 20:08.560 --> 20:10.640 So please vote. 20:12.300 --> 20:17.480 At last, and last, there's the way through local and state laws. 20:17.980 --> 20:26.820 We've seen that with data protection, that when the federal level in the US doesn't move enough, the states engage more. 20:26.820 --> 20:33.520 There are some examples that state and local level have moved that are pretty promising. 20:33.520 --> 20:54.080 They don't have the reach to create general nationwide regulation, of course, but they're creating smaller scale uses, use cases for regulation, like regulatory sandboxes, but in real life, that could be useful once the federal level is ready to engage. 20:54.740 --> 21:14.600 So for instance, the California Consumer Privacy Act, CCPA, that took effect last year, showed that a US state can issue impactful legislation that addresses concerns about data protection and that significant fines can be levied against misuse of personal data. 21:14.900 --> 21:18.100 So the CCPA is far from perfect. 21:18.360 --> 21:27.840 And it started to be enforced only now, but I don't really see how it will be effectively enforced, given so many imperfections. 21:27.840 --> 21:30.120 But still, it was quite a step. 21:30.120 --> 21:35.140 And many US states have meanwhile issued their own privacy legislation. 21:35.140 --> 21:42.020 However, none of them is as far reaching as the California law. 21:43.020 --> 22:01.460 A more recent example, so more specifically from the AI field, again, from California, is the BOT, the Bolstering Online Transparency Act, which makes it unlawful for any person to use a bot to communicate or interact with another person in California online, 22:01.460 --> 22:05.710 with the intent to mislead the other person about its artificial identity. 22:06.260 --> 22:18.260 This legislation specifically applies to bots that intend to influence voters, as well as intentionally deceptive bots used to sell goods and services. 22:18.260 --> 22:25.360 It doesn't make bots illegal, but it requires them to identify themselves as non-human. 22:25.900 --> 22:39.060 So the law took effect a month ago, July 1st, only in California, and it doesn't clearly address how it applies to bots that don't originate in the Golden State, and how enforcement would work then. 22:39.560 --> 22:51.980 So one last example for local action on municipal level, this time regarding facial recognition technology, which is the most controversial area of AI use today. 22:51.980 --> 23:07.660 In this case, when so far no federal guidelines exist to limit or standardize its use, and few state rules are in place, cities feel left to decide for themselves what, if anything, to do. 23:07.980 --> 23:18.600 So the city and county of San Francisco last year passed a bill to ban facial recognition technologies for police and city agency use. 23:18.740 --> 23:22.380 So other US cities have followed suit. 23:22.380 --> 23:34.740 In this case, it is only one sector, government, that is required to refrain from the use of this technology, while the private sector and individual can still use it. 23:34.740 --> 23:48.400 But it shows an awareness that with the current state of technology development, organizations clearly aren't ready to use facial recognition in a safe, secure, and responsible manner. 23:48.640 --> 23:55.880 And that there is a need for clear legislation that, on a federal level, so far no one has managed to pass. 23:56.320 --> 24:07.340 As I mentioned before, what most technology companies are saying is that they look not to DC, but to Europe when it comes to effective regulation in that field. 24:08.480 --> 24:25.280 So what I've been doing recently is to interview companies in the Bay Area working on AI systems about the expected impact of European regulation on them, and about their preparedness and their sense of agreement with these principles. 24:25.920 --> 24:40.140 My focus, I mentioned that, is on small and medium-sized companies that are less in the focus of the media and that face the additional challenges competing with the tech giants in a field where size clearly matters. 24:40.140 --> 24:45.200 Because the more data you can gather, the better your AI works. 24:45.320 --> 24:59.460 So I'm also interested in this type of company because the general impression with GDPR was that the small players have been losing ground due to compliance costs, and that regulation blocks them more than the big ones. 24:59.900 --> 25:22.020 So far, my interviews show that most, if not all, contacts have an understanding of the importance of rule-setting around AI-based systems, and that they report having thought about limiting their own developments until political guidance is given what is admissible and what is not. 25:22.060 --> 25:28.840 All of them had experienced some kind of adaptation process to GDPR two years ago. 25:28.840 --> 25:31.980 Some of them were my clients, so I helped them with that. 25:32.420 --> 25:48.300 And they reported that having spent considerable money and time on compliance with rules that originated in Europe, but that several of them applied to all of their products and all users wherever they are located. 25:48.640 --> 26:03.620 Now, with AI regulation coming up, they expect the same to happen, but there is disagreement on whether this kind of general regulation is even possible in a broad field like AI. 26:03.620 --> 26:06.380 So they consider it too broad to fall under one rule. 26:06.380 --> 26:16.520 That includes small applications that are considered to simply make life easier, as well as large systems of mass surveillance. 26:17.240 --> 26:30.460 So importantly, many of them also said that they expected European legislation to be faster, more comprehensive, and more enforceable than any US attempt for regulation. 26:30.460 --> 26:51.140 Maybe that's a lesson learned from GDPR and doesn't mean that this would be the same with the upcoming policies in the AI sector, but it's a good indicator of the urge many players feel to prepare for rules and not to leave the space entirely to the big players and their lobbying power. 26:51.900 --> 27:07.580 So in my mind, although the EU's upcoming rules are geared primarily to European firms and will become law only potentially sometime in the future, US companies need to look at them already now. 27:07.740 --> 27:15.220 So it won't be enough to just wait and see, and it's better to prepare for the probability of AI regulation in Europe now. 27:17.600 --> 27:28.280 So again, if you have thoughts about this as an individual, but also if you work for a company that fits into what I'm looking at, I'd love if you reached out and gave me a feedback. 27:28.280 --> 27:31.620 Here's my website and Twitter handle at the bottom of the page. 27:31.620 --> 27:43.140 And if you're interested in reading more about this field and add your comments, have a look at the white paper that we're working on with the Mozilla Foundation at the link in the center of the slide. 27:43.380 --> 27:45.280 So thank you for attending. 27:45.720 --> 27:48.440 I wish I could have seen you all in person. 27:48.500 --> 27:51.420 I hope you're all safe and stay healthy. 27:51.480 --> 27:53.880 And I'm looking forward to your questions. 28:00.800 --> 28:02.800 And welcome back. 28:02.800 --> 28:06.180 We have Julia here with us. 28:06.480 --> 28:11.920 If anybody has any questions, drop them in the Q&A channel and Discord. 28:12.260 --> 28:14.340 We've got a few here. 28:14.380 --> 28:15.880 Julia, I enjoyed your talk. 28:15.880 --> 28:22.320 Is there a document with more in-depth info about the requirements of trustworthy AI in high-risk applications? 28:23.860 --> 28:39.320 Yeah, so I directed the person who asked to the assessment list that has been published by the High-Level Expert Group this July, a couple weeks ago. 28:39.320 --> 28:47.300 And I think that shows a bit of what they expect, you know, in terms of actual requirements, very concretely. 28:47.340 --> 28:51.380 But of course, I mean, remember that this hasn't been finalized yet. 28:51.380 --> 28:52.800 This is just a proposal. 28:52.800 --> 28:55.180 Well, it's actually not even a concrete proposal yet. 28:55.180 --> 28:59.860 It will become a proposal probably around the beginning of 2021. 28:59.860 --> 29:01.660 So it's all in the making. 29:02.740 --> 29:03.080 Cool. 29:03.080 --> 29:04.220 Thank you. 29:04.600 --> 29:07.200 And then here's another question. 29:07.460 --> 29:18.120 What would you like to see happen with CCPA in regards to clarifying bots, whether from California or not, on how bots identify as non-humans prior to contact? 29:18.120 --> 29:21.220 What type of enforcement or not would you like to see? 29:22.780 --> 29:29.340 Yeah, so I think there are two aspects here, bots specifically, and then CCPA. 29:29.360 --> 29:50.200 For CCPA, I think my main doubt is that the Attorney General in California won't or doesn't have the bandwidth, really, the capacity to really enforce all the cases, because it really depends on him and his team to find and to inquire and then to enforce. 29:50.340 --> 29:57.860 And I don't think this will really work, especially because the text is not that clear. 29:58.640 --> 30:01.400 And so that is my main doubt. 30:01.400 --> 30:03.100 I mean, there are lots of others. 30:03.100 --> 30:10.800 You know, CCPA is really kind of not very well designed in terms of wording and terminology. 30:11.760 --> 30:15.900 But it's been enforceable only since 1st of July this year. 30:15.900 --> 30:18.100 So we'll see how that goes. 30:18.100 --> 30:25.780 In terms of bots, I think it's most useful to look at the bot bill that's been in effect also since July 1st here in California. 30:34.060 --> 30:48.080 It's a I think, I mean, also there, the application, the actual what it applies to is a bit narrow, but it shows how this could work. 30:48.080 --> 30:54.100 You know, you really, when you're contacted by a bot, you know exactly whether you're talking to you or not. 30:54.380 --> 30:57.580 And that would be the same also in the European proposal. 30:57.580 --> 31:03.580 I mean, not exactly the same that we don't know yet, but the idea is the same. 31:03.580 --> 31:04.800 That'd be interesting. 31:05.200 --> 31:07.960 All right, we have one last question here. 31:07.960 --> 31:17.920 Do you think that AI will develop that much to actually take us over, take us over in the sense that they might steal our jobs, our houses, our budget, etc? 31:19.200 --> 31:25.300 Well, I mean, there, of course, the eternal question is, what do you define as AI? 31:25.300 --> 31:53.440 And if we take a broad look at it, your wording of the question is, sounds a bit dramatic, but I mean, in a way, in some sectors, you could be right, because in the end, it's really important for humans to control what they're creating and to actually have a last say in what the technology is supposed to do. 31:53.440 --> 32:01.760 And so I think if we, I think the awareness is there to find rules, to make that, to make sure that this happens. 32:02.080 --> 32:10.440 So I do think that in some sectors, there's a potential for actual harm, yes. 32:10.780 --> 32:12.320 And for dramatic harm. 32:12.320 --> 32:15.500 I mean, this is in the very high risk sectors, of course. 32:15.500 --> 32:23.020 But even in the less high risk sectors, I think that, you know, democracy, fundamental rights, are already at stake. 32:23.020 --> 32:23.580 Yes. 32:24.500 --> 32:25.460 Wow. 32:25.980 --> 32:29.240 Well, thank you for taking the time to join us today. 32:30.320 --> 32:36.140 If you have any more questions, put them in the Discord and Julia will be able to answer them there. 32:36.940 --> 32:38.100 Thank you. 32:38.460 --> 32:39.000 Thank you. 32:39.000 --> 32:41.100 Our next talk up will be...