[00:00.000 --> 00:07.820]  All right, it looks like we are resetting the stream. One quick second, let me get a
[00:07.820 --> 00:15.510]  heads up by our recording folks. Let's see. Let me just double-check while we
[00:15.510 --> 00:19.810]  get this, get this sorted. Yeah, and sometimes I would announce a class like
[00:19.810 --> 00:26.210]  welcome to Chem 107 or something, just something weird. I like to have fun with my
[00:26.210 --> 00:31.890]  students, but when I was teaching, it was good. I don't think I was ever bold
[00:31.890 --> 00:37.970]  enough to try anything like that. Or maybe I just had more pity for my
[00:37.970 --> 00:40.130]  students. I don't know, one of those two.
[00:42.730 --> 00:50.010]  Well, I lucked out and taught a, what do you call it, a simple class on...
[00:50.610 --> 00:55.550]  it was we were building an electric vehicle. Oh, hi everybody. For those of you
[00:55.550 --> 00:59.250]  that are just tuning in, we're holding out for one quick second while we get
[00:59.250 --> 01:07.290]  our recording AV equipment going. Mr. The Fixer, let me know when things are
[01:07.290 --> 01:10.370]  ready to go and we will commence.
[01:13.750 --> 01:19.190]  Let's see, it looks like... all right, so looks like things are ready. So let us
[01:19.190 --> 01:26.650]  begin. We're introducing our speakers. So everybody, welcome to the very first talk
[01:26.650 --> 01:33.450]  of Ethics Village on day two. So today we have Larry Lewis and we have Diane
[01:33.450 --> 01:37.010]  Bavacek, and they were going to be talking a little bit about killer
[01:37.010 --> 01:43.390]  robots, so reconsidered. So a little bit about Larry. Larry is the director of the
[01:43.390 --> 01:48.030]  Center for Autonomy and Artificial Intelligence at CNA. His areas of
[01:48.030 --> 01:52.790]  expertise include military applications of artificial intelligence and autonomy,
[01:52.790 --> 01:57.690]  reducing civilian casualties, identifying lessons from current operations, security
[01:57.690 --> 02:03.550]  systems, and counterterrorism. So Larry spearheaded the first database approach
[02:03.550 --> 02:08.990]  to protecting civilians in conflict, analyzing military operational data in
[02:08.990 --> 02:13.490]  conjunction with open source data. He has worked extensively to help reduce, to
[02:13.490 --> 02:17.510]  help militaries reduce civilian casualties in their operations. Larry has
[02:17.510 --> 02:21.130]  also served as a senior advisor to the Department of State's Assistant
[02:21.130 --> 02:27.050]  Secretary for Democracy, Human Rights, and Labor, and was on the U.S. delegation to
[02:27.050 --> 02:31.770]  the United Nations regarding lethal autonomous weapon systems or laws. Larry
[02:31.770 --> 02:36.650]  currently holds a PhD in physical chemistry from Rice University. And our
[02:36.650 --> 02:42.010]  other speaker, Diane Bavacek, is an analyst at CNA, where she researches
[02:42.010 --> 02:46.850]  military operations and policy with a focus on emerging technologies, including
[02:46.850 --> 02:50.950]  autonomy, artificial intelligence, unmanned systems, and the cyberspace
[02:50.950 --> 02:57.090]  domain. She deployed abroad on the USS Vinson during Operation Iraqi Freedom
[02:57.090 --> 03:02.150]  and Operation Inherent Resolve. Diane currently has a PhD in mathematics from
[03:02.150 --> 03:08.690]  the University of Michigan. And without further ado, welcome again, y'all, and
[03:08.690 --> 03:13.330]  thank you again for doing this talk for Ethics Village 2020. Let me queue up
[03:13.330 --> 03:25.110]  those slides and I will pass it off to you. So I'm gonna go mute. Great. So before we
[03:25.110 --> 03:28.850]  get to the slides, just a few introductory comments. So thank you.
[03:28.970 --> 03:35.570]  Thank you for joining us. Diane and I are really excited to talk to you guys. Just
[03:35.710 --> 03:41.290]  a few things. If this is your first acquaintance with CNA, we're a
[03:41.290 --> 03:45.970]  nonprofit research organization and we provide objective and independent
[03:48.010 --> 03:52.930]  analysis and perspectives to the U.S. government, including the U.S. military,
[03:52.930 --> 04:00.070]  but also other parts of the government. And several years ago, CNA
[04:00.070 --> 04:05.490]  established a Center for Autonomy and Artificial Intelligence because those
[04:05.490 --> 04:11.170]  technologies are so important to the U.S. and national security priorities. So
[04:11.170 --> 04:19.910]  and a key focus for us at CNA is AI safety. So we're going to be talking about that today.
[04:20.110 --> 04:27.470]  This presentation incorporates a number of different areas of CNA research. At the same
[04:27.470 --> 04:35.600]  time, this presentation reflects our opinion and not that of CNA or the U.S. government.
[04:36.520 --> 04:43.280]  Finally, we really want your participation today. So we want this to be participatory.
[04:43.280 --> 04:51.060]  So you can send us a question in the Discord general text chat. We're monitoring it. I have
[04:51.060 --> 04:57.720]  it open right here. So feel free, please, to ask questions. We'd love to answer them.
[04:57.720 --> 05:03.000]  We're aiming to finish this presentation and the general question and answer in an hour.
[05:03.000 --> 05:08.400]  But we are also happy to stick around longer if we haven't gotten your questions. So feel
[05:08.400 --> 05:13.580]  free to stick around if we haven't answered it, and we will gladly get to it. So with that,
[05:13.580 --> 05:16.980]  we'll go ahead and get started. And Diane is going to start us off.
[05:17.820 --> 05:24.300]  Thanks, Larry. And let me echo what Larry said. Thanks to the Ethics Village for hosting us.
[05:24.880 --> 05:30.400]  Steve, you can go ahead to the next slide, please. Thanks. So what are we going to talk
[05:30.400 --> 05:37.680]  to you about today? So here's kind of a general outline. Militaries are turning to artificial
[05:37.680 --> 05:43.020]  intelligence. And the U.S. Department of Defense is certainly one of those that sees
[05:43.020 --> 05:50.780]  moving forward with AI as an imperative. And the DoD is going to need the tech sector,
[05:50.780 --> 05:56.020]  the help of the tech sector, the help of AI researchers in order to be competitive here.
[05:56.860 --> 06:03.920]  There are risks associated with incorporating AI into military applications. A lot of risks and
[06:03.920 --> 06:08.240]  challenges that are really important and crucial getting right. But there are also a lot of
[06:08.240 --> 06:12.180]  misconceptions that come up in this conversation. And so we're going to talk both about challenges
[06:12.180 --> 06:19.920]  and the misconceptions. And DoD practices and policies that are in place right now that address
[06:19.920 --> 06:26.940]  some of these risks. And again, to go back to the role of the tech sector and AI researchers here,
[06:26.940 --> 06:32.680]  addressing some of the challenges associated with AI in this setting is something that we
[06:32.680 --> 06:38.800]  think the tech sector can really help DoD to do. In addition to risks, though, there are actually
[06:38.800 --> 06:45.840]  benefits, opportunities associated with integrating AI into military applications. So there's a real
[06:45.840 --> 06:52.340]  potential for benefits like fewer civilian casualties or protecting hospitals, different
[06:52.340 --> 06:57.440]  humanitarian benefits that we'll talk through. And again, to bring back to what is the role
[06:57.440 --> 07:02.880]  of tech sector here, we've seen through history that outside influences can motivate DoD to try
[07:02.880 --> 07:11.680]  harder in these areas. So what we have here is a new paradigm with AI coming into applications
[07:11.680 --> 07:17.280]  in the military and more broadly. And we think that you should see this as an opportunity to
[07:18.240 --> 07:24.760]  help and influence the way that AI is used in warfighting. So this is our aim. The plan for
[07:24.760 --> 07:29.000]  this talk is that I'm going to talk through the first few points and then Larry will talk
[07:29.000 --> 07:35.180]  through the later points. And then we will spend some time going through the questions that we see
[07:35.180 --> 07:45.040]  in the chat. All right, go to the next slide, please. So what are we talking about here,
[07:45.040 --> 07:51.200]  first of all? How are we thinking about the definition of AI? So to be consistent with how
[07:51.200 --> 07:56.200]  the U.S. government uses this term, we're thinking of this broadly. So one definition you can think
[07:56.200 --> 08:01.800]  of for AI is a set of techniques, including machine learning, that is designed to approximate
[08:01.960 --> 08:13.080]  a cognitive task. So one kind of piece that I want to get at, at this point, is to talk about
[08:13.080 --> 08:19.700]  how people group AI into kind of two broad bins. So people talk about artificial general intelligence.
[08:19.900 --> 08:25.920]  And the idea there is kind of thinking of the Terminator, sort of, right? Is artificial
[08:25.920 --> 08:31.700]  intelligence that can do what a human can do? That can understand or learn any cognitive task
[08:31.700 --> 08:37.920]  that a human can do. So artificial general intelligence is not the state of technology today,
[08:37.920 --> 08:46.520]  and it's not expected to come to fruition in the foreseeable future at this point. So this is not
[08:46.520 --> 08:49.900]  going to be the focus of our talk today. What we're going to focus on is the other category,
[08:49.900 --> 08:57.860]  which is narrow artificial intelligence. So narrow AI is AI that's focused on a specific task,
[08:57.860 --> 09:02.440]  something like image classification, right? So it carries out a specific task within a specific
[09:02.440 --> 09:12.180]  context. Why is AI relevant to warfighting? Well, one reason is that AI helps you make better
[09:12.180 --> 09:19.200]  decisions. So if you might want the best possible decision-making tools to be used in areas of your
[09:19.200 --> 09:23.840]  life that you care a lot about, right? Maybe the person managing your investments or a doctor
[09:23.840 --> 09:29.580]  giving you diagnosis, if you want them to have access to the best technologies to help them with
[09:29.580 --> 09:35.580]  decision-making, right? It's the same idea. This is why military commanders, this is why U.S.
[09:35.580 --> 09:45.740]  government leadership wants to draw from this technology. Also in warfighting, AI can be useful
[09:45.740 --> 09:52.260]  if your military is facing a threat that is vast or numerous or very dynamic. So think about
[09:52.260 --> 09:58.720]  something like a drone swarm attacking a naval ship, right? So that's the kind of setting that
[09:58.720 --> 10:03.280]  DoD cares very much about, being able to combat that threat. And so this is another place where
[10:03.280 --> 10:10.280]  AI can be really critical. So for all of these reasons, DoD and other militaries around the world
[10:10.280 --> 10:16.360]  are moving forward with exploring and investing in artificial intelligence.
[10:17.460 --> 10:22.960]  What type of military applications are we talking about here? So a few examples, right? And I've
[10:22.960 --> 10:29.540]  listed a few bins here in the slide. So AI can help with by providing decision aids on the
[10:29.540 --> 10:34.880]  battlefield. So things like detecting anomalies in behavior, the behavior of say people on the
[10:34.880 --> 10:42.880]  ground or shipping traffic. Sensor data fusion is another application. Human machine teaming
[10:43.280 --> 10:52.240]  is also a key area of AI applications. So think about maybe an unmanned ground system that is
[10:52.240 --> 10:57.220]  assisting a soldier by carrying a lot of heavy gear or by providing surveillance where the
[10:57.220 --> 11:03.020]  soldier is patrolling. And business applications are also something that are going to be important
[11:03.020 --> 11:11.940]  to DoD going forward. AI uses in say preventative maintenance, network defense, or even to figure
[11:11.940 --> 11:21.600]  out how to target recruiting efforts. But an AI is going to, you know, there are important
[11:21.600 --> 11:26.000]  challenges of artificial intelligence that comes into those type of applications, but that's not
[11:26.000 --> 11:33.100]  the most contentious type of applications that we're talking about here. So DoD and other
[11:33.100 --> 11:39.940]  militaries are also considering how you might integrate AI into autonomous systems or weapon
[11:39.940 --> 11:46.260]  systems, right? So autonomous systems being systems that operate in some sense on their own,
[11:46.260 --> 11:52.260]  right? And you can combine that with a lethality component too. And so here's a case where you
[11:52.260 --> 11:58.820]  really see that it's critical that we make sure to get it right moving forward with this technology.
[12:00.180 --> 12:09.920]  Next slide. Okay, so DoD and the U.S. government more broadly are thinking about how to use
[12:09.920 --> 12:16.320]  AI in national security. So they're already moving ahead with creating strategies and guidance,
[12:16.320 --> 12:22.040]  making some organizational changes, and investing in research and development. And so I've listed
[12:23.120 --> 12:30.480]  some examples of these types of developments in the slide. The one that I'll anchor on here
[12:30.480 --> 12:38.000]  is this first sub-bullet, which is the recently established DoD AI strategy.
[12:38.740 --> 12:43.920]  And this is a document that... this and most of the other documents that I reference here
[12:44.500 --> 12:48.640]  are things that are available open source. So you can just Google and find at least an
[12:48.640 --> 12:55.160]  unclassified summary of the strategy. So a couple of things that I'll just list off here,
[12:55.160 --> 13:00.420]  just to anchor on this piece, a couple of things that they say in the strategy, right? So they
[13:00.420 --> 13:06.920]  emphasize that the women and men in the U.S. armed forces are the real source of strength.
[13:06.920 --> 13:13.260]  And the intent is for AI to empower but not replace those who serve.
[13:14.240 --> 13:19.120]  Another thing that the strategy says is that AI is poised to change the character
[13:19.120 --> 13:25.020]  of the future battlefield and the pace of the threats that we must face.
[13:25.880 --> 13:32.260]  And so I just want to point out that you can see from that line that the DoD is... the context that
[13:32.260 --> 13:38.460]  DoD is thinking about this in is one in which they expect that adversaries will also be using
[13:38.460 --> 13:46.300]  this technology. So that's kind of the landscape that we're thinking of here. Go ahead to the next
[13:46.300 --> 13:56.200]  slide. Thanks. So AI really represents a paradigm in terms of the relationship between DoD and the
[13:56.200 --> 14:02.820]  private sector. That's a new and interesting dynamic, which is that in the past, we're used
[14:02.820 --> 14:08.840]  to thinking of DoD as having unique needs with research and development and that they very much
[14:08.840 --> 14:16.420]  bankroll the research in order to develop platforms they need, like military aircraft
[14:16.420 --> 14:23.820]  or develop weapon systems technologies. Even early versions of the internet were something that DoD
[14:24.540 --> 14:28.960]  bankrolled because it was something that they needed and the developments weren't available
[14:28.960 --> 14:36.600]  elsewhere. So that's the dynamic that DoD is used to having. But bringing AI into the picture
[14:37.240 --> 14:42.860]  is a big change because obviously AI is something that has many uses outside of
[14:42.860 --> 14:47.600]  just military applications. And because of that, tech sector is already moving forward
[14:48.400 --> 14:56.120]  rapidly with research and development. So the graph here is one where we kind of try to get
[14:56.120 --> 15:05.120]  at this snapshot of the difference between U.S. government and tech sector spending here,
[15:05.120 --> 15:17.360]  which is that you can see the lower bars in the graph are the budget that U.S. government overall,
[15:17.360 --> 15:24.840]  information technology research and development, including AI. And the taller light purple bars
[15:24.840 --> 15:32.680]  are we just took just five top tech companies and it is their summed research and development
[15:32.680 --> 15:38.780]  investments. And so you can see that just looking at that snapshot, DoD and the U.S. government
[15:38.780 --> 15:43.120]  overall are going to be dwarfed by this type of research and development spending. And so
[15:43.120 --> 15:48.740]  this just illustrates that DoD has a real dependency on what's happening in the private
[15:48.740 --> 15:57.420]  sector, what's happening with external research and development in AI. All right, next slide, please.
[15:59.380 --> 16:06.980]  Okay, so now let's talk about risks. Actually, let's try, hopefully this will work, to see if
[16:06.980 --> 16:11.460]  you guys can, this is one of the areas where we really want to try to have as much of a dialogue
[16:11.460 --> 16:18.180]  as we can with you, the listeners. So I'm going to talk through sort of the landscape of risks.
[16:18.180 --> 16:24.280]  As I'm doing that, we would love it if you would type into the Discord chat, what are your biggest
[16:24.280 --> 16:29.660]  concerns here with regard to military applications of AI, right? Or what are questions or thoughts
[16:29.660 --> 16:36.300]  that you have on this? And after I talk through some of these risks, we'll see if Larry can read
[16:36.300 --> 16:44.940]  off some of what you all wrote in. Okay, so AI, also thinking of AI potential use in lethal
[16:44.940 --> 16:53.500]  autonomy, these are really controversial ideas in the public sphere. I don't know if
[16:53.500 --> 16:59.220]  how many of you were at DEFCON and came to Ethics Village sessions last year, you may have seen
[16:59.780 --> 17:05.380]  a talk given by a group that's a proponent of preemptively banning lethal autonomous
[17:05.380 --> 17:14.420]  weapon systems. So there's a real kind of an active debate and dialogue about some of
[17:14.420 --> 17:21.200]  these questions. And so what are the concerns that people are voicing here? All right, well,
[17:21.200 --> 17:26.160]  one concern is that AI will destroy the world. So that's one that we've seen, for example, Elon Musk
[17:27.420 --> 17:32.320]  say quite publicly, even as I'll mention, even as, of course, Elon Musk has simultaneously
[17:32.320 --> 17:41.060]  been investing in AI research. But AI and lethal autonomy are inherently unlawful or unethical.
[17:41.060 --> 17:47.420]  This is an assumption that is made by the proponents of banning these types of technologies.
[17:48.160 --> 17:54.920]  There are questions about accountability, concerns about discrimination, which means the ability of
[17:55.600 --> 18:04.140]  a weapon system to tell a combatant from a civilian. Of course, biases are also
[18:04.880 --> 18:12.740]  an important challenge and concern when it comes to AI. Let's go to the next slide.
[18:15.940 --> 18:27.040]  So embedded in these concerns are some real and critical challenges that are actually going to be
[18:27.040 --> 18:31.260]  they appear they're going to be surmountable, but they're important challenges to think about.
[18:31.260 --> 18:36.720]  But also embedded in these are some real misconceptions. Okay, so just to talk through
[18:36.720 --> 18:45.160]  some of this, AI will destroy the world. Okay, so that concern is really rooted in
[18:45.760 --> 18:50.720]  worries about artificial general intelligence. And that's not what we're talking about here.
[18:51.340 --> 19:00.120]  But what about questions of lawfulness, ethics, and accountability? So those next two points.
[19:01.080 --> 19:13.060]  So these concerns are rooted in a misunderstanding of how DoD operates. So DoD follows many domestic
[19:13.060 --> 19:19.820]  and international laws, follows many ethical principles. And these laws and principles
[19:20.880 --> 19:27.300]  don't depend on the type of technology that you're using. So the laws of war apply no matter
[19:27.300 --> 19:34.360]  what weapon you're using. So similarly, accountability. Accountability is also
[19:34.360 --> 19:42.140]  baked into how DoD operates. From acquisition, test and evaluation standards, to the chain of
[19:42.140 --> 19:48.620]  command. And also, this doesn't depend on the type of technology that you're talking about.
[19:50.180 --> 19:57.440]  Although certainly things like test and evaluation specific processes will have to be updated
[19:58.440 --> 20:03.320]  for new technologies as they come through, including for artificial intelligence.
[20:04.460 --> 20:11.340]  So that said, there definitely, not to discount, there will absolutely be challenges
[20:11.340 --> 20:19.300]  in developing military applications of AI. So in particular, you want to make sure that it really
[20:19.300 --> 20:27.280]  works. So integrated into that is making sure that systems can be discriminant, that systems
[20:27.280 --> 20:32.420]  will, or use of systems will address bias. Basically, you want to make sure that systems
[20:32.420 --> 20:41.040]  will operate as intended in any other ways. And so just a couple quick comments here.
[20:41.040 --> 20:47.380]  We say that these concerns are surmountable. So some of the way to think about that here
[20:47.380 --> 20:56.500]  is that the goal is not necessarily to, say, develop some kind of AI-enabled weapon system
[20:56.500 --> 21:02.840]  that is able to discriminate civilians from combatants in all possible situations.
[21:03.260 --> 21:10.180]  So the point is that you would only approve a system for use in settings in which you knew
[21:10.180 --> 21:15.040]  it was going to be able to function properly. So that's kind of the aim and the goal here.
[21:15.040 --> 21:20.280]  This isn't going to be something that works in all settings. Of course, that's true of
[21:20.920 --> 21:27.180]  legacy non-AI-enabled systems and weapons as well.
[21:28.480 --> 21:35.740]  Also, we're not necessarily aiming for, say, systems that are learning in real time,
[21:35.740 --> 21:40.540]  especially weapon systems. So it's more likely to be thinking about at this stage,
[21:40.540 --> 21:46.560]  right, maybe systems that will be collecting data in real time, but that maybe it would not
[21:46.560 --> 21:52.540]  go ahead and update how it's making decisions until those updates can be validated.
[21:53.000 --> 21:56.960]  So keep some of those important caveats in mind here.
[21:58.720 --> 22:04.060]  Larry, did anybody write anything good in the chat about concerns and risks?
[22:06.360 --> 22:16.360]  Yes, definitely. So a few things here are... there's a link to a report that is a really
[22:16.360 --> 22:25.340]  good report from early 2019. And it talks about different risks of AI. And I think it's actually
[22:26.620 --> 22:33.560]  good to kind of frame this. So we are talking specifically about military applications of AI.
[22:33.560 --> 22:42.140]  So the report that we do recommend you look at talks about broader risks of AI. And it
[22:42.140 --> 22:47.760]  talks about three different kinds of risks. One is digital security. So digital security,
[22:47.760 --> 22:54.020]  physical security, political security. So digital security goes to cyber attacks and the kinds of
[22:54.020 --> 23:01.040]  infiltration I can do. Physical security, they give the example of the potential for using an
[23:01.040 --> 23:09.420]  autonomous self-driving car as a weapon. So like a terrorist attack that you can use that.
[23:09.420 --> 23:16.700]  And then third, they talk about political security. And that includes misuse of surveillance,
[23:16.700 --> 23:24.660]  propaganda, misinformation. And these are all really great points. So the fact that we don't
[23:24.660 --> 23:30.480]  have those on there is because we're kind of talking about the more narrow military applications
[23:30.480 --> 23:35.320]  of AI. But it's important to acknowledge that there are broader risks as well.
[23:35.320 --> 23:40.120]  Larry, what is that? What report is it that you're referring to? Or who made it?
[23:40.140 --> 23:50.460]  Yeah, there's a link. And see, actually, who did it? I've looked at the link at this before.
[23:50.460 --> 23:58.480]  There are many authors from different organizations. So University of Oxford,
[23:58.480 --> 24:07.520]  Future of Humanity Institute, CNAS was also a contributor, OpenAI. So it's a multi-author,
[24:07.520 --> 24:14.580]  multi-organization report. But the link is in the general text channel. So definitely,
[24:14.580 --> 24:25.260]  it's good to check out. Yeah. So another comment is about bias and how bias seems like a really
[24:25.260 --> 24:32.680]  big deal, which is it absolutely is. And I guess one comment to make about that is that
[24:32.680 --> 24:40.880]  humans are biased, right? Humans have lots of cognitive biases. And so AI applications that
[24:40.880 --> 24:47.400]  are built by humans that leverage data that are developed by humans should also be expected to
[24:47.400 --> 24:52.800]  be biased. So that is absolutely true. The one thing to me that's a key difference is that
[24:52.800 --> 25:00.620]  with AI applications, you can explicitly examine those biases and try to correct for them. So
[25:00.620 --> 25:08.520]  that's actually a valuable thing that we can do. However, too often, we don't do that. So we need
[25:08.520 --> 25:16.460]  to be very deliberate about trying to characterize and correct those biases. Let's see. Another
[25:16.460 --> 25:22.180]  comment is majority of DoD spending on AI goes to warfighting and lethality,
[25:22.180 --> 25:28.940]  not logistics, humanitarian assistance, predictive maintenance, logistics. Thank you for that
[25:28.940 --> 25:34.720]  comment. We're actually going to... that is a big topic of our talk. So please keep on
[25:35.620 --> 25:43.800]  listening and participating. Do AI weapon systems fall under the Geneva Convention?
[25:43.800 --> 25:48.760]  Um, so if it's used for armed conflict, then it should.
[25:49.020 --> 25:53.060]  And I think you're going to you're going to continue to talk about this,
[25:53.640 --> 26:01.180]  kind of the legal aspect of this. bleed over from DoD and the intelligence community.
[26:02.060 --> 26:09.920]  And that's, that's also it. That is a tricky issue today, because because of the
[26:10.280 --> 26:20.980]  um, like, we have some some US activities that do kind of combine the DoD and intelligence
[26:20.980 --> 26:26.720]  community. So that is something that kind of needs to be watched. And then how do we ethically
[26:26.720 --> 26:35.240]  import bias into AI? And again, I think, going back to the earlier comment about bias, bias is
[26:35.240 --> 26:41.980]  really a significant issue we need to be careful about. And I think, you know, yes, the Microsoft
[26:41.980 --> 26:50.260]  chatbot is a great example. And how, again, we can't just do something and then put it out in
[26:50.260 --> 26:55.940]  the world and then assume it's going to work well, we need to monitor it, we need to be explicitly
[26:56.920 --> 27:05.240]  monitoring for bias and intervening as well. So that kind of gets it what I said a minute ago
[27:05.240 --> 27:09.020]  about making sure to validate as as you're learning, right, we're gonna we're gonna have
[27:09.020 --> 27:18.270]  to validate that learning. So great comments. Okay, great. Well, thank you guys all and keep
[27:18.270 --> 27:25.150]  them coming. Um, we will we will come back to looking at the at the comments in a little bit.
[27:25.750 --> 27:32.170]  I'll go go ahead for the next slide, please. Okay, I'm gonna say a few words now. Okay, right. So,
[27:32.170 --> 27:40.090]  so there are misconceptions out there, but also a lot of very real challenges and risks. And DOD
[27:40.090 --> 27:48.310]  is doing some things now that address some of these risks. And so any, any, any discussion of
[27:48.310 --> 27:54.410]  this topic, right needs to have have this information as part of the context. So to run
[27:54.410 --> 27:58.630]  through some of that, right, some some of the things that DOD does that address these risks
[27:58.630 --> 28:03.570]  are just general DOD practices, and some of them are things specifically being developed for
[28:03.570 --> 28:09.570]  AI in a related context. So, so first, I'll talk about the general practices. Okay, so,
[28:09.570 --> 28:16.790]  so legal considerations, right. So, so, to kind of reiterate something I mentioned before,
[28:16.790 --> 28:23.670]  DOD incorporates legal considerations into, you know, pretty much everything it does development
[28:23.670 --> 28:31.490]  certification, use of any systems. So, so, and these laws, you know, laws of war, they apply no
[28:31.490 --> 28:36.810]  matter what type of technology you're using, right, things like proportionality, that, that,
[28:36.810 --> 28:40.510]  that's a principle that does not depend on the weapon that you're using. And so,
[28:40.510 --> 28:48.170]  so these things generally apply independent of the technology. Practices also help to
[28:48.810 --> 28:53.850]  mitigate some of these risks, right. So test and evaluation processes need to make sure that
[28:53.850 --> 28:59.230]  systems function as intended. Whenever you have a new system, also training will be established
[28:59.230 --> 29:05.510]  for that and operating procedures for using any kind of system, especially weapon systems.
[29:06.650 --> 29:13.970]  Policy is also a really important part of how DOD operates. So one example I'll give here
[29:14.470 --> 29:21.270]  is a 2016 executive order that's focused on minimizing civilian casualty.
[29:21.690 --> 29:27.710]  So this executive order says a lot of really great things. And I want to just highlight one,
[29:27.710 --> 29:32.750]  one piece of it, which I think is especially interesting in this context. So, so the, the EO
[29:32.750 --> 29:39.510]  says that the U.S. is going to develop, acquire, and field weapon systems and other technological
[29:39.510 --> 29:46.270]  capabilities that further enable the discriminant use of force in different operational contexts.
[29:46.670 --> 29:51.770]  So what are we saying there, right? We're saying that if there's technology that comes along that
[29:51.770 --> 29:55.810]  will allow us to be more discriminant in war fighting, you know, we're going to look at using
[29:55.810 --> 30:01.210]  that. And so, so keep that in mind because what Larry's going to talk about in a, in a minute or
[30:01.210 --> 30:07.190]  two here is, is the, you know, the very real possibility that AI can, can help us be more
[30:07.190 --> 30:15.970]  discriminant in war fighting. See, please go to the next slide. So now I'll talk just briefly
[30:15.970 --> 30:23.970]  about some AI specific steps that the DoD is taking. I mentioned the AI strategy earlier. DoD
[30:23.970 --> 30:31.570]  also very recently adopted some ethics, AI ethics principles. These came from the work of the
[30:31.570 --> 30:38.790]  Defense Innovation Board, right? And so these are, these are principles that, that include,
[30:38.790 --> 30:44.270]  you know, appropriate levels of human judgment, the intent that that will allow it if they exhibit
[30:44.270 --> 30:50.210]  unintended escalatory behavior. So this is, this is really the intent of DoD moving forward.
[30:50.910 --> 30:58.250]  So another, another related policy dimension, right, is a DoD has a directive on lethal
[30:58.250 --> 31:01.860]  autonomous weapon systems, which was actually from all the way back in 2012,
[31:02.930 --> 31:07.830]  which again says that appropriate levels of human judgment will always be used when it comes to
[31:07.830 --> 31:15.370]  lethal autonomy. And, and it also puts out that in many cases, a lethal autonomous weapon system
[31:15.370 --> 31:20.210]  would have to undergo a senior level review before it would ever be used by DoD forces.
[31:20.570 --> 31:25.890]  Test and evaluation requirements are being looked at, right? So all of these steps are good and they
[31:25.890 --> 31:31.890]  do start to get at a lot of the risks that we talked about. But as, as Larry alluded to,
[31:31.890 --> 31:37.870]  with talking about bias a minute ago, right, we, we would put out there that more is needed here.
[31:37.870 --> 31:44.350]  And in particular, what we don't see right now and really, really is necessary is a comprehensive
[31:44.350 --> 31:51.550]  approach to identifying and systematically addressing risks. So I've, I've talked here
[31:51.550 --> 31:58.190]  about how AI represents a new dynamic between DoD and the tech sector, and how there are a lot
[31:58.190 --> 32:03.190]  of challenges out there. And we're already working to address some of those challenges.
[32:03.610 --> 32:08.010]  I want to turn things over now to Larry, and we'll hear a little bit about the opportunities.
[32:11.910 --> 32:14.250]  Great. So if you can go to the next slide, please.
[32:15.650 --> 32:20.230]  Okay. So we've heard about that there are risks of military use of AI,
[32:20.230 --> 32:26.750]  how those risks can be different than those that are sometimes commonly discussed, and that we need
[32:26.750 --> 32:31.950]  to be more deliberate in addressing them. At the same time, we'll highlight that there are
[32:31.950 --> 32:39.530]  opportunities. I'll talk about three things. One is there is potential for good with the use of
[32:39.530 --> 32:46.370]  technology. AI and autonomous systems can be used deliberately for humanitarian benefits.
[32:46.370 --> 32:53.090]  Second of all, history shows that outside influence can help, can help the U.S. military
[32:53.090 --> 33:02.590]  work harder on focusing on humanitarian issues. And finally, times, things are, times are changing,
[33:02.590 --> 33:08.230]  and there are new opportunities where the tech sector can have an influence on national security
[33:08.230 --> 33:15.130]  policy that it has not had in the past. So if you can go to the next slide, please.
[33:17.830 --> 33:27.110]  Okay. So, so an argument is often made how advanced technology has humanitarian benefits.
[33:27.110 --> 33:31.870]  So an example that I hear a lot is, well, we have precision-guided munitions,
[33:31.870 --> 33:38.170]  they're much better for civilians on the battlefield than dumb bombs. Okay. And that is,
[33:38.170 --> 33:44.810]  that is generally true. At the same time, that what that really is, is a humanitarian side benefit
[33:44.810 --> 33:52.810]  of a military aim, right? So precision-guided munitions were not created to help innocent
[33:52.810 --> 33:57.550]  civilians. They were created to help the military to hit targets more effectively,
[33:58.090 --> 34:04.710]  and they happen to have a humanitarian side benefit. But technology can also be used
[34:04.710 --> 34:11.150]  deliberately to help address pressing humanitarian challenges and war. And some of those
[34:11.150 --> 34:18.130]  opportunities include reducing civilian casualties, you know, avoiding tragic attacks
[34:18.130 --> 34:25.410]  on hospitals and other protected sites, and also deconfliction of humanitarian aid on the battlefield
[34:25.410 --> 34:33.630]  so that those, you know, so that World Food Program trucks aren't targeted in military attacks.
[34:33.630 --> 34:44.830]  Next slide. So here's an example. So protecting hospitals in conflict. Medical care in conflict
[34:44.830 --> 34:52.310]  was the very first issue taking up as a measure of international humanitarian law. We talked about
[34:52.310 --> 34:59.210]  the Geneva Conventions, and there are other legal agreements that governments have agreed to over
[34:59.210 --> 35:05.290]  the years. Medical care in conflict was the very first issue taken up. And of course, the origin of
[35:05.290 --> 35:12.410]  ICRC was centered around providing medical care in conflict, making sure it was not interrupted
[35:13.210 --> 35:22.230]  by a war. So it was the first thing, and yet attacks on hospitals continue 150 years later
[35:22.230 --> 35:29.490]  after ICRC was created, after the first international law was created to help
[35:30.310 --> 35:38.510]  conflict be more humane. The picture here is an MSF hospital in Yemen that was destroyed by an
[35:38.510 --> 35:46.650]  airstrike by the Saudi-led coalition in August 2016. This kind of thing, this kind of attack,
[35:46.650 --> 35:52.370]  happens too many times. And you see the international community trying to do things.
[35:52.370 --> 35:57.310]  I think of the United Nations, they had a Security Council Resolution 2286
[35:58.250 --> 36:05.870]  that calls the international community to action. We need to do more. And yet nothing
[36:05.870 --> 36:11.190]  practical has been done. So we have a really pressing problem with protecting hospitals.
[36:11.790 --> 36:14.190]  So to go to the next slide.
[36:16.070 --> 36:25.490]  There are things we can do. So we're a research organization. And in my grad school,
[36:25.490 --> 36:33.890]  I have a PhD in physical chemistry. So I had a laboratory with a laser and with
[36:34.550 --> 36:39.610]  liquid nitrogen kind of fuming to make things really cold and a vacuum chamber, and it smelled
[36:39.610 --> 36:47.810]  horrible. And that was kind of the environment that I worked in. Now at CNA, the world is our
[36:47.810 --> 36:57.410]  laboratory. So we go out and we go to places like Yemen and Afghanistan and Iraq and Syria,
[36:57.410 --> 37:03.370]  and we try to get the data to help us to better understand these kinds of problems.
[37:03.960 --> 37:10.230]  So one of the things I did is look at attacks on hospitals and looking at the data and say,
[37:10.230 --> 37:15.590]  okay, why do these things happen? What goes wrong when they're attacked? And our analysis shows that
[37:15.590 --> 37:21.790]  there are two kinds of failures that both have to happen simultaneously for a hospital to be
[37:21.790 --> 37:29.790]  attacked. One is a failure in deconfliction. And that is, if you look at modern militaries,
[37:29.790 --> 37:35.330]  they try to maintain a list of locations that are supposed to be protected and not attacked,
[37:35.330 --> 37:42.630]  and they call this a no-strike list. So they're supposed to be looking at lists when they are
[37:42.630 --> 37:49.570]  about to attack someone. And an attack on a medical facility means that this deconfliction step
[37:49.570 --> 37:55.390]  has failed. But that's not the only thing they have to do. They're also supposed to try to
[37:55.770 --> 38:01.790]  and avoid attacks on medical facilities. And one way is that, again, through the Geneva
[38:01.790 --> 38:09.690]  Conventions, identification of hospitals can be facilitated by the hospitals displaying a
[38:09.690 --> 38:16.870]  Red Cross or a Red Crescent. So an attack means that that identification process has also failed.
[38:16.870 --> 38:23.310]  So if you can fix either of these problems, you can stop the attack. And it'd be nice to work on
[38:23.310 --> 38:27.650]  both, but if you can make one of these things successful instead of a failure,
[38:27.650 --> 38:33.980]  you prevented a tragic attack on a hospital. So if you go to the next slide...
[38:36.670 --> 38:44.870]  technology can help reduce attacks on hospitals in various ways. So one example is Australia.
[38:44.870 --> 38:50.290]  They're using machine learning right now. They're exploring machine learning applications to help
[38:50.290 --> 38:55.650]  automatic recognition of protected symbols, like the Red Cross and the Red Crescent.
[38:56.090 --> 39:02.490]  So it doesn't automate the attack. Basically, it's a safety net. So if you're looking to make
[39:02.490 --> 39:07.870]  an attack and the machine learning algorithm senses that there's a Red Cross or a Red Crescent
[39:07.870 --> 39:20.190]  in the target area, it says, hey, stop. You need to stop. There are also opportunities
[39:20.190 --> 39:26.410]  to use technology to strengthen the deconfliction process. For example, White Flag is being
[39:26.410 --> 39:31.710]  developed. It's a protocol developed by the Netherlands. And that is an encrypted technology
[39:31.710 --> 39:36.790]  to strengthen this deconfliction process. And that's really important because what we see
[39:37.630 --> 39:44.010]  as we look at the data on this is that often the information that would stop an attack is
[39:44.010 --> 39:49.130]  known somewhere, but there is an information flow problem and it's not reaching the person
[39:49.130 --> 39:53.370]  that is making the decision whether to attack or not. So if we can improve that,
[39:53.370 --> 39:59.750]  then we can also help reduce attacks on hospitals. And the protection of hospitals
[39:59.750 --> 40:04.850]  is just one area. It's an important area, but just one area where technology can be used
[40:04.850 --> 40:11.410]  specifically for humanitarian benefit. We've also looked at how to reduce civilian casualties,
[40:11.410 --> 40:16.050]  how to detect civilian casualties, which is also an issue. Oftentimes you'll hear
[40:17.230 --> 40:22.350]  you'll hear militaries say, you know, they did a strike. There are accusations about civilian
[40:22.350 --> 40:28.650]  casualties and they say, nope, we didn't do it. And what the data says is oftentimes they did,
[40:28.650 --> 40:35.570]  but they're just not very good at detecting civilian casualties. So, and if you can't detect
[40:35.570 --> 40:40.530]  them, then you don't know you have a problem to solve. So this is also a really important issue
[40:40.530 --> 40:46.050]  that can be helped by technology. There's also anticipating and avoiding reverberating
[40:46.050 --> 40:51.350]  long-term effects of attacks. So essential services, that's something that militaries
[40:51.350 --> 40:56.230]  around the world really do poorly. And that could also be helped. And then there's also
[40:56.230 --> 41:03.210]  traceability and accountability for violations. The New York Times had a really powerful article
[41:03.210 --> 41:10.600]  on how Russia and the Syrian regime were specifically targeting hospitals in Syria.
[41:10.930 --> 41:18.370]  And we can potentially use technology to not only, not only avoid things, but also provide
[41:18.370 --> 41:23.730]  more accountability, provide basically the proof that says, Hey, this is the smoking gun.
[41:23.730 --> 41:31.190]  You guys did this and you can't just argue, you know, fog of war. So these are just some
[41:31.190 --> 41:36.870]  examples of how technology can be used for, for good and for humanitarian benefits.
[41:37.270 --> 41:38.730]  So the next slide, please.
[41:39.130 --> 41:44.770]  Hey, Larry, can I jump in with a good point that somebody made in one of the chats about what
[41:44.770 --> 41:48.350]  you're talking about here, which is that, you know, if you have these, these algorithms that
[41:48.350 --> 41:55.530]  are designed to detect red crosses and then not shoot at that thing, right? The question is,
[41:55.530 --> 41:59.590]  well, doesn't that mean that all of the bad guys are just going to start putting red crosses on,
[41:59.590 --> 42:01.530]  on their, on their stuff?
[42:03.790 --> 42:11.930]  Sure. And I mean, yeah. So, so there, there, there are states and actors that try to follow
[42:11.930 --> 42:17.110]  the law, the law of war, internationally humanitarian law, like the genomic conventions.
[42:17.110 --> 42:21.290]  And then there are some that don't. That's actually why I included that last piece there,
[42:21.290 --> 42:28.230]  traceability and accountability. Because with, with, you know, with technology,
[42:28.230 --> 42:35.690]  you can also better document how actors, and that may be a state like Russia, or it may be
[42:35.890 --> 42:42.190]  a non-state armed group, but if they are intentionally flouting requirements of the law,
[42:42.190 --> 42:49.670]  you now have documented this and you can put more pressure on them to, to behave differently.
[42:51.910 --> 42:57.090]  And try to use different ways to, to, to positively influence them.
[42:57.090 --> 43:00.610]  Yeah. And, and one of the things I think you're getting at when you're talking about flouting the
[43:00.610 --> 43:07.750]  law is that it is, goes against international humanitarian law to put those red cross symbols,
[43:07.750 --> 43:11.710]  red crescent symbols on something that is not actually a protected site. So,
[43:11.710 --> 43:17.290]  so some bad actors, right, would be doing that even, even without AI technology.
[43:17.290 --> 43:23.410]  Right. Right. Yeah, certainly, certainly they can, they can do that now. And sometimes they do.
[43:23.950 --> 43:32.950]  That is, that is not, that is not an excuse to, to just be reckless. Right. So, so we need,
[43:32.950 --> 43:37.870]  still need to, to have, to strengthen our protection. But at the same time,
[43:37.870 --> 43:43.510]  this is also an opportunity to avoid the bad behavior that we see today.
[43:46.410 --> 43:53.950]  Okay. So finally, I mean, so the military cares about humanitarian issues. But at the same time,
[43:53.950 --> 43:58.630]  there's, there's not a lot of, of strong leadership on this issue. And that's,
[43:58.630 --> 44:03.950]  that's really because of a combination of formal responsibilities. If you look
[44:03.950 --> 44:09.410]  organizationally at DOD, you know, they have, they have people in charge of lots of different
[44:09.410 --> 44:14.810]  things, you know, developing weapons systems, developing, you know, lethal capabilities.
[44:15.070 --> 44:20.550]  There's really not that same kind of leadership on, hey, how can we, how can we get better at
[44:20.550 --> 44:28.270]  humanitarian issues? At the same time, there, there, there has been, there have been leaders
[44:28.270 --> 44:32.850]  who have really focused on this issue, but then they rotate out. And so it does,
[44:32.850 --> 44:41.970]  long lasting focus. So, so what, what can we do? History shows that non-governmental influences can
[44:41.970 --> 44:48.170]  strongly encourage progress in this area. And typically that has been through media,
[44:48.170 --> 44:52.710]  you know, media accounts. And I think of the uncounted article, for example, in 2017,
[44:53.170 --> 44:59.850]  by Asma Khan was one great example of this, how one powerful, well-researched article
[44:59.850 --> 45:07.230]  basically influenced DOD to, to be more introspective and to do kind of internal
[45:07.230 --> 45:14.330]  reviews that actually make real changes. And civil society, you know, NGOs, academia can
[45:14.330 --> 45:21.470]  have a role. And then of course the UN and ICRC. So they have all, at different times,
[45:21.470 --> 45:28.430]  had important contributions to help the military to do better. And I listed,
[45:29.330 --> 45:35.090]  listed three specific examples. You know, there are, these are actually examples, real examples
[45:35.090 --> 45:43.950]  where outside influences encourage the military to try harder, both in Afghanistan and that really,
[45:43.950 --> 45:47.770]  they really turned things around and did things differently. And the civilian casualty numbers
[45:47.770 --> 45:54.490]  went down. Drone strikes and counter-terrorism, you know, leading to the presidential policy
[45:54.490 --> 46:00.390]  directive, the presidential policy guidance issued by President Obama for counter-terrorism
[46:00.390 --> 46:05.590]  operations. And then the campaign in Iraq and Syria, again, there were things like,
[46:05.590 --> 46:12.530]  you know, counted article and also air wars that contributed to greater scrutiny and led to
[46:12.530 --> 46:17.590]  differences in how the U.S. military has done things. If you go to the next slide.
[46:18.830 --> 46:26.550]  So, so bottom line, there, there's a, there's an opportunity here. So if you combine kind of
[46:26.550 --> 46:32.790]  the things we've been talking about, the community needs the tech sector as they have not before.
[46:32.790 --> 46:37.770]  So, you know, looking at that graph we had earlier, you know, the, just the top five
[46:39.050 --> 46:45.910]  tech companies in the U.S., their research and development, you know, now 15 times larger than
[46:45.910 --> 46:52.550]  the entire tech development budget of the United States government. And that ratio is increasing
[46:52.550 --> 47:02.290]  over time. So the U.S. government sees it needs the tech sector to be competitive. And you see
[47:02.290 --> 47:06.450]  this in DoD wanting to work with the tech sector and they have a number of initiatives. There's
[47:06.450 --> 47:11.330]  the defense innovation board. And of course they developed the AI principles that DoD then
[47:11.890 --> 47:16.830]  accepted. You have the defense innovation unit, defense digital service, and then there are other
[47:16.830 --> 47:24.350]  initiatives that DoD is trying to engage more with the tech sector. So we just wanted to highlight
[47:24.350 --> 47:30.070]  this is an opportunity for the tech sector, right? So this is an opportunity to engage and have
[47:30.070 --> 47:37.790]  influence. So as DoD says, Hey, we want to talk, we want to work together. There are opportunities
[47:37.790 --> 47:43.950]  for tech companies and people working in the tech sector to influence the areas for engagement and
[47:43.950 --> 47:52.830]  for cooperation. And also voice concerns about how these issues can be addressed. And then,
[47:52.830 --> 47:58.590]  you know, I mentioned, you know, right now there's academia, there's a civil society,
[47:58.590 --> 48:05.850]  there's a UN that are engaging now with DoD on discussions and humanitarian issues. I was in
[48:05.850 --> 48:11.550]  such a discussion yesterday, right? There was nobody in the tech sector in that discussion,
[48:11.550 --> 48:19.670]  but there could be, right? As a stakeholder and as a potential solution. So this is an opportunity.
[48:19.670 --> 48:24.330]  And then I also just wanted to highlight, this is not just for the US, right? So I mentioned
[48:24.330 --> 48:32.890]  Australia doing work. Also the Netherlands are working on this protocol, white flag,
[48:32.890 --> 48:39.810]  white flag is an open source solution. So it's also, it doesn't have to be a proprietary solution
[48:39.810 --> 48:44.930]  for the US or it could be other open source solutions that help address humanitarian
[48:44.930 --> 48:54.490]  issues that basically could be offered to all parties to conflict. So I think that's,
[48:54.490 --> 49:00.810]  if you go to the next slide. So now we'd love to have a discussion and we also have
[49:00.810 --> 49:12.110]  contact information here. So we'd love to continue to answer questions or things you'd
[49:12.110 --> 49:17.390]  like to raise, but also in the future, we'd love to hear from you and we can continue those
[49:17.390 --> 49:24.550]  discussions. So open to any questions you may have. Yeah, absolutely. I'll certainly echo what Larry
[49:24.550 --> 49:30.290]  has said there. And so I've been reading through the comments that people have been posting.
[49:30.290 --> 49:36.570]  So I'm just going to go ahead and kind of read off the things that I've seen there.
[49:37.050 --> 49:42.650]  So one of the things that kind of the main thing that a couple of different folks have raised
[49:42.650 --> 49:50.330]  are concerns around proliferation, right? So one person raised a worry, and this is very timely,
[49:50.530 --> 49:55.990]  a worry about the trickle down of military technology being used in domestic law enforcement,
[49:55.990 --> 50:03.310]  right? Or being used abroad to stifle dissent. There was also some concerns brought up about,
[50:03.310 --> 50:09.870]  well, if other, say, bad actor nations get these technologies, that's something that's going to
[50:09.870 --> 50:15.790]  serve as a force multiplier for those bad bad actors, dictators or whatever. And so
[50:17.730 --> 50:24.170]  a mention of, let's see, so the person wrote in that the best way to keep control of technology
[50:24.670 --> 50:29.970]  is to not develop it. And there's another mention of mutually assured destruction.
[50:30.970 --> 50:38.270]  So there's kind of a lot baked up in those proliferation concerns. I might start to
[50:38.270 --> 50:44.430]  address those by saying that certainly thinking about this technology and how we want to move
[50:44.430 --> 50:52.350]  forward with thinking about exporting or sharing it is absolutely something that DOD and the U.S.
[50:52.350 --> 50:56.970]  government is taking very seriously and really thinking about right now.
[50:58.090 --> 51:02.190]  Another thing I'll... and so, again, it's going to be one of these contexts where
[51:02.830 --> 51:08.990]  there's some hard challenges, you know, wrapped up in there. But at the same time, right,
[51:08.990 --> 51:14.770]  keep in mind that, you know, as we talked about with that earlier slide with the graph on it,
[51:14.770 --> 51:21.570]  most of the R&D that's going on in this space is not happening by state actors. It's not happening
[51:21.570 --> 51:29.590]  by the U.S. government or DOD. So that's a reality to keep in mind, thinking about proliferation
[51:29.590 --> 51:37.110]  also. And one other thing I'll mention, just thinking about the domestic law enforcement
[51:37.650 --> 51:44.290]  context, is just be aware that a lot of the rules and laws that DOD has to follow
[51:45.350 --> 51:53.270]  and what it makes are... so domestic law enforcement is not subject to those laws.
[51:53.270 --> 51:57.790]  So, again, I think there are things to worry about here, but I'm not sure if
[51:57.790 --> 52:03.650]  DOD's development of these technologies... that may not be the biggest concern in some
[52:03.650 --> 52:08.970]  of these cases. Larry, would you add anything on any of those topics?
[52:10.350 --> 52:17.470]  So, yeah, so great questions. One thing I wanted to mention, especially for the
[52:17.470 --> 52:24.810]  export and proliferation piece, because I've spent a couple years at state
[52:24.810 --> 52:31.090]  where this was a really key concern. And one thing that became very apparent to me,
[52:31.090 --> 52:38.690]  in the case of Yemen, so, you know, the U.S. has provided a lot of arms, you know,
[52:38.690 --> 52:43.250]  weapons systems, intelligence to the Saudi-led coalition in Yemen. And,
[52:44.530 --> 52:50.210]  you know, there were lots of discussions about, is it legal to do that? And I think
[52:50.210 --> 52:58.190]  I saw one report kind of summarized it as lawful, but awful. So the laws and policies
[52:58.190 --> 53:06.630]  that the U.S. applies to security assistance, providing stuff to other countries,
[53:06.630 --> 53:12.350]  it considers some things. It considers things like diversion, which is, are you going to give,
[53:12.350 --> 53:16.830]  we want to safeguard that if we give this, if the U.S. gives this to another country,
[53:16.830 --> 53:21.990]  they won't give it to somebody else. And there's also something called Leahy law that guards
[53:21.990 --> 53:26.430]  against gross violations of human rights. So if they're doing like war crimes, really horrible
[53:26.430 --> 53:33.290]  things, then that can stop transfers. But the more general civilian protection considerations
[53:33.290 --> 53:38.570]  that we're talking about really are not well incorporated in security assistance. And so
[53:39.290 --> 53:45.090]  that is something that the state department is really trying to think about doing. We're talking
[53:45.090 --> 53:53.410]  to them about, you know, new policies and new ways to think about this. But one thing that can help
[53:53.410 --> 54:02.810]  is if you can provide equipment, including weapons, that have some of these safeguards,
[54:02.810 --> 54:09.010]  right? So right now, the Saudi-led coalition, how many hospitals have they struck? I've lost
[54:09.010 --> 54:16.310]  count. It's so many, so many, right? And I mean, you can certainly say, well, we're not going to
[54:16.310 --> 54:22.150]  give them anything, right? But they've also shown that they'll just buy from the Russians.
[54:22.150 --> 54:32.930]  So it'd be great to stop them, okay? I'm in favor of that. But if war is going to continue,
[54:32.930 --> 54:41.790]  are there things that can be done to make war more humane and safer for the civilians that are in the
[54:41.790 --> 54:51.370]  conflict areas? And so the answer is yes, and that technology can help. And that can certainly be the
[54:51.370 --> 55:00.010]  case for exports as well. And as I'm reading the chats coming in, I'll just say that folks are
[55:00.010 --> 55:06.430]  kind of doubling down on concerns about domestic law enforcement again, right? And so predator
[55:06.430 --> 55:15.370]  drones at protests, you know, an interest in, you know, can we have voting where we select that
[55:15.370 --> 55:22.030]  we're gonna not fund these types of technologies? Maybe that's in a military or in a law enforcement
[55:22.030 --> 55:31.070]  context. So let's see, and a note that killer robots are perfect for war crimes. So back to
[55:31.070 --> 55:36.110]  the armed conflict case. So I'll say again, with respect to the domestic law enforcement front,
[55:36.110 --> 55:43.150]  right, we're just seeing that, you know, some of this is not, maybe not necessarily an issue
[55:43.150 --> 55:50.930]  limited to AI developments, right? We're seeing it now. It's a real problem in domestic law
[55:50.930 --> 55:56.630]  enforcement. And so, you know, it's important that we fix the laws and fix the practices that are in
[55:56.630 --> 56:03.230]  place now, even before we have sort of new classes of weapons that the military is using.
[56:03.930 --> 56:13.130]  Um, let's see one, another comment that somebody wrote into the chat was, was concerned about AI
[56:13.130 --> 56:19.430]  vulnerabilities, right, being especially problematic in this context, they mentioned
[56:19.430 --> 56:27.310]  adversarial attacks, data poisoning, right. So these are, so these are getting technical
[56:27.310 --> 56:32.630]  challenges, right, that are going to be hard challenges to to solve, right. So this goes back
[56:32.630 --> 56:37.270]  to, there are a lot of challenges and just getting getting the technology right to actually
[56:37.270 --> 56:45.510]  make these applications work. And so I guess one thing I would say here is is that, that,
[56:45.510 --> 56:51.250]  you know, yeah, these are hard problems. I think we, you know, we, DoD will need the tech sector,
[56:51.250 --> 56:56.130]  private industries help to get good solutions here. And in cases where we can't be confident
[56:56.130 --> 57:04.290]  in these technologies, especially in in lethal, or weapons contexts, those are cases where we need
[57:04.290 --> 57:09.230]  to not not deploy systems, right, unless we can, unless we can be confident with how they're going
[57:09.230 --> 57:19.790]  to operate. Can I make one comment about the, you know, I do hear this a lot about, you know,
[57:19.790 --> 57:26.110]  killer robots are perfect for war crimes, right. And so I think it's useful kind of just to build
[57:26.230 --> 57:32.690]  a framework here. So there are two kinds of parties that are involved in conflict. Some can
[57:32.690 --> 57:41.310]  be states, or it can be armed groups. One is parties that want to follow the law. Okay. And
[57:41.310 --> 57:48.110]  then there are some that don't, they show impunity. And so I think technology applies differently to
[57:48.110 --> 57:56.810]  the two of them. So those that that wants to apply the law, then this kind of technology can
[57:56.810 --> 58:05.030]  be useful for, for helping, helping them to better comply with the law and their legal obligations.
[58:05.030 --> 58:10.710]  But even beyond the law, because in the US, we talked about there are legal things we're required
[58:10.710 --> 58:17.710]  to do. But then we also want to try even harder than what we're required to do, right. The law
[58:17.710 --> 58:24.510]  is just the basic requirements. Diane mentioned the, the US civilian casualty executive order in
[58:24.510 --> 58:30.830]  2016. That goes beyond what is required by the law. It says, you know, there are things we're
[58:30.830 --> 58:37.870]  required to do, but we're going to do more because it's important. So, so for the group of states or
[58:37.870 --> 58:43.690]  parties that want to comply with the law, you can help comply with the law. And then you can even
[58:43.690 --> 58:49.510]  go further. And we talked about some ways to do that. Then there's others that don't comply with
[58:49.510 --> 58:53.730]  the law, right. And there's the Russians and the Syrians. And, you know, there are different,
[58:53.730 --> 58:59.190]  different, you know, different terrorist groups out there. They just don't care. They exploit the
[58:59.190 --> 59:07.310]  law for their own benefits. And in those cases, technology can also be used for accountability
[59:08.110 --> 59:14.590]  and to traceability and to try to highlight some of the things that they do. Honestly,
[59:14.590 --> 59:18.230]  you know, there were, there were things I remember in the counter ISIS
[59:20.430 --> 59:29.650]  conflict in, in Iraq and Syria, ISIS did horrible, horrible things, and most people have no idea.
[59:29.650 --> 59:38.870]  Right. And so, and just better, better showing those things can help in themselves to influence
[59:38.870 --> 59:43.750]  those groups, just like the New York Times article, talking about the Russians and
[59:44.370 --> 59:51.610]  delivery striking hospitals. Okay. They, they did that quietly. When the New York Times came out,
[59:51.610 --> 59:56.510]  Russia started to squirm, right? Because they don't want to be known for that. They want to do
[59:56.510 --> 01:00:02.950]  it for various interests. But when you start putting the spotlight on them, that can help,
[01:00:03.530 --> 01:00:10.050]  that can help influence them in themselves. We should also be thinking about how can we better
[01:00:10.050 --> 01:00:16.050]  use that information to really deter them from doing those terrible things. So just wanted to
[01:00:16.050 --> 01:00:21.770]  kind of give that overall framework for war crimes. So Larry, another comment that somebody
[01:00:21.770 --> 01:00:27.970]  wrote in that I think is a really interesting one is a question of whether, right, so DOD has
[01:00:27.970 --> 01:00:34.190]  these AI policies. Should, should those be codified in law? Because it's, it's really
[01:00:34.190 --> 01:00:49.220]  easy to change norms and guidelines. What would you say to that? So, so I remember actually having
[01:00:49.400 --> 01:00:57.700]  a conversation with Eric Schmidt about, he's part of the, the National Security Commission for AI.
[01:00:57.700 --> 01:01:04.160]  And so there was this discussion about, you know, law versus policy. And, you know, the tricky thing
[01:01:04.160 --> 01:01:10.700]  is that it is so early that oftentimes we don't even really know what we're talking about, you
[01:01:10.700 --> 01:01:17.320]  know, and, and these kinds of conversations are so valuable because we're, we're, we're trying to
[01:01:17.320 --> 01:01:21.460]  figure out what is really important. And that, that's one reason why we've been asking these
[01:01:21.460 --> 01:01:25.620]  questions too, right? You know, what do you think is important? Because we want to hear from, from all
[01:01:25.620 --> 01:01:34.760]  of you. And, but, but trying to make a law too soon has risks, right? Because then you've sort of
[01:01:34.760 --> 01:01:42.360]  codified your limited understanding. And it's really, it's harder to change law than it is
[01:01:42.360 --> 01:01:51.160]  to change policy. So, so in these early stages and really, you know, DoD has not done much.
[01:01:51.160 --> 01:01:57.160]  They've done a lot of talking. There really has not been a lot. Diane mentioned the, the DoD
[01:01:57.160 --> 01:02:04.060]  directive on, on autonomous weapons that was created in 2012. And it calls for the senior
[01:02:04.060 --> 01:02:10.280]  level review for additional scrutiny for systems that would meet certain requirements. They
[01:02:10.280 --> 01:02:17.140]  haven't done a single one because they're just not there yet. So I think... And just,
[01:02:17.140 --> 01:02:21.520]  they haven't done a single one because, because nobody's made it, nobody's tried, nobody's asked,
[01:02:21.520 --> 01:02:30.320]  right? Just to, just to clarify that. Yeah. Yeah. So I think, you know, it's, it's kind of easy to
[01:02:30.320 --> 01:02:38.060]  say, you know, yeah, there's so much that can be done, but in reality, militaries move slowly.
[01:02:38.060 --> 01:02:45.220]  They're built to move slowly. They're these huge bureaucratic structures that are designed to
[01:02:45.220 --> 01:02:51.840]  reduce risk. And so they move really slowly. And so I think at this point, we, we don't
[01:02:51.840 --> 01:02:56.800]  necessarily need a law. We need to actually figure out what the important things are.
[01:02:57.200 --> 01:03:02.560]  And we're not, we're not even there yet. So, so that's why we really appreciate this discussion
[01:03:02.560 --> 01:03:07.260]  today to try to help figure out what are the things we really need to be worried about.
[01:03:08.060 --> 01:03:12.080]  Yeah, thanks. And, and I'm tracking, just looking through the chat here, I think there's,
[01:03:12.580 --> 01:03:16.460]  there's, there's one more comment that I'm seeing that I don't think we've, we've addressed yet that
[01:03:16.460 --> 01:03:23.460]  I'll read off here, which is just that I think the person who recommended this, this report that you
[01:03:23.460 --> 01:03:30.600]  talked about earlier, so this, so they, they raised a few issues from that report. This was the
[01:03:30.600 --> 01:03:37.160]  vulnerabilities, AI vulnerabilities issue. They also raised the point that there's a dynamic where
[01:03:37.160 --> 01:03:46.500]  offense is easier than defense in some of the space. And so I think that, that one thing I would,
[01:03:46.500 --> 01:03:51.100]  you know, I would say two things there, right, which is that, you know, one is that I don't,
[01:03:51.100 --> 01:03:56.680]  I don't think that's special of the AI. And I would also say that, you know, we did some
[01:03:56.680 --> 01:04:04.120]  research a little while back looking at lethal autonomy, you know, potential dynamics or uses
[01:04:04.120 --> 01:04:10.220]  within, hypothetical uses within the military. And, you know, back to this, a drone swarm
[01:04:10.220 --> 01:04:15.200]  attacking a ship example, I would say just don't discount that, that, you know, certainly it's
[01:04:15.200 --> 01:04:18.760]  right that in a lot of contexts, the offense is easier than defense, but there are still
[01:04:20.340 --> 01:04:26.740]  really important defensive contexts where technologies like this can be really helpful.
[01:04:26.740 --> 01:04:30.400]  And, and those are cases where we're talking about, you know, protecting the lives of our
[01:04:30.400 --> 01:04:35.660]  own soldiers. So they're ones that we, you know, I think as a country and certainly our military
[01:04:36.460 --> 01:04:39.140]  regards very highly and is very important.
[01:04:40.280 --> 01:04:46.520]  If I can just kind of add to that, the, the context thing is really important. So, you know,
[01:04:46.520 --> 01:04:51.780]  I was, I was part of the US delegation to the UN and in some of these talks on lethal autonomous
[01:04:51.780 --> 01:05:00.260]  systems. And one thing that was often lost is that the context really is important. So, so you have
[01:05:00.400 --> 01:05:05.340]  some, some people would talk about, well, how will you use a lethal autonomous weapons system
[01:05:05.340 --> 01:05:11.160]  in an urban environment against, you know, against groups that aren't wearing uniforms
[01:05:11.160 --> 01:05:15.840]  and they're, you know, hiding within the population. And then, you know, basically the
[01:05:15.840 --> 01:05:23.280]  answer of, you know, the US and Australia and the UK was like, we won't. And what, what we really,
[01:05:23.280 --> 01:05:31.160]  what they kind of want this stuff for is, is for missile defense in, in a maritime setting,
[01:05:31.160 --> 01:05:36.480]  right? So there's no civilians anywhere around but they have hypersonic missiles coming in
[01:05:36.480 --> 01:05:41.820]  and they want, they want to have capability that helps them to, to better defend against
[01:05:41.820 --> 01:05:48.900]  those kinds of things. So, so just, just recognizing that there are some contexts
[01:05:48.900 --> 01:05:54.980]  where you can use some of this technology with a much lower risk to civilians than others.
[01:05:55.300 --> 01:06:01.720]  And just having a more nuanced conversation where we recognize that. And, and, you know,
[01:06:01.720 --> 01:06:06.020]  militaries need to recognize that too, right? They do internal reviews where they try to
[01:06:06.020 --> 01:06:10.100]  recognize risk and the context is really important for those,
[01:06:10.100 --> 01:06:14.300]  those decisions on whether they think it's okay or not to use them.
[01:06:14.900 --> 01:06:21.020]  Yeah, thanks. And so one more person wrote in a good comment that, that I'll read off here,
[01:06:21.020 --> 01:06:27.440]  right? Which is the concerns about, right? AI needs big data a lot of times to, to work.
[01:06:27.440 --> 01:06:34.260]  And so what about data security there? So do you have any, any comments on that, Larry?
[01:06:34.340 --> 01:06:40.380]  So I think this is, this is, I mean, we talked about, there are some risks that,
[01:06:40.380 --> 01:06:45.120]  that, you know, DoD is like, is really handling well. And then there are some that
[01:06:45.740 --> 01:06:53.600]  I think they're behind. And I think this is one of them. And it's, you know, we talked about AI
[01:06:53.600 --> 01:07:01.200]  and applications. I think this is one reason why, you know, DoD is not as, as making as much
[01:07:01.200 --> 01:07:07.260]  progress as people may think. Because there are all these enablers that they need to have
[01:07:07.980 --> 01:07:14.000]  enabled to use AI effectively. And the data situation with DoD is really problematic.
[01:07:14.560 --> 01:07:20.620]  They, they, they have lots of different networks. You know, they're not necessarily secure.
[01:07:20.620 --> 01:07:24.980]  They don't necessarily talk to each other. Well, they're, they're not always formatted
[01:07:24.980 --> 01:07:30.380]  in a consistent way or quality control. And so that makes it really difficult
[01:07:30.380 --> 01:07:39.320]  to actually use that data in a, in a constructive way. So, so DoD is going to have to get a handle
[01:07:39.320 --> 01:07:45.960]  on that. And part of, part of that is going to be trying to make data more secure. I will say
[01:07:45.960 --> 01:07:54.480]  in the few serious AI applications that, that I'm aware of, there has been kind of this recognition,
[01:07:54.480 --> 01:08:00.380]  both of they need to be careful about data poisoning. And, and that has implications
[01:08:00.380 --> 01:08:07.300]  both on storage, but also on who you get to do your, your data processing, right? You can't
[01:08:07.300 --> 01:08:14.520]  just necessarily outsource this to some commercial organization that may, may or may not be the most
[01:08:14.520 --> 01:08:19.700]  trustworthy. And then the storage. So it's a great question. I think.
[01:08:21.860 --> 01:08:29.300]  Yeah. Okay. Well, thank you guys so much. We really appreciate hearing your, your thoughts
[01:08:29.300 --> 01:08:33.800]  and comments. I'll reiterate what Larry said a minute ago, which is the, please feel free to
[01:08:33.800 --> 01:08:38.340]  reach out to us. You can either do it through discord or through our email addresses that we
[01:08:38.340 --> 01:08:42.900]  put up a few minutes ago. We would love to hear, you know, if there's, if there's more that you
[01:08:42.900 --> 01:08:47.180]  want to talk about that we didn't get to, again, we would, you know, we, we don't have all the
[01:08:47.180 --> 01:08:52.720]  answers, but we are really interested in hearing your thoughts and concerns, even if they're not
[01:08:52.720 --> 01:09:00.860]  things that are, can be perfectly answered today. So, so thank you again, Ethics Village for hosting
[01:09:00.860 --> 01:09:06.280]  us. And I really appreciate everybody's, everybody's participation here.
[01:09:08.520 --> 01:09:14.800]  Awesome. We'd like to, again, thank you to our speakers, Dan and Larry for our very first talk
[01:09:14.800 --> 01:09:20.840]  of day two. For those of you that are here for the rest of the few hours, stay tuned for our
[01:09:20.840 --> 01:09:22.980]  next few talks. Our next talk will be from our guest speaker, Dan. So, Dan, thank you so much for