All right, it looks like we are resetting the stream. One quick second, let me get a
heads up by our recording folks. Let's see. Let me just double-check while we
get this, get this sorted. Yeah, and sometimes I would announce a class like
welcome to Chem 107 or something, just something weird. I like to have fun with my
students, but when I was teaching, it was good. I don't think I was ever bold
enough to try anything like that. Or maybe I just had more pity for my
students. I don't know, one of those two.
Well, I lucked out and taught a, what do you call it, a simple class on...
it was we were building an electric vehicle. Oh, hi everybody. For those of you
that are just tuning in, we're holding out for one quick second while we get
our recording AV equipment going. Mr. The Fixer, let me know when things are
ready to go and we will commence.
Let's see, it looks like... all right, so looks like things are ready. So let us
begin. We're introducing our speakers. So everybody, welcome to the very first talk
of Ethics Village on day two. So today we have Larry Lewis and we have Diane
Bavacek, and they were going to be talking a little bit about killer
robots, so reconsidered. So a little bit about Larry. Larry is the director of the
Center for Autonomy and Artificial Intelligence at CNA. His areas of
expertise include military applications of artificial intelligence and autonomy,
reducing civilian casualties, identifying lessons from current operations, security
systems, and counterterrorism. So Larry spearheaded the first database approach
to protecting civilians in conflict, analyzing military operational data in
conjunction with open source data. He has worked extensively to help reduce, to
help militaries reduce civilian casualties in their operations. Larry has
also served as a senior advisor to the Department of State's Assistant
Secretary for Democracy, Human Rights, and Labor, and was on the U.S. delegation to
the United Nations regarding lethal autonomous weapon systems or laws. Larry
currently holds a PhD in physical chemistry from Rice University. And our
other speaker, Diane Bavacek, is an analyst at CNA, where she researches
military operations and policy with a focus on emerging technologies, including
autonomy, artificial intelligence, unmanned systems, and the cyberspace
domain. She deployed abroad on the USS Vinson during Operation Iraqi Freedom
and Operation Inherent Resolve. Diane currently has a PhD in mathematics from
the University of Michigan. And without further ado, welcome again, y'all, and
thank you again for doing this talk for Ethics Village 2020. Let me queue up
those slides and I will pass it off to you. So I'm gonna go mute. Great. So before we
get to the slides, just a few introductory comments. So thank you.
Thank you for joining us. Diane and I are really excited to talk to you guys. Just
a few things. If this is your first acquaintance with CNA, we're a
nonprofit research organization and we provide objective and independent
analysis and perspectives to the U.S. government, including the U.S. military,
but also other parts of the government. And several years ago, CNA
established a Center for Autonomy and Artificial Intelligence because those
technologies are so important to the U.S. and national security priorities. So
and a key focus for us at CNA is AI safety. So we're going to be talking about that today.
This presentation incorporates a number of different areas of CNA research. At the same
time, this presentation reflects our opinion and not that of CNA or the U.S. government.
Finally, we really want your participation today. So we want this to be participatory.
So you can send us a question in the Discord general text chat. We're monitoring it. I have
it open right here. So feel free, please, to ask questions. We'd love to answer them.
We're aiming to finish this presentation and the general question and answer in an hour.
But we are also happy to stick around longer if we haven't gotten your questions. So feel
free to stick around if we haven't answered it, and we will gladly get to it. So with that,
we'll go ahead and get started. And Diane is going to start us off.
Thanks, Larry. And let me echo what Larry said. Thanks to the Ethics Village for hosting us.
Steve, you can go ahead to the next slide, please. Thanks. So what are we going to talk
to you about today? So here's kind of a general outline. Militaries are turning to artificial
intelligence. And the U.S. Department of Defense is certainly one of those that sees
moving forward with AI as an imperative. And the DoD is going to need the tech sector,
the help of the tech sector, the help of AI researchers in order to be competitive here.
There are risks associated with incorporating AI into military applications. A lot of risks and
challenges that are really important and crucial getting right. But there are also a lot of
misconceptions that come up in this conversation. And so we're going to talk both about challenges
and the misconceptions. And DoD practices and policies that are in place right now that address
some of these risks. And again, to go back to the role of the tech sector and AI researchers here,
addressing some of the challenges associated with AI in this setting is something that we
think the tech sector can really help DoD to do. In addition to risks, though, there are actually
benefits, opportunities associated with integrating AI into military applications. So there's a real
potential for benefits like fewer civilian casualties or protecting hospitals, different
humanitarian benefits that we'll talk through. And again, to bring back to what is the role
of tech sector here, we've seen through history that outside influences can motivate DoD to try
harder in these areas. So what we have here is a new paradigm with AI coming into applications
in the military and more broadly. And we think that you should see this as an opportunity to
help and influence the way that AI is used in warfighting. So this is our aim. The plan for
this talk is that I'm going to talk through the first few points and then Larry will talk
through the later points. And then we will spend some time going through the questions that we see
in the chat. All right, go to the next slide, please. So what are we talking about here,
first of all? How are we thinking about the definition of AI? So to be consistent with how
the U.S. government uses this term, we're thinking of this broadly. So one definition you can think
of for AI is a set of techniques, including machine learning, that is designed to approximate
a cognitive task. So one kind of piece that I want to get at, at this point, is to talk about
how people group AI into kind of two broad bins. So people talk about artificial general intelligence.
And the idea there is kind of thinking of the Terminator, sort of, right? Is artificial
intelligence that can do what a human can do? That can understand or learn any cognitive task
that a human can do. So artificial general intelligence is not the state of technology today,
and it's not expected to come to fruition in the foreseeable future at this point. So this is not
going to be the focus of our talk today. What we're going to focus on is the other category,
which is narrow artificial intelligence. So narrow AI is AI that's focused on a specific task,
something like image classification, right? So it carries out a specific task within a specific
context. Why is AI relevant to warfighting? Well, one reason is that AI helps you make better
decisions. So if you might want the best possible decision-making tools to be used in areas of your
life that you care a lot about, right? Maybe the person managing your investments or a doctor
giving you diagnosis, if you want them to have access to the best technologies to help them with
decision-making, right? It's the same idea. This is why military commanders, this is why U.S.
government leadership wants to draw from this technology. Also in warfighting, AI can be useful
if your military is facing a threat that is vast or numerous or very dynamic. So think about
something like a drone swarm attacking a naval ship, right? So that's the kind of setting that
DoD cares very much about, being able to combat that threat. And so this is another place where
AI can be really critical. So for all of these reasons, DoD and other militaries around the world
are moving forward with exploring and investing in artificial intelligence.
What type of military applications are we talking about here? So a few examples, right? And I've
listed a few bins here in the slide. So AI can help with by providing decision aids on the
battlefield. So things like detecting anomalies in behavior, the behavior of say people on the
ground or shipping traffic. Sensor data fusion is another application. Human machine teaming
is also a key area of AI applications. So think about maybe an unmanned ground system that is
assisting a soldier by carrying a lot of heavy gear or by providing surveillance where the
soldier is patrolling. And business applications are also something that are going to be important
to DoD going forward. AI uses in say preventative maintenance, network defense, or even to figure
out how to target recruiting efforts. But an AI is going to, you know, there are important
challenges of artificial intelligence that comes into those type of applications, but that's not
the most contentious type of applications that we're talking about here. So DoD and other
militaries are also considering how you might integrate AI into autonomous systems or weapon
systems, right? So autonomous systems being systems that operate in some sense on their own,
right? And you can combine that with a lethality component too. And so here's a case where you
really see that it's critical that we make sure to get it right moving forward with this technology.
Next slide. Okay, so DoD and the U.S. government more broadly are thinking about how to use
AI in national security. So they're already moving ahead with creating strategies and guidance,
making some organizational changes, and investing in research and development. And so I've listed
some examples of these types of developments in the slide. The one that I'll anchor on here
is this first sub-bullet, which is the recently established DoD AI strategy.
And this is a document that... this and most of the other documents that I reference here
are things that are available open source. So you can just Google and find at least an
unclassified summary of the strategy. So a couple of things that I'll just list off here,
just to anchor on this piece, a couple of things that they say in the strategy, right? So they
emphasize that the women and men in the U.S. armed forces are the real source of strength.
And the intent is for AI to empower but not replace those who serve.
Another thing that the strategy says is that AI is poised to change the character
of the future battlefield and the pace of the threats that we must face.
And so I just want to point out that you can see from that line that the DoD is... the context that
DoD is thinking about this in is one in which they expect that adversaries will also be using
this technology. So that's kind of the landscape that we're thinking of here. Go ahead to the next
slide. Thanks. So AI really represents a paradigm in terms of the relationship between DoD and the
private sector. That's a new and interesting dynamic, which is that in the past, we're used
to thinking of DoD as having unique needs with research and development and that they very much
bankroll the research in order to develop platforms they need, like military aircraft
or develop weapon systems technologies. Even early versions of the internet were something that DoD
bankrolled because it was something that they needed and the developments weren't available
elsewhere. So that's the dynamic that DoD is used to having. But bringing AI into the picture
is a big change because obviously AI is something that has many uses outside of
just military applications. And because of that, tech sector is already moving forward
rapidly with research and development. So the graph here is one where we kind of try to get
at this snapshot of the difference between U.S. government and tech sector spending here,
which is that you can see the lower bars in the graph are the budget that U.S. government overall,
information technology research and development, including AI. And the taller light purple bars
are we just took just five top tech companies and it is their summed research and development
investments. And so you can see that just looking at that snapshot, DoD and the U.S. government
overall are going to be dwarfed by this type of research and development spending. And so
this just illustrates that DoD has a real dependency on what's happening in the private
sector, what's happening with external research and development in AI. All right, next slide, please.
Okay, so now let's talk about risks. Actually, let's try, hopefully this will work, to see if
you guys can, this is one of the areas where we really want to try to have as much of a dialogue
as we can with you, the listeners. So I'm going to talk through sort of the landscape of risks.
As I'm doing that, we would love it if you would type into the Discord chat, what are your biggest
concerns here with regard to military applications of AI, right? Or what are questions or thoughts
that you have on this? And after I talk through some of these risks, we'll see if Larry can read
off some of what you all wrote in. Okay, so AI, also thinking of AI potential use in lethal
autonomy, these are really controversial ideas in the public sphere. I don't know if
how many of you were at DEFCON and came to Ethics Village sessions last year, you may have seen
a talk given by a group that's a proponent of preemptively banning lethal autonomous
weapon systems. So there's a real kind of an active debate and dialogue about some of
these questions. And so what are the concerns that people are voicing here? All right, well,
one concern is that AI will destroy the world. So that's one that we've seen, for example, Elon Musk
say quite publicly, even as I'll mention, even as, of course, Elon Musk has simultaneously
been investing in AI research. But AI and lethal autonomy are inherently unlawful or unethical.
This is an assumption that is made by the proponents of banning these types of technologies.
There are questions about accountability, concerns about discrimination, which means the ability of
a weapon system to tell a combatant from a civilian. Of course, biases are also
an important challenge and concern when it comes to AI. Let's go to the next slide.
So embedded in these concerns are some real and critical challenges that are actually going to be
they appear they're going to be surmountable, but they're important challenges to think about.
But also embedded in these are some real misconceptions. Okay, so just to talk through
some of this, AI will destroy the world. Okay, so that concern is really rooted in
worries about artificial general intelligence. And that's not what we're talking about here.
But what about questions of lawfulness, ethics, and accountability? So those next two points.
So these concerns are rooted in a misunderstanding of how DoD operates. So DoD follows many domestic
and international laws, follows many ethical principles. And these laws and principles
don't depend on the type of technology that you're using. So the laws of war apply no matter
what weapon you're using. So similarly, accountability. Accountability is also
baked into how DoD operates. From acquisition, test and evaluation standards, to the chain of
command. And also, this doesn't depend on the type of technology that you're talking about.
Although certainly things like test and evaluation specific processes will have to be updated
for new technologies as they come through, including for artificial intelligence.
So that said, there definitely, not to discount, there will absolutely be challenges
in developing military applications of AI. So in particular, you want to make sure that it really
works. So integrated into that is making sure that systems can be discriminant, that systems
will, or use of systems will address bias. Basically, you want to make sure that systems
will operate as intended in any other ways. And so just a couple quick comments here.
We say that these concerns are surmountable. So some of the way to think about that here
is that the goal is not necessarily to, say, develop some kind of AI-enabled weapon system
that is able to discriminate civilians from combatants in all possible situations.
So the point is that you would only approve a system for use in settings in which you knew
it was going to be able to function properly. So that's kind of the aim and the goal here.
This isn't going to be something that works in all settings. Of course, that's true of
legacy non-AI-enabled systems and weapons as well.
Also, we're not necessarily aiming for, say, systems that are learning in real time,
especially weapon systems. So it's more likely to be thinking about at this stage,
right, maybe systems that will be collecting data in real time, but that maybe it would not
go ahead and update how it's making decisions until those updates can be validated.
So keep some of those important caveats in mind here.
Larry, did anybody write anything good in the chat about concerns and risks?
Yes, definitely. So a few things here are... there's a link to a report that is a really
good report from early 2019. And it talks about different risks of AI. And I think it's actually
good to kind of frame this. So we are talking specifically about military applications of AI.
So the report that we do recommend you look at talks about broader risks of AI. And it
talks about three different kinds of risks. One is digital security. So digital security,
physical security, political security. So digital security goes to cyber attacks and the kinds of
infiltration I can do. Physical security, they give the example of the potential for using an
autonomous self-driving car as a weapon. So like a terrorist attack that you can use that.
And then third, they talk about political security. And that includes misuse of surveillance,
propaganda, misinformation. And these are all really great points. So the fact that we don't
have those on there is because we're kind of talking about the more narrow military applications
of AI. But it's important to acknowledge that there are broader risks as well.
Larry, what is that? What report is it that you're referring to? Or who made it?
Yeah, there's a link. And see, actually, who did it? I've looked at the link at this before.
There are many authors from different organizations. So University of Oxford,
Future of Humanity Institute, CNAS was also a contributor, OpenAI. So it's a multi-author,
multi-organization report. But the link is in the general text channel. So definitely,
it's good to check out. Yeah. So another comment is about bias and how bias seems like a really
big deal, which is it absolutely is. And I guess one comment to make about that is that
humans are biased, right? Humans have lots of cognitive biases. And so AI applications that
are built by humans that leverage data that are developed by humans should also be expected to
be biased. So that is absolutely true. The one thing to me that's a key difference is that
with AI applications, you can explicitly examine those biases and try to correct for them. So
that's actually a valuable thing that we can do. However, too often, we don't do that. So we need
to be very deliberate about trying to characterize and correct those biases. Let's see. Another
comment is majority of DoD spending on AI goes to warfighting and lethality,
not logistics, humanitarian assistance, predictive maintenance, logistics. Thank you for that
comment. We're actually going to... that is a big topic of our talk. So please keep on
listening and participating. Do AI weapon systems fall under the Geneva Convention?
Um, so if it's used for armed conflict, then it should.
And I think you're going to you're going to continue to talk about this,
kind of the legal aspect of this. bleed over from DoD and the intelligence community.
And that's, that's also it. That is a tricky issue today, because because of the
um, like, we have some some US activities that do kind of combine the DoD and intelligence
community. So that is something that kind of needs to be watched. And then how do we ethically
import bias into AI? And again, I think, going back to the earlier comment about bias, bias is
really a significant issue we need to be careful about. And I think, you know, yes, the Microsoft
chatbot is a great example. And how, again, we can't just do something and then put it out in
the world and then assume it's going to work well, we need to monitor it, we need to be explicitly
monitoring for bias and intervening as well. So that kind of gets it what I said a minute ago
about making sure to validate as as you're learning, right, we're gonna we're gonna have
to validate that learning. So great comments. Okay, great. Well, thank you guys all and keep
them coming. Um, we will we will come back to looking at the at the comments in a little bit.
I'll go go ahead for the next slide, please. Okay, I'm gonna say a few words now. Okay, right. So,
so there are misconceptions out there, but also a lot of very real challenges and risks. And DOD
is doing some things now that address some of these risks. And so any, any, any discussion of
this topic, right needs to have have this information as part of the context. So to run
through some of that, right, some some of the things that DOD does that address these risks
are just general DOD practices, and some of them are things specifically being developed for
AI in a related context. So, so first, I'll talk about the general practices. Okay, so,
so legal considerations, right. So, so, to kind of reiterate something I mentioned before,
DOD incorporates legal considerations into, you know, pretty much everything it does development
certification, use of any systems. So, so, and these laws, you know, laws of war, they apply no
matter what type of technology you're using, right, things like proportionality, that, that,
that's a principle that does not depend on the weapon that you're using. And so,
so these things generally apply independent of the technology. Practices also help to
mitigate some of these risks, right. So test and evaluation processes need to make sure that
systems function as intended. Whenever you have a new system, also training will be established
for that and operating procedures for using any kind of system, especially weapon systems.
Policy is also a really important part of how DOD operates. So one example I'll give here
is a 2016 executive order that's focused on minimizing civilian casualty.
So this executive order says a lot of really great things. And I want to just highlight one,
one piece of it, which I think is especially interesting in this context. So, so the, the EO
says that the U.S. is going to develop, acquire, and field weapon systems and other technological
capabilities that further enable the discriminant use of force in different operational contexts.
So what are we saying there, right? We're saying that if there's technology that comes along that
will allow us to be more discriminant in war fighting, you know, we're going to look at using
that. And so, so keep that in mind because what Larry's going to talk about in a, in a minute or
two here is, is the, you know, the very real possibility that AI can, can help us be more
discriminant in war fighting. See, please go to the next slide. So now I'll talk just briefly
about some AI specific steps that the DoD is taking. I mentioned the AI strategy earlier. DoD
also very recently adopted some ethics, AI ethics principles. These came from the work of the
Defense Innovation Board, right? And so these are, these are principles that, that include,
you know, appropriate levels of human judgment, the intent that that will allow it if they exhibit
unintended escalatory behavior. So this is, this is really the intent of DoD moving forward.
So another, another related policy dimension, right, is a DoD has a directive on lethal
autonomous weapon systems, which was actually from all the way back in 2012,
which again says that appropriate levels of human judgment will always be used when it comes to
lethal autonomy. And, and it also puts out that in many cases, a lethal autonomous weapon system
would have to undergo a senior level review before it would ever be used by DoD forces.
Test and evaluation requirements are being looked at, right? So all of these steps are good and they
do start to get at a lot of the risks that we talked about. But as, as Larry alluded to,
with talking about bias a minute ago, right, we, we would put out there that more is needed here.
And in particular, what we don't see right now and really, really is necessary is a comprehensive
approach to identifying and systematically addressing risks. So I've, I've talked here
about how AI represents a new dynamic between DoD and the tech sector, and how there are a lot
of challenges out there. And we're already working to address some of those challenges.
I want to turn things over now to Larry, and we'll hear a little bit about the opportunities.
Great. So if you can go to the next slide, please.
Okay. So we've heard about that there are risks of military use of AI,
how those risks can be different than those that are sometimes commonly discussed, and that we need
to be more deliberate in addressing them. At the same time, we'll highlight that there are
opportunities. I'll talk about three things. One is there is potential for good with the use of
technology. AI and autonomous systems can be used deliberately for humanitarian benefits.
Second of all, history shows that outside influence can help, can help the U.S. military
work harder on focusing on humanitarian issues. And finally, times, things are, times are changing,
and there are new opportunities where the tech sector can have an influence on national security
policy that it has not had in the past. So if you can go to the next slide, please.
Okay. So, so an argument is often made how advanced technology has humanitarian benefits.
So an example that I hear a lot is, well, we have precision-guided munitions,
they're much better for civilians on the battlefield than dumb bombs. Okay. And that is,
that is generally true. At the same time, that what that really is, is a humanitarian side benefit
of a military aim, right? So precision-guided munitions were not created to help innocent
civilians. They were created to help the military to hit targets more effectively,
and they happen to have a humanitarian side benefit. But technology can also be used
deliberately to help address pressing humanitarian challenges and war. And some of those
opportunities include reducing civilian casualties, you know, avoiding tragic attacks
on hospitals and other protected sites, and also deconfliction of humanitarian aid on the battlefield
so that those, you know, so that World Food Program trucks aren't targeted in military attacks.
Next slide. So here's an example. So protecting hospitals in conflict. Medical care in conflict
was the very first issue taking up as a measure of international humanitarian law. We talked about
the Geneva Conventions, and there are other legal agreements that governments have agreed to over
the years. Medical care in conflict was the very first issue taken up. And of course, the origin of
ICRC was centered around providing medical care in conflict, making sure it was not interrupted
by a war. So it was the first thing, and yet attacks on hospitals continue 150 years later
after ICRC was created, after the first international law was created to help
conflict be more humane. The picture here is an MSF hospital in Yemen that was destroyed by an
airstrike by the Saudi-led coalition in August 2016. This kind of thing, this kind of attack,
happens too many times. And you see the international community trying to do things.
I think of the United Nations, they had a Security Council Resolution 2286
that calls the international community to action. We need to do more. And yet nothing
practical has been done. So we have a really pressing problem with protecting hospitals.
So to go to the next slide.
There are things we can do. So we're a research organization. And in my grad school,
I have a PhD in physical chemistry. So I had a laboratory with a laser and with
liquid nitrogen kind of fuming to make things really cold and a vacuum chamber, and it smelled
horrible. And that was kind of the environment that I worked in. Now at CNA, the world is our
laboratory. So we go out and we go to places like Yemen and Afghanistan and Iraq and Syria,
and we try to get the data to help us to better understand these kinds of problems.
So one of the things I did is look at attacks on hospitals and looking at the data and say,
okay, why do these things happen? What goes wrong when they're attacked? And our analysis shows that
there are two kinds of failures that both have to happen simultaneously for a hospital to be
attacked. One is a failure in deconfliction. And that is, if you look at modern militaries,
they try to maintain a list of locations that are supposed to be protected and not attacked,
and they call this a no-strike list. So they're supposed to be looking at lists when they are
about to attack someone. And an attack on a medical facility means that this deconfliction step
has failed. But that's not the only thing they have to do. They're also supposed to try to
and avoid attacks on medical facilities. And one way is that, again, through the Geneva
Conventions, identification of hospitals can be facilitated by the hospitals displaying a
Red Cross or a Red Crescent. So an attack means that that identification process has also failed.
So if you can fix either of these problems, you can stop the attack. And it'd be nice to work on
both, but if you can make one of these things successful instead of a failure,
you prevented a tragic attack on a hospital. So if you go to the next slide...
technology can help reduce attacks on hospitals in various ways. So one example is Australia.
They're using machine learning right now. They're exploring machine learning applications to help
automatic recognition of protected symbols, like the Red Cross and the Red Crescent.
So it doesn't automate the attack. Basically, it's a safety net. So if you're looking to make
an attack and the machine learning algorithm senses that there's a Red Cross or a Red Crescent
in the target area, it says, hey, stop. You need to stop. There are also opportunities
to use technology to strengthen the deconfliction process. For example, White Flag is being
developed. It's a protocol developed by the Netherlands. And that is an encrypted technology
to strengthen this deconfliction process. And that's really important because what we see
as we look at the data on this is that often the information that would stop an attack is
known somewhere, but there is an information flow problem and it's not reaching the person
that is making the decision whether to attack or not. So if we can improve that,
then we can also help reduce attacks on hospitals. And the protection of hospitals
is just one area. It's an important area, but just one area where technology can be used
specifically for humanitarian benefit. We've also looked at how to reduce civilian casualties,
how to detect civilian casualties, which is also an issue. Oftentimes you'll hear
you'll hear militaries say, you know, they did a strike. There are accusations about civilian
casualties and they say, nope, we didn't do it. And what the data says is oftentimes they did,
but they're just not very good at detecting civilian casualties. So, and if you can't detect
them, then you don't know you have a problem to solve. So this is also a really important issue
that can be helped by technology. There's also anticipating and avoiding reverberating
long-term effects of attacks. So essential services, that's something that militaries
around the world really do poorly. And that could also be helped. And then there's also
traceability and accountability for violations. The New York Times had a really powerful article
on how Russia and the Syrian regime were specifically targeting hospitals in Syria.
And we can potentially use technology to not only, not only avoid things, but also provide
more accountability, provide basically the proof that says, Hey, this is the smoking gun.
You guys did this and you can't just argue, you know, fog of war. So these are just some
examples of how technology can be used for, for good and for humanitarian benefits.
So the next slide, please.
Hey, Larry, can I jump in with a good point that somebody made in one of the chats about what
you're talking about here, which is that, you know, if you have these, these algorithms that
are designed to detect red crosses and then not shoot at that thing, right? The question is,
well, doesn't that mean that all of the bad guys are just going to start putting red crosses on,
on their, on their stuff?
Sure. And I mean, yeah. So, so there, there, there are states and actors that try to follow
the law, the law of war, internationally humanitarian law, like the genomic conventions.
And then there are some that don't. That's actually why I included that last piece there,
traceability and accountability. Because with, with, you know, with technology,
you can also better document how actors, and that may be a state like Russia, or it may be
a non-state armed group, but if they are intentionally flouting requirements of the law,
you now have documented this and you can put more pressure on them to, to behave differently.
And try to use different ways to, to, to positively influence them.
Yeah. And, and one of the things I think you're getting at when you're talking about flouting the
law is that it is, goes against international humanitarian law to put those red cross symbols,
red crescent symbols on something that is not actually a protected site. So,
so some bad actors, right, would be doing that even, even without AI technology.
Right. Right. Yeah, certainly, certainly they can, they can do that now. And sometimes they do.
That is, that is not, that is not an excuse to, to just be reckless. Right. So, so we need,
still need to, to have, to strengthen our protection. But at the same time,
this is also an opportunity to avoid the bad behavior that we see today.
Okay. So finally, I mean, so the military cares about humanitarian issues. But at the same time,
there's, there's not a lot of, of strong leadership on this issue. And that's,
that's really because of a combination of formal responsibilities. If you look
organizationally at DOD, you know, they have, they have people in charge of lots of different
things, you know, developing weapons systems, developing, you know, lethal capabilities.
There's really not that same kind of leadership on, hey, how can we, how can we get better at
humanitarian issues? At the same time, there, there, there has been, there have been leaders
who have really focused on this issue, but then they rotate out. And so it does,
long lasting focus. So, so what, what can we do? History shows that non-governmental influences can
strongly encourage progress in this area. And typically that has been through media,
you know, media accounts. And I think of the uncounted article, for example, in 2017,
by Asma Khan was one great example of this, how one powerful, well-researched article
basically influenced DOD to, to be more introspective and to do kind of internal
reviews that actually make real changes. And civil society, you know, NGOs, academia can
have a role. And then of course the UN and ICRC. So they have all, at different times,
had important contributions to help the military to do better. And I listed,
listed three specific examples. You know, there are, these are actually examples, real examples
where outside influences encourage the military to try harder, both in Afghanistan and that really,
they really turned things around and did things differently. And the civilian casualty numbers
went down. Drone strikes and counter-terrorism, you know, leading to the presidential policy
directive, the presidential policy guidance issued by President Obama for counter-terrorism
operations. And then the campaign in Iraq and Syria, again, there were things like,
you know, counted article and also air wars that contributed to greater scrutiny and led to
differences in how the U.S. military has done things. If you go to the next slide.
So, so bottom line, there, there's a, there's an opportunity here. So if you combine kind of
the things we've been talking about, the community needs the tech sector as they have not before.
So, you know, looking at that graph we had earlier, you know, the, just the top five
tech companies in the U.S., their research and development, you know, now 15 times larger than
the entire tech development budget of the United States government. And that ratio is increasing
over time. So the U.S. government sees it needs the tech sector to be competitive. And you see
this in DoD wanting to work with the tech sector and they have a number of initiatives. There's
the defense innovation board. And of course they developed the AI principles that DoD then
accepted. You have the defense innovation unit, defense digital service, and then there are other
initiatives that DoD is trying to engage more with the tech sector. So we just wanted to highlight
this is an opportunity for the tech sector, right? So this is an opportunity to engage and have
influence. So as DoD says, Hey, we want to talk, we want to work together. There are opportunities
for tech companies and people working in the tech sector to influence the areas for engagement and
for cooperation. And also voice concerns about how these issues can be addressed. And then,
you know, I mentioned, you know, right now there's academia, there's a civil society,
there's a UN that are engaging now with DoD on discussions and humanitarian issues. I was in
such a discussion yesterday, right? There was nobody in the tech sector in that discussion,
but there could be, right? As a stakeholder and as a potential solution. So this is an opportunity.
And then I also just wanted to highlight, this is not just for the US, right? So I mentioned
Australia doing work. Also the Netherlands are working on this protocol, white flag,
white flag is an open source solution. So it's also, it doesn't have to be a proprietary solution
for the US or it could be other open source solutions that help address humanitarian
issues that basically could be offered to all parties to conflict. So I think that's,
if you go to the next slide. So now we'd love to have a discussion and we also have
contact information here. So we'd love to continue to answer questions or things you'd
like to raise, but also in the future, we'd love to hear from you and we can continue those
discussions. So open to any questions you may have. Yeah, absolutely. I'll certainly echo what Larry
has said there. And so I've been reading through the comments that people have been posting.
So I'm just going to go ahead and kind of read off the things that I've seen there.
So one of the things that kind of the main thing that a couple of different folks have raised
are concerns around proliferation, right? So one person raised a worry, and this is very timely,
a worry about the trickle down of military technology being used in domestic law enforcement,
right? Or being used abroad to stifle dissent. There was also some concerns brought up about,
well, if other, say, bad actor nations get these technologies, that's something that's going to
serve as a force multiplier for those bad bad actors, dictators or whatever. And so
a mention of, let's see, so the person wrote in that the best way to keep control of technology
is to not develop it. And there's another mention of mutually assured destruction.
So there's kind of a lot baked up in those proliferation concerns. I might start to
address those by saying that certainly thinking about this technology and how we want to move
forward with thinking about exporting or sharing it is absolutely something that DOD and the U.S.
government is taking very seriously and really thinking about right now.
Another thing I'll... and so, again, it's going to be one of these contexts where
there's some hard challenges, you know, wrapped up in there. But at the same time, right,
keep in mind that, you know, as we talked about with that earlier slide with the graph on it,
most of the R&D that's going on in this space is not happening by state actors. It's not happening
by the U.S. government or DOD. So that's a reality to keep in mind, thinking about proliferation
also. And one other thing I'll mention, just thinking about the domestic law enforcement
context, is just be aware that a lot of the rules and laws that DOD has to follow
and what it makes are... so domestic law enforcement is not subject to those laws.
So, again, I think there are things to worry about here, but I'm not sure if
DOD's development of these technologies... that may not be the biggest concern in some
of these cases. Larry, would you add anything on any of those topics?
So, yeah, so great questions. One thing I wanted to mention, especially for the
export and proliferation piece, because I've spent a couple years at state
where this was a really key concern. And one thing that became very apparent to me,
in the case of Yemen, so, you know, the U.S. has provided a lot of arms, you know,
weapons systems, intelligence to the Saudi-led coalition in Yemen. And,
you know, there were lots of discussions about, is it legal to do that? And I think
I saw one report kind of summarized it as lawful, but awful. So the laws and policies
that the U.S. applies to security assistance, providing stuff to other countries,
it considers some things. It considers things like diversion, which is, are you going to give,
we want to safeguard that if we give this, if the U.S. gives this to another country,
they won't give it to somebody else. And there's also something called Leahy law that guards
against gross violations of human rights. So if they're doing like war crimes, really horrible
things, then that can stop transfers. But the more general civilian protection considerations
that we're talking about really are not well incorporated in security assistance. And so
that is something that the state department is really trying to think about doing. We're talking
to them about, you know, new policies and new ways to think about this. But one thing that can help
is if you can provide equipment, including weapons, that have some of these safeguards,
right? So right now, the Saudi-led coalition, how many hospitals have they struck? I've lost
count. It's so many, so many, right? And I mean, you can certainly say, well, we're not going to
give them anything, right? But they've also shown that they'll just buy from the Russians.
So it'd be great to stop them, okay? I'm in favor of that. But if war is going to continue,
are there things that can be done to make war more humane and safer for the civilians that are in the
conflict areas? And so the answer is yes, and that technology can help. And that can certainly be the
case for exports as well. And as I'm reading the chats coming in, I'll just say that folks are
kind of doubling down on concerns about domestic law enforcement again, right? And so predator
drones at protests, you know, an interest in, you know, can we have voting where we select that
we're gonna not fund these types of technologies? Maybe that's in a military or in a law enforcement
context. So let's see, and a note that killer robots are perfect for war crimes. So back to
the armed conflict case. So I'll say again, with respect to the domestic law enforcement front,
right, we're just seeing that, you know, some of this is not, maybe not necessarily an issue
limited to AI developments, right? We're seeing it now. It's a real problem in domestic law
enforcement. And so, you know, it's important that we fix the laws and fix the practices that are in
place now, even before we have sort of new classes of weapons that the military is using.
Um, let's see one, another comment that somebody wrote into the chat was, was concerned about AI
vulnerabilities, right, being especially problematic in this context, they mentioned
adversarial attacks, data poisoning, right. So these are, so these are getting technical
challenges, right, that are going to be hard challenges to to solve, right. So this goes back
to, there are a lot of challenges and just getting getting the technology right to actually
make these applications work. And so I guess one thing I would say here is is that, that,
you know, yeah, these are hard problems. I think we, you know, we, DoD will need the tech sector,
private industries help to get good solutions here. And in cases where we can't be confident
in these technologies, especially in in lethal, or weapons contexts, those are cases where we need
to not not deploy systems, right, unless we can, unless we can be confident with how they're going
to operate. Can I make one comment about the, you know, I do hear this a lot about, you know,
killer robots are perfect for war crimes, right. And so I think it's useful kind of just to build
a framework here. So there are two kinds of parties that are involved in conflict. Some can
be states, or it can be armed groups. One is parties that want to follow the law. Okay. And
then there are some that don't, they show impunity. And so I think technology applies differently to
the two of them. So those that that wants to apply the law, then this kind of technology can
be useful for, for helping, helping them to better comply with the law and their legal obligations.
But even beyond the law, because in the US, we talked about there are legal things we're required
to do. But then we also want to try even harder than what we're required to do, right. The law
is just the basic requirements. Diane mentioned the, the US civilian casualty executive order in
2016. That goes beyond what is required by the law. It says, you know, there are things we're
required to do, but we're going to do more because it's important. So, so for the group of states or
parties that want to comply with the law, you can help comply with the law. And then you can even
go further. And we talked about some ways to do that. Then there's others that don't comply with
the law, right. And there's the Russians and the Syrians. And, you know, there are different,
different, you know, different terrorist groups out there. They just don't care. They exploit the
law for their own benefits. And in those cases, technology can also be used for accountability
and to traceability and to try to highlight some of the things that they do. Honestly,
you know, there were, there were things I remember in the counter ISIS
conflict in, in Iraq and Syria, ISIS did horrible, horrible things, and most people have no idea.
Right. And so, and just better, better showing those things can help in themselves to influence
those groups, just like the New York Times article, talking about the Russians and
delivery striking hospitals. Okay. They, they did that quietly. When the New York Times came out,
Russia started to squirm, right? Because they don't want to be known for that. They want to do
it for various interests. But when you start putting the spotlight on them, that can help,
that can help influence them in themselves. We should also be thinking about how can we better
use that information to really deter them from doing those terrible things. So just wanted to
kind of give that overall framework for war crimes. So Larry, another comment that somebody
wrote in that I think is a really interesting one is a question of whether, right, so DOD has
these AI policies. Should, should those be codified in law? Because it's, it's really
easy to change norms and guidelines. What would you say to that? So, so I remember actually having
a conversation with Eric Schmidt about, he's part of the, the National Security Commission for AI.
And so there was this discussion about, you know, law versus policy. And, you know, the tricky thing
is that it is so early that oftentimes we don't even really know what we're talking about, you
know, and, and these kinds of conversations are so valuable because we're, we're, we're trying to
figure out what is really important. And that, that's one reason why we've been asking these
questions too, right? You know, what do you think is important? Because we want to hear from, from all
of you. And, but, but trying to make a law too soon has risks, right? Because then you've sort of
codified your limited understanding. And it's really, it's harder to change law than it is
to change policy. So, so in these early stages and really, you know, DoD has not done much.
They've done a lot of talking. There really has not been a lot. Diane mentioned the, the DoD
directive on, on autonomous weapons that was created in 2012. And it calls for the senior
level review for additional scrutiny for systems that would meet certain requirements. They
haven't done a single one because they're just not there yet. So I think... And just,
they haven't done a single one because, because nobody's made it, nobody's tried, nobody's asked,
right? Just to, just to clarify that. Yeah. Yeah. So I think, you know, it's, it's kind of easy to
say, you know, yeah, there's so much that can be done, but in reality, militaries move slowly.
They're built to move slowly. They're these huge bureaucratic structures that are designed to
reduce risk. And so they move really slowly. And so I think at this point, we, we don't
necessarily need a law. We need to actually figure out what the important things are.
And we're not, we're not even there yet. So, so that's why we really appreciate this discussion
today to try to help figure out what are the things we really need to be worried about.
Yeah, thanks. And, and I'm tracking, just looking through the chat here, I think there's,
there's, there's one more comment that I'm seeing that I don't think we've, we've addressed yet that
I'll read off here, which is just that I think the person who recommended this, this report that you
talked about earlier, so this, so they, they raised a few issues from that report. This was the
vulnerabilities, AI vulnerabilities issue. They also raised the point that there's a dynamic where
offense is easier than defense in some of the space. And so I think that, that one thing I would,
you know, I would say two things there, right, which is that, you know, one is that I don't,
I don't think that's special of the AI. And I would also say that, you know, we did some
research a little while back looking at lethal autonomy, you know, potential dynamics or uses
within, hypothetical uses within the military. And, you know, back to this, a drone swarm
attacking a ship example, I would say just don't discount that, that, you know, certainly it's
right that in a lot of contexts, the offense is easier than defense, but there are still
really important defensive contexts where technologies like this can be really helpful.
And, and those are cases where we're talking about, you know, protecting the lives of our
own soldiers. So they're ones that we, you know, I think as a country and certainly our military
regards very highly and is very important.
If I can just kind of add to that, the, the context thing is really important. So, you know,
I was, I was part of the US delegation to the UN and in some of these talks on lethal autonomous
systems. And one thing that was often lost is that the context really is important. So, so you have
some, some people would talk about, well, how will you use a lethal autonomous weapons system
in an urban environment against, you know, against groups that aren't wearing uniforms
and they're, you know, hiding within the population. And then, you know, basically the
answer of, you know, the US and Australia and the UK was like, we won't. And what, what we really,
what they kind of want this stuff for is, is for missile defense in, in a maritime setting,
right? So there's no civilians anywhere around but they have hypersonic missiles coming in
and they want, they want to have capability that helps them to, to better defend against
those kinds of things. So, so just, just recognizing that there are some contexts
where you can use some of this technology with a much lower risk to civilians than others.
And just having a more nuanced conversation where we recognize that. And, and, you know,
militaries need to recognize that too, right? They do internal reviews where they try to
recognize risk and the context is really important for those,
those decisions on whether they think it's okay or not to use them.
Yeah, thanks. And so one more person wrote in a good comment that, that I'll read off here,
right? Which is the concerns about, right? AI needs big data a lot of times to, to work.
And so what about data security there? So do you have any, any comments on that, Larry?
So I think this is, this is, I mean, we talked about, there are some risks that,
that, you know, DoD is like, is really handling well. And then there are some that
I think they're behind. And I think this is one of them. And it's, you know, we talked about AI
and applications. I think this is one reason why, you know, DoD is not as, as making as much
progress as people may think. Because there are all these enablers that they need to have
enabled to use AI effectively. And the data situation with DoD is really problematic.
They, they, they have lots of different networks. You know, they're not necessarily secure.
They don't necessarily talk to each other. Well, they're, they're not always formatted
in a consistent way or quality control. And so that makes it really difficult
to actually use that data in a, in a constructive way. So, so DoD is going to have to get a handle
on that. And part of, part of that is going to be trying to make data more secure. I will say
in the few serious AI applications that, that I'm aware of, there has been kind of this recognition,
both of they need to be careful about data poisoning. And, and that has implications
both on storage, but also on who you get to do your, your data processing, right? You can't
just necessarily outsource this to some commercial organization that may, may or may not be the most
trustworthy. And then the storage. So it's a great question. I think.
Yeah. Okay. Well, thank you guys so much. We really appreciate hearing your, your thoughts
and comments. I'll reiterate what Larry said a minute ago, which is the, please feel free to
reach out to us. You can either do it through discord or through our email addresses that we
put up a few minutes ago. We would love to hear, you know, if there's, if there's more that you
want to talk about that we didn't get to, again, we would, you know, we, we don't have all the
answers, but we are really interested in hearing your thoughts and concerns, even if they're not
things that are, can be perfectly answered today. So, so thank you again, Ethics Village for hosting
us. And I really appreciate everybody's, everybody's participation here.
Awesome. We'd like to, again, thank you to our speakers, Dan and Larry for our very first talk
of day two. For those of you that are here for the rest of the few hours, stay tuned for our
next few talks. Our next talk will be from our guest speaker, Dan. So, Dan, thank you so much for