1
00:00:00,000 --> 00:00:04,380
All right, it looks like we are resetting the stream.

2
00:00:04,600 --> 00:00:09,980
One quick second, let me get a heads up by our recording folks.

3
00:00:09,980 --> 00:00:10,740
Let's see.

4
00:00:13,690 --> 00:00:17,490
Let me just double-check while we get this, get this sorted.

5
00:00:17,710 --> 00:00:22,910
Yeah, and sometimes I would announce a class like welcome to Chem 107 or something, just something weird.

6
00:00:23,230 --> 00:00:28,830
I like to have fun with my students, but when I was teaching, it was good.

7
00:00:30,470 --> 00:00:33,950
I don't think I was ever bold enough to try anything like that.

8
00:00:36,010 --> 00:00:38,550
Or maybe I just had more pity for my students.

9
00:00:38,550 --> 00:00:40,130
I don't know, one of those two.

10
00:00:42,730 --> 00:00:50,010
Well, I lucked out and taught a, what do you call it, a simple class on...

11
00:00:50,610 --> 00:00:52,710
it was we were building an electric vehicle.

12
00:00:53,750 --> 00:00:54,950
Oh, hi everybody.

13
00:00:54,950 --> 00:01:02,310
For those of you that are just tuning in, we're holding out for one quick second while we get our recording AV equipment going.

14
00:01:02,970 --> 00:01:03,690
Mr.

15
00:01:03,690 --> 00:01:10,370
The Fixer, let me know when things are ready to go and we will commence.

16
00:01:13,750 --> 00:01:18,330
Let's see, it looks like... all right, so looks like things are ready.

17
00:01:18,330 --> 00:01:20,290
So let us begin.

18
00:01:20,490 --> 00:01:23,490
We're introducing our speakers.

19
00:01:23,490 --> 00:01:29,770
So everybody, welcome to the very first talk of Ethics Village on day two.

20
00:01:29,910 --> 00:01:39,110
So today we have Larry Lewis and we have Diane Bavacek, and they were going to be talking a little bit about killer robots, so reconsidered.

21
00:01:39,110 --> 00:01:41,130
So a little bit about Larry.

22
00:01:41,130 --> 00:01:47,190
Larry is the director of the Center for Autonomy and Artificial Intelligence at CNA.

23
00:01:47,190 --> 00:02:00,310
His areas of expertise include military applications of artificial intelligence and autonomy, reducing civilian casualties, identifying lessons from current operations, security systems, and counterterrorism.

24
00:02:00,550 --> 00:02:10,870
So Larry spearheaded the first database approach to protecting civilians in conflict, analyzing military operational data in conjunction with open source data.

25
00:02:10,870 --> 00:02:17,250
He has worked extensively to help reduce, to help militaries reduce civilian casualties in their operations.

26
00:02:17,250 --> 00:02:25,750
Larry has also served as a senior advisor to the Department of State's Assistant Secretary for Democracy, Human Rights, and Labor, and was on the U.S.

27
00:02:25,750 --> 00:02:31,410
delegation to the United Nations regarding lethal autonomous weapon systems or laws.

28
00:02:31,590 --> 00:02:35,690
Larry currently holds a PhD in physical chemistry from Rice University.

29
00:02:36,510 --> 00:02:51,850
And our other speaker, Diane Bavacek, is an analyst at CNA, where she researches military operations and policy with a focus on emerging technologies, including autonomy, artificial intelligence, unmanned systems, and the cyberspace domain.

30
00:02:51,850 --> 00:02:59,350
She deployed abroad on the USS Vinson during Operation Iraqi Freedom and Operation Inherent Resolve.

31
00:02:59,350 --> 00:03:03,850
Diane currently has a PhD in mathematics from the University of Michigan.

32
00:03:04,230 --> 00:03:12,110
And without further ado, welcome again, y'all, and thank you again for doing this talk for Ethics Village 2020.

33
00:03:12,110 --> 00:03:17,510
Let me queue up those slides and I will pass it off to you.

34
00:03:17,830 --> 00:03:22,050
So I'm gonna go mute.

35
00:03:23,070 --> 00:03:24,070
Great.

36
00:03:24,070 --> 00:03:27,810
So before we get to the slides, just a few introductory comments.

37
00:03:27,810 --> 00:03:28,850
So thank you.

38
00:03:28,970 --> 00:03:30,630
Thank you for joining us.

39
00:03:30,630 --> 00:03:34,670
Diane and I are really excited to talk to you guys.

40
00:03:35,050 --> 00:03:36,550
Just a few things.

41
00:03:36,550 --> 00:03:51,250
If this is your first acquaintance with CNA, we're a nonprofit research organization and we provide objective and independent analysis and perspectives to the U.S.

42
00:03:51,250 --> 00:03:52,350
government, including the U.S.

43
00:03:52,350 --> 00:03:55,910
military, but also other parts of the government.

44
00:03:56,970 --> 00:04:08,270
And several years ago, CNA established a Center for Autonomy and Artificial Intelligence because those technologies are so important to the U.S.

45
00:04:08,270 --> 00:04:10,910
and national security priorities.

46
00:04:10,910 --> 00:04:17,090
So and a key focus for us at CNA is AI safety.

47
00:04:17,110 --> 00:04:19,910
So we're going to be talking about that today.

48
00:04:20,110 --> 00:04:26,870
This presentation incorporates a number of different areas of CNA research.

49
00:04:26,870 --> 00:04:34,970
At the same time, this presentation reflects our opinion and not that of CNA or the U.S.

50
00:04:34,970 --> 00:04:35,600
government.

51
00:04:36,520 --> 00:04:40,340
Finally, we really want your participation today.

52
00:04:40,380 --> 00:04:43,280
So we want this to be participatory.

53
00:04:43,280 --> 00:04:49,540
So you can send us a question in the Discord general text chat.

54
00:04:49,540 --> 00:04:50,800
We're monitoring it.

55
00:04:50,800 --> 00:04:52,360
I have it open right here.

56
00:04:52,940 --> 00:04:56,400
So feel free, please, to ask questions.

57
00:04:56,400 --> 00:04:57,720
We'd love to answer them.

58
00:04:57,720 --> 00:05:03,000
We're aiming to finish this presentation and the general question and answer in an hour.

59
00:05:03,000 --> 00:05:07,920
But we are also happy to stick around longer if we haven't gotten your questions.

60
00:05:07,920 --> 00:05:12,760
So feel free to stick around if we haven't answered it, and we will gladly get to it.

61
00:05:12,760 --> 00:05:14,920
So with that, we'll go ahead and get started.

62
00:05:14,920 --> 00:05:16,980
And Diane is going to start us off.

63
00:05:17,820 --> 00:05:18,640
Thanks, Larry.

64
00:05:18,640 --> 00:05:20,160
And let me echo what Larry said.

65
00:05:20,160 --> 00:05:24,300
Thanks to the Ethics Village for hosting us.

66
00:05:24,880 --> 00:05:27,220
Steve, you can go ahead to the next slide, please.

67
00:05:27,580 --> 00:05:28,380
Thanks.

68
00:05:28,940 --> 00:05:31,720
So what are we going to talk to you about today?

69
00:05:32,580 --> 00:05:35,440
So here's kind of a general outline.

70
00:05:35,840 --> 00:05:38,860
Militaries are turning to artificial intelligence.

71
00:05:39,040 --> 00:05:39,780
And the U.S.

72
00:05:39,780 --> 00:05:46,520
Department of Defense is certainly one of those that sees moving forward with AI as an imperative.

73
00:05:47,480 --> 00:05:56,020
And the DoD is going to need the tech sector, the help of the tech sector, the help of AI researchers in order to be competitive here.

74
00:05:56,860 --> 00:06:02,040
There are risks associated with incorporating AI into military applications.

75
00:06:02,440 --> 00:06:07,420
A lot of risks and challenges that are really important and crucial getting right.

76
00:06:07,420 --> 00:06:10,420
But there are also a lot of misconceptions that come up in this conversation.

77
00:06:10,420 --> 00:06:13,520
And so we're going to talk both about challenges and the misconceptions.

78
00:06:14,140 --> 00:06:21,380
And DoD practices and policies that are in place right now that address some of these risks.

79
00:06:21,900 --> 00:06:36,000
And again, to go back to the role of the tech sector and AI researchers here, addressing some of the challenges associated with AI in this setting is something that we think the tech sector can really help DoD to do.

80
00:06:36,580 --> 00:06:45,200
In addition to risks, though, there are actually benefits, opportunities associated with integrating AI into military applications.

81
00:06:45,200 --> 00:06:54,740
So there's a real potential for benefits like fewer civilian casualties or protecting hospitals, different humanitarian benefits that we'll talk through.

82
00:06:55,100 --> 00:07:05,660
And again, to bring back to what is the role of tech sector here, we've seen through history that outside influences can motivate DoD to try harder in these areas.

83
00:07:06,260 --> 00:07:13,860
So what we have here is a new paradigm with AI coming into applications in the military and more broadly.

84
00:07:14,000 --> 00:07:22,780
And we think that you should see this as an opportunity to help and influence the way that AI is used in warfighting.

85
00:07:22,840 --> 00:07:24,060
So this is our aim.

86
00:07:24,160 --> 00:07:30,720
The plan for this talk is that I'm going to talk through the first few points and then Larry will talk through the later points.

87
00:07:30,720 --> 00:07:36,180
And then we will spend some time going through the questions that we see in the chat.

88
00:07:37,920 --> 00:07:39,800
All right, go to the next slide, please.

89
00:07:42,960 --> 00:07:45,960
So what are we talking about here, first of all?

90
00:07:45,960 --> 00:07:48,720
How are we thinking about the definition of AI?

91
00:07:48,760 --> 00:07:51,680
So to be consistent with how the U.S.

92
00:07:51,680 --> 00:07:54,060
government uses this term, we're thinking of this broadly.

93
00:07:54,060 --> 00:08:03,200
So one definition you can think of for AI is a set of techniques, including machine learning, that is designed to approximate a cognitive task.

94
00:08:06,680 --> 00:08:16,340
So one kind of piece that I want to get at, at this point, is to talk about how people group AI into kind of two broad bins.

95
00:08:16,340 --> 00:08:19,700
So people talk about artificial general intelligence.

96
00:08:19,900 --> 00:08:24,880
And the idea there is kind of thinking of the Terminator, sort of, right?

97
00:08:25,160 --> 00:08:29,000
Is artificial intelligence that can do what a human can do?

98
00:08:29,000 --> 00:08:33,140
That can understand or learn any cognitive task that a human can do.

99
00:08:33,480 --> 00:08:45,680
So artificial general intelligence is not the state of technology today, and it's not expected to come to fruition in the foreseeable future at this point.

100
00:08:45,760 --> 00:08:47,940
So this is not going to be the focus of our talk today.

101
00:08:47,940 --> 00:08:52,540
What we're going to focus on is the other category, which is narrow artificial intelligence.

102
00:08:52,900 --> 00:08:59,800
So narrow AI is AI that's focused on a specific task, something like image classification, right?

103
00:08:59,800 --> 00:09:03,480
So it carries out a specific task within a specific context.

104
00:09:05,300 --> 00:09:09,140
Why is AI relevant to warfighting?

105
00:09:09,160 --> 00:09:13,200
Well, one reason is that AI helps you make better decisions.

106
00:09:13,440 --> 00:09:21,040
So if you might want the best possible decision-making tools to be used in areas of your life that you care a lot about, right?

107
00:09:21,040 --> 00:09:31,020
Maybe the person managing your investments or a doctor giving you diagnosis, if you want them to have access to the best technologies to help them with decision-making, right?

108
00:09:31,020 --> 00:09:31,820
It's the same idea.

109
00:09:31,820 --> 00:09:35,580
This is why military commanders, this is why U.S.

110
00:09:35,580 --> 00:09:40,660
government leadership wants to draw from this technology.

111
00:09:41,720 --> 00:09:51,660
Also in warfighting, AI can be useful if your military is facing a threat that is vast or numerous or very dynamic.

112
00:09:51,660 --> 00:09:57,160
So think about something like a drone swarm attacking a naval ship, right?

113
00:09:57,160 --> 00:10:01,900
So that's the kind of setting that DoD cares very much about, being able to combat that threat.

114
00:10:01,900 --> 00:10:06,040
And so this is another place where AI can be really critical.

115
00:10:06,480 --> 00:10:16,360
So for all of these reasons, DoD and other militaries around the world are moving forward with exploring and investing in artificial intelligence.

116
00:10:17,460 --> 00:10:20,860
What type of military applications are we talking about here?

117
00:10:21,020 --> 00:10:22,740
So a few examples, right?

118
00:10:22,740 --> 00:10:25,100
And I've listed a few bins here in the slide.

119
00:10:25,440 --> 00:10:30,440
So AI can help with by providing decision aids on the battlefield.

120
00:10:30,460 --> 00:10:36,660
So things like detecting anomalies in behavior, the behavior of say people on the ground or shipping traffic.

121
00:10:37,340 --> 00:10:40,680
Sensor data fusion is another application.

122
00:10:41,480 --> 00:10:48,440
Human machine teaming is also a key area of AI applications.

123
00:10:48,440 --> 00:10:58,820
So think about maybe an unmanned ground system that is assisting a soldier by carrying a lot of heavy gear or by providing surveillance where the soldier is patrolling.

124
00:10:59,460 --> 00:11:05,020
And business applications are also something that are going to be important to DoD going forward.

125
00:11:05,340 --> 00:11:14,460
AI uses in say preventative maintenance, network defense, or even to figure out how to target recruiting efforts.

126
00:11:16,940 --> 00:11:31,760
But an AI is going to, you know, there are important challenges of artificial intelligence that comes into those type of applications, but that's not the most contentious type of applications that we're talking about here.

127
00:11:31,880 --> 00:11:41,420
So DoD and other militaries are also considering how you might integrate AI into autonomous systems or weapon systems, right?

128
00:11:41,420 --> 00:11:46,560
So autonomous systems being systems that operate in some sense on their own, right?

129
00:11:46,560 --> 00:11:50,940
And you can combine that with a lethality component too.

130
00:11:50,940 --> 00:11:58,820
And so here's a case where you really see that it's critical that we make sure to get it right moving forward with this technology.

131
00:12:00,180 --> 00:12:01,400
Next slide.

132
00:12:03,100 --> 00:12:05,960
Okay, so DoD and the U.S.

133
00:12:05,960 --> 00:12:12,380
government more broadly are thinking about how to use AI in national security.

134
00:12:12,500 --> 00:12:21,340
So they're already moving ahead with creating strategies and guidance, making some organizational changes, and investing in research and development.

135
00:12:21,340 --> 00:12:28,980
And so I've listed some examples of these types of developments in the slide.

136
00:12:28,980 --> 00:12:38,000
The one that I'll anchor on here is this first sub-bullet, which is the recently established DoD AI strategy.

137
00:12:38,740 --> 00:12:46,680
And this is a document that... this and most of the other documents that I reference here are things that are available open source.

138
00:12:46,680 --> 00:12:50,800
So you can just Google and find at least an unclassified summary of the strategy.

139
00:12:51,580 --> 00:12:59,460
So a couple of things that I'll just list off here, just to anchor on this piece, a couple of things that they say in the strategy, right?

140
00:12:59,460 --> 00:13:04,380
So they emphasize that the women and men in the U.S.

141
00:13:04,380 --> 00:13:06,920
armed forces are the real source of strength.

142
00:13:06,920 --> 00:13:13,260
And the intent is for AI to empower but not replace those who serve.

143
00:13:14,240 --> 00:13:25,020
Another thing that the strategy says is that AI is poised to change the character of the future battlefield and the pace of the threats that we must face.

144
00:13:25,880 --> 00:13:39,820
And so I just want to point out that you can see from that line that the DoD is... the context that DoD is thinking about this in is one in which they expect that adversaries will also be using this technology.

145
00:13:40,440 --> 00:13:44,380
So that's kind of the landscape that we're thinking of here.

146
00:13:45,640 --> 00:13:47,180
Go ahead to the next slide.

147
00:13:48,140 --> 00:13:48,980
Thanks.

148
00:13:49,880 --> 00:13:57,160
So AI really represents a paradigm in terms of the relationship between DoD and the private sector.

149
00:13:57,160 --> 00:14:19,380
That's a new and interesting dynamic, which is that in the past, we're used to thinking of DoD as having unique needs with research and development and that they very much bankroll the research in order to develop platforms they need, like military aircraft or develop weapon systems technologies.

150
00:14:20,000 --> 00:14:30,040
Even early versions of the internet were something that DoD bankrolled because it was something that they needed and the developments weren't available elsewhere.

151
00:14:30,100 --> 00:14:33,080
So that's the dynamic that DoD is used to having.

152
00:14:33,080 --> 00:14:44,460
But bringing AI into the picture is a big change because obviously AI is something that has many uses outside of just military applications.

153
00:14:44,460 --> 00:14:51,940
And because of that, tech sector is already moving forward rapidly with research and development.

154
00:14:52,320 --> 00:15:03,240
So the graph here is one where we kind of try to get at this snapshot of the difference between U.S.

155
00:15:03,240 --> 00:15:14,840
government and tech sector spending here, which is that you can see the lower bars in the graph are the budget that U.S.

156
00:15:14,840 --> 00:15:21,580
government overall, information technology research and development, including AI.

157
00:15:22,040 --> 00:15:33,440
And the taller light purple bars are we just took just five top tech companies and it is their summed research and development investments.

158
00:15:33,440 --> 00:15:38,600
And so you can see that just looking at that snapshot, DoD and the U.S.

159
00:15:38,600 --> 00:15:42,860
government overall are going to be dwarfed by this type of research and development spending.

160
00:15:42,860 --> 00:15:53,520
And so this just illustrates that DoD has a real dependency on what's happening in the private sector, what's happening with external research and development in AI.

161
00:15:55,680 --> 00:15:57,420
All right, next slide, please.

162
00:15:59,380 --> 00:16:01,480
Okay, so now let's talk about risks.

163
00:16:02,660 --> 00:16:14,020
Actually, let's try, hopefully this will work, to see if you guys can, this is one of the areas where we really want to try to have as much of a dialogue as we can with you, the listeners.

164
00:16:14,020 --> 00:16:18,180
So I'm going to talk through sort of the landscape of risks.

165
00:16:18,180 --> 00:16:28,280
As I'm doing that, we would love it if you would type into the Discord chat, what are your biggest concerns here with regard to military applications of AI, right?

166
00:16:28,280 --> 00:16:30,860
Or what are questions or thoughts that you have on this?

167
00:16:31,420 --> 00:16:38,140
And after I talk through some of these risks, we'll see if Larry can read off some of what you all wrote in.

168
00:16:39,180 --> 00:16:52,400
Okay, so AI, also thinking of AI potential use in lethal autonomy, these are really controversial ideas in the public sphere.

169
00:16:52,680 --> 00:17:06,760
I don't know if how many of you were at DEFCON and came to Ethics Village sessions last year, you may have seen a talk given by a group that's a proponent of preemptively banning lethal autonomous weapon systems.

170
00:17:07,900 --> 00:17:15,220
So there's a real kind of an active debate and dialogue about some of these questions.

171
00:17:15,220 --> 00:17:19,320
And so what are the concerns that people are voicing here?

172
00:17:20,200 --> 00:17:23,720
All right, well, one concern is that AI will destroy the world.

173
00:17:23,780 --> 00:17:34,900
So that's one that we've seen, for example, Elon Musk say quite publicly, even as I'll mention, even as, of course, Elon Musk has simultaneously been investing in AI research.

174
00:17:35,260 --> 00:17:41,060
But AI and lethal autonomy are inherently unlawful or unethical.

175
00:17:41,060 --> 00:17:47,420
This is an assumption that is made by the proponents of banning these types of technologies.

176
00:17:48,160 --> 00:18:01,820
There are questions about accountability, concerns about discrimination, which means the ability of a weapon system to tell a combatant from a civilian.

177
00:18:02,480 --> 00:18:09,760
Of course, biases are also an important challenge and concern when it comes to AI.

178
00:18:10,860 --> 00:18:12,740
Let's go to the next slide.

179
00:18:15,940 --> 00:18:31,260
So embedded in these concerns are some real and critical challenges that are actually going to be they appear they're going to be surmountable, but they're important challenges to think about.

180
00:18:31,260 --> 00:18:34,640
But also embedded in these are some real misconceptions.

181
00:18:35,340 --> 00:18:39,820
Okay, so just to talk through some of this, AI will destroy the world.

182
00:18:40,220 --> 00:18:48,360
Okay, so that concern is really rooted in worries about artificial general intelligence.

183
00:18:48,360 --> 00:18:50,720
And that's not what we're talking about here.

184
00:18:51,340 --> 00:18:57,900
But what about questions of lawfulness, ethics, and accountability?

185
00:18:57,900 --> 00:19:00,120
So those next two points.

186
00:19:01,080 --> 00:19:08,800
So these concerns are rooted in a misunderstanding of how DoD operates.

187
00:19:08,800 --> 00:19:17,800
So DoD follows many domestic and international laws, follows many ethical principles.

188
00:19:17,800 --> 00:19:24,380
And these laws and principles don't depend on the type of technology that you're using.

189
00:19:24,380 --> 00:19:29,060
So the laws of war apply no matter what weapon you're using.

190
00:19:29,520 --> 00:19:32,880
So similarly, accountability.

191
00:19:32,880 --> 00:19:37,140
Accountability is also baked into how DoD operates.

192
00:19:37,140 --> 00:19:43,180
From acquisition, test and evaluation standards, to the chain of command.

193
00:19:43,180 --> 00:19:48,620
And also, this doesn't depend on the type of technology that you're talking about.

194
00:19:50,180 --> 00:20:03,320
Although certainly things like test and evaluation specific processes will have to be updated for new technologies as they come through, including for artificial intelligence.

195
00:20:04,460 --> 00:20:15,780
So that said, there definitely, not to discount, there will absolutely be challenges in developing military applications of AI.

196
00:20:16,040 --> 00:20:20,360
So in particular, you want to make sure that it really works.

197
00:20:21,660 --> 00:20:30,840
So integrated into that is making sure that systems can be discriminant, that systems will, or use of systems will address bias.

198
00:20:30,940 --> 00:20:36,900
Basically, you want to make sure that systems will operate as intended in any other ways.

199
00:20:37,600 --> 00:20:41,040
And so just a couple quick comments here.

200
00:20:41,040 --> 00:20:44,480
We say that these concerns are surmountable.

201
00:20:44,480 --> 00:21:02,840
So some of the way to think about that here is that the goal is not necessarily to, say, develop some kind of AI-enabled weapon system that is able to discriminate civilians from combatants in all possible situations.

202
00:21:03,260 --> 00:21:12,180
So the point is that you would only approve a system for use in settings in which you knew it was going to be able to function properly.

203
00:21:12,180 --> 00:21:15,040
So that's kind of the aim and the goal here.

204
00:21:15,040 --> 00:21:18,860
This isn't going to be something that works in all settings.

205
00:21:18,860 --> 00:21:27,180
Of course, that's true of legacy non-AI-enabled systems and weapons as well.

206
00:21:28,480 --> 00:21:37,340
Also, we're not necessarily aiming for, say, systems that are learning in real time, especially weapon systems.

207
00:21:37,340 --> 00:21:52,540
So it's more likely to be thinking about at this stage, right, maybe systems that will be collecting data in real time, but that maybe it would not go ahead and update how it's making decisions until those updates can be validated.

208
00:21:53,000 --> 00:21:56,960
So keep some of those important caveats in mind here.

209
00:21:58,720 --> 00:22:04,060
Larry, did anybody write anything good in the chat about concerns and risks?

210
00:22:06,360 --> 00:22:08,460
Yes, definitely.

211
00:22:08,460 --> 00:22:19,620
So a few things here are... there's a link to a report that is a really good report from early 2019.

212
00:22:19,620 --> 00:22:23,600
And it talks about different risks of AI.

213
00:22:23,600 --> 00:22:29,700
And I think it's actually good to kind of frame this.

214
00:22:29,700 --> 00:22:33,560
So we are talking specifically about military applications of AI.

215
00:22:33,560 --> 00:22:41,880
So the report that we do recommend you look at talks about broader risks of AI.

216
00:22:41,880 --> 00:22:44,360
And it talks about three different kinds of risks.

217
00:22:44,360 --> 00:22:46,100
One is digital security.

218
00:22:46,100 --> 00:22:50,340
So digital security, physical security, political security.

219
00:22:50,340 --> 00:22:56,120
So digital security goes to cyber attacks and the kinds of infiltration I can do.

220
00:22:56,200 --> 00:23:06,100
Physical security, they give the example of the potential for using an autonomous self-driving car as a weapon.

221
00:23:06,100 --> 00:23:09,420
So like a terrorist attack that you can use that.

222
00:23:09,420 --> 00:23:12,820
And then third, they talk about political security.

223
00:23:13,020 --> 00:23:18,660
And that includes misuse of surveillance, propaganda, misinformation.

224
00:23:18,660 --> 00:23:21,620
And these are all really great points.

225
00:23:22,660 --> 00:23:31,160
So the fact that we don't have those on there is because we're kind of talking about the more narrow military applications of AI.

226
00:23:31,160 --> 00:23:35,320
But it's important to acknowledge that there are broader risks as well.

227
00:23:35,320 --> 00:23:37,120
Larry, what is that?

228
00:23:37,120 --> 00:23:38,780
What report is it that you're referring to?

229
00:23:38,780 --> 00:23:40,120
Or who made it?

230
00:23:40,140 --> 00:23:41,960
Yeah, there's a link.

231
00:23:42,240 --> 00:23:47,280
And see, actually, who did it?

232
00:23:48,100 --> 00:23:50,460
I've looked at the link at this before.

233
00:23:50,460 --> 00:23:54,760
There are many authors from different organizations.

234
00:23:54,800 --> 00:24:04,460
So University of Oxford, Future of Humanity Institute, CNAS was also a contributor, OpenAI.

235
00:24:04,480 --> 00:24:09,320
So it's a multi-author, multi-organization report.

236
00:24:09,320 --> 00:24:13,480
But the link is in the general text channel.

237
00:24:13,480 --> 00:24:15,960
So definitely, it's good to check out.

238
00:24:17,140 --> 00:24:17,900
Yeah.

239
00:24:18,740 --> 00:24:28,020
So another comment is about bias and how bias seems like a really big deal, which is it absolutely is.

240
00:24:28,320 --> 00:24:35,140
And I guess one comment to make about that is that humans are biased, right?

241
00:24:35,980 --> 00:24:38,840
Humans have lots of cognitive biases.

242
00:24:38,840 --> 00:24:48,140
And so AI applications that are built by humans that leverage data that are developed by humans should also be expected to be biased.

243
00:24:48,140 --> 00:24:49,480
So that is absolutely true.

244
00:24:49,480 --> 00:25:00,500
The one thing to me that's a key difference is that with AI applications, you can explicitly examine those biases and try to correct for them.

245
00:25:00,500 --> 00:25:04,260
So that's actually a valuable thing that we can do.

246
00:25:04,260 --> 00:25:07,140
However, too often, we don't do that.

247
00:25:07,140 --> 00:25:14,560
So we need to be very deliberate about trying to characterize and correct those biases.

248
00:25:15,400 --> 00:25:16,220
Let's see.

249
00:25:16,220 --> 00:25:27,280
Another comment is majority of DoD spending on AI goes to warfighting and lethality, not logistics, humanitarian assistance, predictive maintenance, logistics.

250
00:25:27,740 --> 00:25:29,540
Thank you for that comment.

251
00:25:29,540 --> 00:25:33,600
We're actually going to... that is a big topic of our talk.

252
00:25:33,600 --> 00:25:37,740
So please keep on listening and participating.

253
00:25:38,780 --> 00:25:43,800
Do AI weapon systems fall under the Geneva Convention?

254
00:25:43,800 --> 00:25:48,760
Um, so if it's used for armed conflict, then it should.

255
00:25:49,020 --> 00:25:56,460
And I think you're going to you're going to continue to talk about this, kind of the legal aspect of this.

256
00:25:57,500 --> 00:26:01,180
bleed over from DoD and the intelligence community.

257
00:26:02,060 --> 00:26:05,560
And that's, that's also it.

258
00:26:05,560 --> 00:26:22,040
That is a tricky issue today, because because of the um, like, we have some some US activities that do kind of combine the DoD and intelligence community.

259
00:26:22,040 --> 00:26:25,340
So that is something that kind of needs to be watched.

260
00:26:25,340 --> 00:26:30,060
And then how do we ethically import bias into AI?

261
00:26:30,060 --> 00:26:38,960
And again, I think, going back to the earlier comment about bias, bias is really a significant issue we need to be careful about.

262
00:26:39,500 --> 00:26:44,720
And I think, you know, yes, the Microsoft chatbot is a great example.

263
00:26:45,440 --> 00:27:01,880
And how, again, we can't just do something and then put it out in the world and then assume it's going to work well, we need to monitor it, we need to be explicitly monitoring for bias and intervening as well.

264
00:27:01,980 --> 00:27:10,420
So that kind of gets it what I said a minute ago about making sure to validate as as you're learning, right, we're gonna we're gonna have to validate that learning.

265
00:27:12,750 --> 00:27:14,470
So great comments.

266
00:27:15,890 --> 00:27:16,730
Okay, great.

267
00:27:16,730 --> 00:27:19,110
Well, thank you guys all and keep them coming.

268
00:27:19,110 --> 00:27:25,150
Um, we will we will come back to looking at the at the comments in a little bit.

269
00:27:25,750 --> 00:27:28,270
I'll go go ahead for the next slide, please.

270
00:27:29,750 --> 00:27:31,490
Okay, I'm gonna say a few words now.

271
00:27:31,490 --> 00:27:31,810
Okay, right.

272
00:27:31,810 --> 00:27:38,670
So, so there are misconceptions out there, but also a lot of very real challenges and risks.

273
00:27:39,170 --> 00:27:44,550
And DOD is doing some things now that address some of these risks.

274
00:27:44,550 --> 00:27:52,950
And so any, any, any discussion of this topic, right needs to have have this information as part of the context.

275
00:27:52,970 --> 00:28:05,430
So to run through some of that, right, some some of the things that DOD does that address these risks are just general DOD practices, and some of them are things specifically being developed for AI in a related context.

276
00:28:05,430 --> 00:28:08,150
So, so first, I'll talk about the general practices.

277
00:28:08,290 --> 00:28:12,230
Okay, so, so legal considerations, right.

278
00:28:12,230 --> 00:28:26,930
So, so, to kind of reiterate something I mentioned before, DOD incorporates legal considerations into, you know, pretty much everything it does development certification, use of any systems.

279
00:28:28,130 --> 00:28:39,970
So, so, and these laws, you know, laws of war, they apply no matter what type of technology you're using, right, things like proportionality, that, that, that's a principle that does not depend on the weapon that you're using.

280
00:28:39,970 --> 00:28:44,510
And so, so these things generally apply independent of the technology.

281
00:28:45,930 --> 00:28:51,110
Practices also help to mitigate some of these risks, right.

282
00:28:51,110 --> 00:28:55,730
So test and evaluation processes need to make sure that systems function as intended.

283
00:28:56,250 --> 00:29:05,510
Whenever you have a new system, also training will be established for that and operating procedures for using any kind of system, especially weapon systems.

284
00:29:06,650 --> 00:29:12,330
Policy is also a really important part of how DOD operates.

285
00:29:12,330 --> 00:29:21,270
So one example I'll give here is a 2016 executive order that's focused on minimizing civilian casualty.

286
00:29:21,690 --> 00:29:25,470
So this executive order says a lot of really great things.

287
00:29:25,470 --> 00:29:31,170
And I want to just highlight one, one piece of it, which I think is especially interesting in this context.

288
00:29:31,170 --> 00:29:35,230
So, so the, the EO says that the U.S.

289
00:29:35,230 --> 00:29:46,270
is going to develop, acquire, and field weapon systems and other technological capabilities that further enable the discriminant use of force in different operational contexts.

290
00:29:46,670 --> 00:29:48,410
So what are we saying there, right?

291
00:29:48,410 --> 00:29:56,710
We're saying that if there's technology that comes along that will allow us to be more discriminant in war fighting, you know, we're going to look at using that.

292
00:29:56,970 --> 00:30:08,990
And so, so keep that in mind because what Larry's going to talk about in a, in a minute or two here is, is the, you know, the very real possibility that AI can, can help us be more discriminant in war fighting.

293
00:30:11,170 --> 00:30:13,130
See, please go to the next slide.

294
00:30:13,670 --> 00:30:20,130
So now I'll talk just briefly about some AI specific steps that the DoD is taking.

295
00:30:20,130 --> 00:30:23,410
I mentioned the AI strategy earlier.

296
00:30:23,490 --> 00:30:29,570
DoD also very recently adopted some ethics, AI ethics principles.

297
00:30:29,570 --> 00:30:34,330
These came from the work of the Defense Innovation Board, right?

298
00:30:34,330 --> 00:30:46,210
And so these are, these are principles that, that include, you know, appropriate levels of human judgment, the intent that that will allow it if they exhibit unintended escalatory behavior.

299
00:30:46,210 --> 00:30:50,210
So this is, this is really the intent of DoD moving forward.

300
00:30:50,910 --> 00:31:09,010
So another, another related policy dimension, right, is a DoD has a directive on lethal autonomous weapon systems, which was actually from all the way back in 2012, which again says that appropriate levels of human judgment will always be used when it comes to lethal autonomy.

301
00:31:09,750 --> 00:31:20,210
And, and it also puts out that in many cases, a lethal autonomous weapon system would have to undergo a senior level review before it would ever be used by DoD forces.

302
00:31:20,570 --> 00:31:23,730
Test and evaluation requirements are being looked at, right?

303
00:31:23,730 --> 00:31:29,370
So all of these steps are good and they do start to get at a lot of the risks that we talked about.

304
00:31:29,770 --> 00:31:37,870
But as, as Larry alluded to, with talking about bias a minute ago, right, we, we would put out there that more is needed here.

305
00:31:37,870 --> 00:31:48,450
And in particular, what we don't see right now and really, really is necessary is a comprehensive approach to identifying and systematically addressing risks.

306
00:31:49,750 --> 00:31:59,410
So I've, I've talked here about how AI represents a new dynamic between DoD and the tech sector, and how there are a lot of challenges out there.

307
00:31:59,410 --> 00:32:03,190
And we're already working to address some of those challenges.

308
00:32:03,610 --> 00:32:08,010
I want to turn things over now to Larry, and we'll hear a little bit about the opportunities.

309
00:32:11,910 --> 00:32:12,550
Great.

310
00:32:12,550 --> 00:32:14,250
So if you can go to the next slide, please.

311
00:32:15,650 --> 00:32:16,170
Okay.

312
00:32:16,170 --> 00:32:29,510
So we've heard about that there are risks of military use of AI, how those risks can be different than those that are sometimes commonly discussed, and that we need to be more deliberate in addressing them.

313
00:32:29,510 --> 00:32:33,170
At the same time, we'll highlight that there are opportunities.

314
00:32:33,410 --> 00:32:35,770
I'll talk about three things.

315
00:32:35,790 --> 00:32:40,270
One is there is potential for good with the use of technology.

316
00:32:40,270 --> 00:32:46,370
AI and autonomous systems can be used deliberately for humanitarian benefits.

317
00:32:46,370 --> 00:32:52,830
Second of all, history shows that outside influence can help, can help the U.S.

318
00:32:52,830 --> 00:32:57,550
military work harder on focusing on humanitarian issues.

319
00:32:57,550 --> 00:33:11,270
And finally, times, things are, times are changing, and there are new opportunities where the tech sector can have an influence on national security policy that it has not had in the past.

320
00:33:11,770 --> 00:33:15,130
So if you can go to the next slide, please.

321
00:33:17,830 --> 00:33:18,390
Okay.

322
00:33:18,390 --> 00:33:27,110
So, so an argument is often made how advanced technology has humanitarian benefits.

323
00:33:27,110 --> 00:33:37,150
So an example that I hear a lot is, well, we have precision-guided munitions, they're much better for civilians on the battlefield than dumb bombs.

324
00:33:37,150 --> 00:33:37,430
Okay.

325
00:33:37,430 --> 00:33:39,870
And that is, that is generally true.

326
00:33:39,870 --> 00:33:47,810
At the same time, that what that really is, is a humanitarian side benefit of a military aim, right?

327
00:33:47,810 --> 00:33:53,570
So precision-guided munitions were not created to help innocent civilians.

328
00:33:53,570 --> 00:34:01,470
They were created to help the military to hit targets more effectively, and they happen to have a humanitarian side benefit.

329
00:34:01,590 --> 00:34:09,910
But technology can also be used deliberately to help address pressing humanitarian challenges and war.

330
00:34:10,050 --> 00:34:33,630
And some of those opportunities include reducing civilian casualties, you know, avoiding tragic attacks on hospitals and other protected sites, and also deconfliction of humanitarian aid on the battlefield so that those, you know, so that World Food Program trucks aren't targeted in military attacks.

331
00:34:33,630 --> 00:34:35,110
Next slide.

332
00:34:36,470 --> 00:34:39,290
So here's an example.

333
00:34:39,290 --> 00:34:43,810
So protecting hospitals in conflict.

334
00:34:43,810 --> 00:34:51,710
Medical care in conflict was the very first issue taking up as a measure of international humanitarian law.

335
00:34:51,710 --> 00:35:00,210
We talked about the Geneva Conventions, and there are other legal agreements that governments have agreed to over the years.

336
00:35:00,210 --> 00:35:03,650
Medical care in conflict was the very first issue taken up.

337
00:35:03,650 --> 00:35:14,850
And of course, the origin of ICRC was centered around providing medical care in conflict, making sure it was not interrupted by a war.

338
00:35:16,030 --> 00:35:31,990
So it was the first thing, and yet attacks on hospitals continue 150 years later after ICRC was created, after the first international law was created to help conflict be more humane.

339
00:35:31,990 --> 00:35:43,890
The picture here is an MSF hospital in Yemen that was destroyed by an airstrike by the Saudi-led coalition in August 2016.

340
00:35:44,010 --> 00:35:48,750
This kind of thing, this kind of attack, happens too many times.

341
00:35:49,170 --> 00:35:52,370
And you see the international community trying to do things.

342
00:35:52,370 --> 00:36:02,190
I think of the United Nations, they had a Security Council Resolution 2286 that calls the international community to action.

343
00:36:02,290 --> 00:36:04,410
We need to do more.

344
00:36:04,930 --> 00:36:07,750
And yet nothing practical has been done.

345
00:36:07,850 --> 00:36:11,190
So we have a really pressing problem with protecting hospitals.

346
00:36:11,790 --> 00:36:14,190
So to go to the next slide.

347
00:36:16,070 --> 00:36:18,330
There are things we can do.

348
00:36:18,330 --> 00:36:21,130
So we're a research organization.

349
00:36:21,370 --> 00:36:27,830
And in my grad school, I have a PhD in physical chemistry.

350
00:36:27,830 --> 00:36:40,690
So I had a laboratory with a laser and with liquid nitrogen kind of fuming to make things really cold and a vacuum chamber, and it smelled horrible.

351
00:36:41,150 --> 00:36:44,850
And that was kind of the environment that I worked in.

352
00:36:44,870 --> 00:36:48,670
Now at CNA, the world is our laboratory.

353
00:36:48,670 --> 00:37:03,370
So we go out and we go to places like Yemen and Afghanistan and Iraq and Syria, and we try to get the data to help us to better understand these kinds of problems.

354
00:37:03,960 --> 00:37:11,930
So one of the things I did is look at attacks on hospitals and looking at the data and say, okay, why do these things happen?

355
00:37:11,930 --> 00:37:14,070
What goes wrong when they're attacked?

356
00:37:14,070 --> 00:37:22,430
And our analysis shows that there are two kinds of failures that both have to happen simultaneously for a hospital to be attacked.

357
00:37:22,570 --> 00:37:26,690
One is a failure in deconfliction.

358
00:37:26,790 --> 00:37:37,350
And that is, if you look at modern militaries, they try to maintain a list of locations that are supposed to be protected and not attacked, and they call this a no-strike list.

359
00:37:37,350 --> 00:37:44,710
So they're supposed to be looking at lists when they are about to attack someone.

360
00:37:45,010 --> 00:37:51,110
And an attack on a medical facility means that this deconfliction step has failed.

361
00:37:51,630 --> 00:37:53,870
But that's not the only thing they have to do.

362
00:37:53,870 --> 00:37:57,910
They're also supposed to try to and avoid attacks on medical facilities.

363
00:37:57,910 --> 00:38:11,590
And one way is that, again, through the Geneva Conventions, identification of hospitals can be facilitated by the hospitals displaying a Red Cross or a Red Crescent.

364
00:38:11,590 --> 00:38:16,490
So an attack means that that identification process has also failed.

365
00:38:16,530 --> 00:38:22,010
So if you can fix either of these problems, you can stop the attack.

366
00:38:22,010 --> 00:38:32,310
And it'd be nice to work on both, but if you can make one of these things successful instead of a failure, you prevented a tragic attack on a hospital.

367
00:38:32,310 --> 00:38:33,980
So if you go to the next slide...

368
00:38:36,670 --> 00:38:42,650
technology can help reduce attacks on hospitals in various ways.

369
00:38:42,650 --> 00:38:44,870
So one example is Australia.

370
00:38:44,870 --> 00:38:47,230
They're using machine learning right now.

371
00:38:47,230 --> 00:38:55,650
They're exploring machine learning applications to help automatic recognition of protected symbols, like the Red Cross and the Red Crescent.

372
00:38:56,090 --> 00:38:58,390
So it doesn't automate the attack.

373
00:38:58,390 --> 00:39:00,410
Basically, it's a safety net.

374
00:39:00,410 --> 00:39:11,850
So if you're looking to make an attack and the machine learning algorithm senses that there's a Red Cross or a Red Crescent in the target area, it says, hey, stop.

375
00:39:12,590 --> 00:39:17,610
You need to stop.

376
00:39:18,310 --> 00:39:24,290
There are also opportunities to use technology to strengthen the deconfliction process.

377
00:39:24,290 --> 00:39:26,930
For example, White Flag is being developed.

378
00:39:26,930 --> 00:39:29,350
It's a protocol developed by the Netherlands.

379
00:39:29,610 --> 00:39:34,670
And that is an encrypted technology to strengthen this deconfliction process.

380
00:39:34,710 --> 00:39:51,690
And that's really important because what we see as we look at the data on this is that often the information that would stop an attack is known somewhere, but there is an information flow problem and it's not reaching the person that is making the decision whether to attack or not.

381
00:39:51,690 --> 00:39:57,630
So if we can improve that, then we can also help reduce attacks on hospitals.

382
00:39:57,770 --> 00:40:01,090
And the protection of hospitals is just one area.

383
00:40:01,090 --> 00:40:07,730
It's an important area, but just one area where technology can be used specifically for humanitarian benefit.

384
00:40:07,730 --> 00:40:14,930
We've also looked at how to reduce civilian casualties, how to detect civilian casualties, which is also an issue.

385
00:40:14,930 --> 00:40:20,650
Oftentimes you'll hear you'll hear militaries say, you know, they did a strike.

386
00:40:20,650 --> 00:40:25,010
There are accusations about civilian casualties and they say, nope, we didn't do it.

387
00:40:25,670 --> 00:40:32,970
And what the data says is oftentimes they did, but they're just not very good at detecting civilian casualties.

388
00:40:33,450 --> 00:40:38,690
So, and if you can't detect them, then you don't know you have a problem to solve.

389
00:40:38,690 --> 00:40:43,450
So this is also a really important issue that can be helped by technology.

390
00:40:43,450 --> 00:40:47,790
There's also anticipating and avoiding reverberating long-term effects of attacks.

391
00:40:47,790 --> 00:40:53,730
So essential services, that's something that militaries around the world really do poorly.

392
00:40:53,730 --> 00:40:55,350
And that could also be helped.

393
00:40:55,350 --> 00:40:59,770
And then there's also traceability and accountability for violations.

394
00:41:00,110 --> 00:41:10,600
The New York Times had a really powerful article on how Russia and the Syrian regime were specifically targeting hospitals in Syria.

395
00:41:10,930 --> 00:41:23,730
And we can potentially use technology to not only, not only avoid things, but also provide more accountability, provide basically the proof that says, Hey, this is the smoking gun.

396
00:41:23,730 --> 00:41:28,430
You guys did this and you can't just argue, you know, fog of war.

397
00:41:29,550 --> 00:41:36,870
So these are just some examples of how technology can be used for, for good and for humanitarian benefits.

398
00:41:37,270 --> 00:41:38,730
So the next slide, please.

399
00:41:39,130 --> 00:41:54,730
Hey, Larry, can I jump in with a good point that somebody made in one of the chats about what you're talking about here, which is that, you know, if you have these, these algorithms that are designed to detect red crosses and then not shoot at that thing,

400
00:41:54,730 --> 00:41:54,890
right?

401
00:41:54,890 --> 00:42:01,530
The question is, well, doesn't that mean that all of the bad guys are just going to start putting red crosses on, on their, on their stuff?

402
00:42:03,790 --> 00:42:04,510
Sure.

403
00:42:04,510 --> 00:42:06,330
And I mean, yeah.

404
00:42:06,330 --> 00:42:17,110
So, so there, there, there are states and actors that try to follow the law, the law of war, internationally humanitarian law, like the genomic conventions.

405
00:42:17,110 --> 00:42:18,790
And then there are some that don't.

406
00:42:18,790 --> 00:42:23,150
That's actually why I included that last piece there, traceability and accountability.

407
00:42:23,910 --> 00:42:48,170
Because with, with, you know, with technology, you can also better document how actors, and that may be a state like Russia, or it may be a non-state armed group, but if they are intentionally flouting requirements of the law, you now have documented this and you can put more pressure on them to,

408
00:42:48,170 --> 00:42:49,670
to behave differently.

409
00:42:51,910 --> 00:42:57,090
And try to use different ways to, to, to positively influence them.

410
00:42:57,090 --> 00:42:57,290
Yeah.

411
00:42:57,290 --> 00:43:10,870
And, and one of the things I think you're getting at when you're talking about flouting the law is that it is, goes against international humanitarian law to put those red cross symbols, red crescent symbols on something that is not actually a protected site.

412
00:43:10,870 --> 00:43:17,290
So, so some bad actors, right, would be doing that even, even without AI technology.

413
00:43:17,290 --> 00:43:18,450
Right.

414
00:43:18,570 --> 00:43:19,190
Right.

415
00:43:19,190 --> 00:43:21,850
Yeah, certainly, certainly they can, they can do that now.

416
00:43:21,850 --> 00:43:23,410
And sometimes they do.

417
00:43:23,950 --> 00:43:29,490
That is, that is not, that is not an excuse to, to just be reckless.

418
00:43:29,570 --> 00:43:30,590
Right.

419
00:43:31,630 --> 00:43:35,850
So, so we need, still need to, to have, to strengthen our protection.

420
00:43:36,450 --> 00:43:43,510
But at the same time, this is also an opportunity to avoid the bad behavior that we see today.

421
00:43:46,410 --> 00:43:47,350
Okay.

422
00:43:47,430 --> 00:43:52,490
So finally, I mean, so the military cares about humanitarian issues.

423
00:43:52,490 --> 00:43:58,030
But at the same time, there's, there's not a lot of, of strong leadership on this issue.

424
00:43:58,030 --> 00:44:03,170
And that's, that's really because of a combination of formal responsibilities.

425
00:44:03,170 --> 00:44:14,810
If you look organizationally at DOD, you know, they have, they have people in charge of lots of different things, you know, developing weapons systems, developing, you know, lethal capabilities.

426
00:44:15,070 --> 00:44:22,350
There's really not that same kind of leadership on, hey, how can we, how can we get better at humanitarian issues?

427
00:44:22,890 --> 00:44:31,890
At the same time, there, there, there has been, there have been leaders who have really focused on this issue, but then they rotate out.

428
00:44:31,890 --> 00:44:35,150
And so it does, long lasting focus.

429
00:44:35,590 --> 00:44:38,150
So, so what, what can we do?

430
00:44:38,150 --> 00:44:44,970
History shows that non-governmental influences can strongly encourage progress in this area.

431
00:44:44,970 --> 00:44:49,170
And typically that has been through media, you know, media accounts.

432
00:44:49,170 --> 00:45:10,070
And I think of the uncounted article, for example, in 2017, by Asma Khan was one great example of this, how one powerful, well-researched article basically influenced DOD to, to be more introspective and to do kind of internal reviews that actually make real changes.

433
00:45:10,110 --> 00:45:15,230
And civil society, you know, NGOs, academia can have a role.

434
00:45:15,230 --> 00:45:17,810
And then of course the UN and ICRC.

435
00:45:17,870 --> 00:45:26,750
So they have all, at different times, had important contributions to help the military to do better.

436
00:45:26,930 --> 00:45:31,690
And I listed, listed three specific examples.

437
00:45:31,690 --> 00:45:46,630
You know, there are, these are actually examples, real examples where outside influences encourage the military to try harder, both in Afghanistan and that really, they really turned things around and did things differently.

438
00:45:46,630 --> 00:45:48,970
And the civilian casualty numbers went down.

439
00:45:49,390 --> 00:46:01,510
Drone strikes and counter-terrorism, you know, leading to the presidential policy directive, the presidential policy guidance issued by President Obama for counter-terrorism operations.

440
00:46:01,570 --> 00:46:14,310
And then the campaign in Iraq and Syria, again, there were things like, you know, counted article and also air wars that contributed to greater scrutiny and led to differences in how the U.S.

441
00:46:14,310 --> 00:46:16,050
military has done things.

442
00:46:16,050 --> 00:46:17,590
If you go to the next slide.

443
00:46:18,830 --> 00:46:24,730
So, so bottom line, there, there's a, there's an opportunity here.

444
00:46:24,730 --> 00:46:32,790
So if you combine kind of the things we've been talking about, the community needs the tech sector as they have not before.

445
00:46:32,790 --> 00:46:51,010
So, you know, looking at that graph we had earlier, you know, the, just the top five tech companies in the U.S., their research and development, you know, now 15 times larger than the entire tech development budget of the United States government.

446
00:46:51,290 --> 00:46:53,910
And that ratio is increasing over time.

447
00:46:53,910 --> 00:46:55,170
So the U.S.

448
00:46:55,170 --> 00:47:01,210
government sees it needs the tech sector to be competitive.

449
00:47:01,410 --> 00:47:06,310
And you see this in DoD wanting to work with the tech sector and they have a number of initiatives.

450
00:47:06,310 --> 00:47:07,590
There's the defense innovation board.

451
00:47:07,590 --> 00:47:12,870
And of course they developed the AI principles that DoD then accepted.

452
00:47:12,930 --> 00:47:22,450
You have the defense innovation unit, defense digital service, and then there are other initiatives that DoD is trying to engage more with the tech sector.

453
00:47:22,550 --> 00:47:27,310
So we just wanted to highlight this is an opportunity for the tech sector, right?

454
00:47:27,310 --> 00:47:31,130
So this is an opportunity to engage and have influence.

455
00:47:31,430 --> 00:47:36,090
So as DoD says, Hey, we want to talk, we want to work together.

456
00:47:36,590 --> 00:47:45,330
There are opportunities for tech companies and people working in the tech sector to influence the areas for engagement and for cooperation.

457
00:47:45,470 --> 00:47:51,510
And also voice concerns about how these issues can be addressed.

458
00:47:51,510 --> 00:48:05,430
And then, you know, I mentioned, you know, right now there's academia, there's a civil society, there's a UN that are engaging now with DoD on discussions and humanitarian issues.

459
00:48:05,430 --> 00:48:08,290
I was in such a discussion yesterday, right?

460
00:48:08,290 --> 00:48:13,770
There was nobody in the tech sector in that discussion, but there could be, right?

461
00:48:13,770 --> 00:48:16,990
As a stakeholder and as a potential solution.

462
00:48:17,570 --> 00:48:19,670
So this is an opportunity.

463
00:48:19,670 --> 00:48:23,670
And then I also just wanted to highlight, this is not just for the US, right?

464
00:48:23,670 --> 00:48:27,230
So I mentioned Australia doing work.

465
00:48:27,810 --> 00:48:35,710
Also the Netherlands are working on this protocol, white flag, white flag is an open source solution.

466
00:48:35,990 --> 00:48:51,550
So it's also, it doesn't have to be a proprietary solution for the US or it could be other open source solutions that help address humanitarian issues that basically could be offered to all parties to conflict.

467
00:48:52,390 --> 00:48:56,250
So I think that's, if you go to the next slide.

468
00:48:56,990 --> 00:49:02,730
So now we'd love to have a discussion and we also have contact information here.

469
00:49:02,990 --> 00:49:18,410
So we'd love to continue to answer questions or things you'd like to raise, but also in the future, we'd love to hear from you and we can continue those discussions.

470
00:49:18,450 --> 00:49:21,670
So open to any questions you may have.

471
00:49:21,710 --> 00:49:22,870
Yeah, absolutely.

472
00:49:22,870 --> 00:49:25,810
I'll certainly echo what Larry has said there.

473
00:49:26,410 --> 00:49:30,290
And so I've been reading through the comments that people have been posting.

474
00:49:30,290 --> 00:49:36,570
So I'm just going to go ahead and kind of read off the things that I've seen there.

475
00:49:37,050 --> 00:49:45,250
So one of the things that kind of the main thing that a couple of different folks have raised are concerns around proliferation, right?

476
00:49:45,250 --> 00:49:56,510
So one person raised a worry, and this is very timely, a worry about the trickle down of military technology being used in domestic law enforcement, right?

477
00:49:56,510 --> 00:50:00,350
Or being used abroad to stifle dissent.

478
00:50:00,850 --> 00:50:15,390
There was also some concerns brought up about, well, if other, say, bad actor nations get these technologies, that's something that's going to serve as a force multiplier for those bad bad actors, dictators or whatever.

479
00:50:15,390 --> 00:50:26,590
And so a mention of, let's see, so the person wrote in that the best way to keep control of technology is to not develop it.

480
00:50:26,590 --> 00:50:29,970
And there's another mention of mutually assured destruction.

481
00:50:30,970 --> 00:50:36,850
So there's kind of a lot baked up in those proliferation concerns.

482
00:50:36,850 --> 00:50:52,350
I might start to address those by saying that certainly thinking about this technology and how we want to move forward with thinking about exporting or sharing it is absolutely something that DOD and the U.S.

483
00:50:52,350 --> 00:50:56,970
government is taking very seriously and really thinking about right now.

484
00:50:58,090 --> 00:51:07,650
Another thing I'll... and so, again, it's going to be one of these contexts where there's some hard challenges, you know, wrapped up in there.

485
00:51:07,650 --> 00:51:20,970
But at the same time, right, keep in mind that, you know, as we talked about with that earlier slide with the graph on it, most of the R&D that's going on in this space is not happening by state actors.

486
00:51:20,970 --> 00:51:22,550
It's not happening by the U.S.

487
00:51:22,550 --> 00:51:23,830
government or DOD.

488
00:51:23,830 --> 00:51:30,670
So that's a reality to keep in mind, thinking about proliferation also.

489
00:51:31,790 --> 00:51:53,270
And one other thing I'll mention, just thinking about the domestic law enforcement context, is just be aware that a lot of the rules and laws that DOD has to follow and what it makes are... so domestic law enforcement is not subject to those laws.

490
00:51:53,270 --> 00:52:04,770
So , again, I think there are things to worry about here, but I'm not sure if DOD's development of these technologies... that may not be the biggest concern in some of these cases.

491
00:52:05,590 --> 00:52:08,970
Larry, would you add anything on any of those topics?

492
00:52:10,350 --> 00:52:15,170
So, yeah, so great questions.

493
00:52:15,170 --> 00:52:27,430
One thing I wanted to mention, especially for the export and proliferation piece, because I've spent a couple years at state where this was a really key concern.

494
00:52:27,430 --> 00:52:34,730
And one thing that became very apparent to me, in the case of Yemen, so, you know, the U.S.

495
00:52:34,730 --> 00:52:42,990
has provided a lot of arms, you know, weapons systems, intelligence to the Saudi-led coalition in Yemen.

496
00:52:42,990 --> 00:52:48,890
And, you know, there were lots of discussions about, is it legal to do that?

497
00:52:48,890 --> 00:52:54,570
And I think I saw one report kind of summarized it as lawful, but awful.

498
00:52:54,650 --> 00:52:59,610
So the laws and policies that the U.S.

499
00:52:59,610 --> 00:53:08,830
applies to security assistance, providing stuff to other countries, it considers some things.

500
00:53:08,830 --> 00:53:15,390
It considers things like diversion, which is, are you going to give, we want to safeguard that if we give this, if the U.S.

501
00:53:15,390 --> 00:53:18,770
gives this to another country, they won't give it to somebody else.

502
00:53:18,770 --> 00:53:23,910
And there's also something called Leahy law that guards against gross violations of human rights.

503
00:53:23,910 --> 00:53:29,270
So if they're doing like war crimes, really horrible things, then that can stop transfers.

504
00:53:29,270 --> 00:53:38,190
But the more general civilian protection considerations that we're talking about really are not well incorporated in security assistance.

505
00:53:38,190 --> 00:53:44,610
And so that is something that the state department is really trying to think about doing.

506
00:53:44,610 --> 00:53:50,790
We're talking to them about, you know, new policies and new ways to think about this.

507
00:53:51,030 --> 00:54:03,230
But one thing that can help is if you can provide equipment, including weapons, that have some of these safeguards, right?

508
00:54:03,230 --> 00:54:08,590
So right now, the Saudi-led coalition, how many hospitals have they struck?

509
00:54:08,590 --> 00:54:09,810
I've lost count.

510
00:54:09,810 --> 00:54:11,950
It's so many, so many, right?

511
00:54:12,770 --> 00:54:18,010
And I mean, you can certainly say, well, we're not going to give them anything, right?

512
00:54:18,010 --> 00:54:22,150
But they've also shown that they'll just buy from the Russians.

513
00:54:22,150 --> 00:54:26,810
So it'd be great to stop them, okay?

514
00:54:26,810 --> 00:54:28,630
I'm in favor of that.

515
00:54:28,930 --> 00:54:42,690
But if war is going to continue, are there things that can be done to make war more humane and safer for the civilians that are in the conflict areas?

516
00:54:42,690 --> 00:54:46,890
And so the answer is yes, and that technology can help.

517
00:54:48,990 --> 00:54:54,750
And that can certainly be the case for exports as well.

518
00:54:55,490 --> 00:55:05,090
And as I'm reading the chats coming in, I'll just say that folks are kind of doubling down on concerns about domestic law enforcement again, right?

519
00:55:05,090 --> 00:55:19,670
And so predator drones at protests, you know, an interest in, you know, can we have voting where we select that we're gonna not fund these types of technologies?

520
00:55:19,670 --> 00:55:23,150
Maybe that's in a military or in a law enforcement context.

521
00:55:25,550 --> 00:55:30,690
So let's see, and a note that killer robots are perfect for war crimes.

522
00:55:30,690 --> 00:55:32,330
So back to the armed conflict case.

523
00:55:32,330 --> 00:55:45,470
So I'll say again, with respect to the domestic law enforcement front, right, we're just seeing that, you know, some of this is not, maybe not necessarily an issue limited to AI developments, right?

524
00:55:45,470 --> 00:55:46,770
We're seeing it now.

525
00:55:47,030 --> 00:55:51,650
It's a real problem in domestic law enforcement.

526
00:55:51,650 --> 00:56:03,230
And so, you know, it's important that we fix the laws and fix the practices that are in place now, even before we have sort of new classes of weapons that the military is using.

527
00:56:03,930 --> 00:56:22,950
Um, let's see one, another comment that somebody wrote into the chat was, was concerned about AI vulnerabilities, right, being especially problematic in this context, they mentioned adversarial attacks, data poisoning, right.

528
00:56:22,950 --> 00:56:31,690
So these are, so these are getting technical challenges, right, that are going to be hard challenges to to solve, right.

529
00:56:31,690 --> 00:56:39,430
So this goes back to, there are a lot of challenges and just getting getting the technology right to actually make these applications work.

530
00:56:40,010 --> 00:56:47,610
And so I guess one thing I would say here is is that, that, you know, yeah, these are hard problems.

531
00:56:47,610 --> 00:56:53,730
I think we, you know, we, DoD will need the tech sector, private industries help to get good solutions here.

532
00:56:53,730 --> 00:57:10,210
And in cases where we can't be confident in these technologies, especially in in lethal, or weapons contexts, those are cases where we need to not not deploy systems, right, unless we can, unless we can be confident with how they're going to operate.

533
00:57:13,710 --> 00:57:22,350
Can I make one comment about the, you know, I do hear this a lot about, you know, killer robots are perfect for war crimes, right.

534
00:57:22,350 --> 00:57:27,050
And so I think it's useful kind of just to build a framework here.

535
00:57:27,050 --> 00:57:32,150
So there are two kinds of parties that are involved in conflict.

536
00:57:32,150 --> 00:57:34,950
Some can be states, or it can be armed groups.

537
00:57:35,570 --> 00:57:40,390
One is parties that want to follow the law.

538
00:57:40,490 --> 00:57:40,810
Okay.

539
00:57:40,810 --> 00:57:44,530
And then there are some that don't, they show impunity.

540
00:57:44,850 --> 00:57:49,050
And so I think technology applies differently to the two of them.

541
00:57:49,050 --> 00:58:05,030
So those that that wants to apply the law, then this kind of technology can be useful for, for helping, helping them to better comply with the law and their legal obligations.

542
00:58:05,030 --> 00:58:11,350
But even beyond the law, because in the US, we talked about there are legal things we're required to do.

543
00:58:11,350 --> 00:58:16,510
But then we also want to try even harder than what we're required to do, right.

544
00:58:16,510 --> 00:58:19,990
The law is just the basic requirements.

545
00:58:20,630 --> 00:58:25,670
Diane mentioned the, the US civilian casualty executive order in 2016.

546
00:58:26,370 --> 00:58:29,210
That goes beyond what is required by the law.

547
00:58:29,210 --> 00:58:33,910
It says, you know, there are things we're required to do, but we're going to do more because it's important.

548
00:58:34,290 --> 00:58:43,030
So, so for the group of states or parties that want to comply with the law, you can help comply with the law.

549
00:58:43,030 --> 00:58:44,490
And then you can even go further.

550
00:58:44,490 --> 00:58:46,450
And we talked about some ways to do that.

551
00:58:46,450 --> 00:58:50,250
Then there's others that don't comply with the law, right.

552
00:58:50,250 --> 00:58:52,210
And there's the Russians and the Syrians.

553
00:58:52,210 --> 00:58:56,570
And, you know, there are different, different, you know, different terrorist groups out there.

554
00:58:56,570 --> 00:58:57,770
They just don't care.

555
00:58:57,770 --> 00:59:01,570
They exploit the law for their own benefits.

556
00:59:02,230 --> 00:59:13,790
And in those cases, technology can also be used for accountability and to traceability and to try to highlight some of the things that they do.

557
00:59:13,810 --> 00:59:29,650
Honestly, you know, there were, there were things I remember in the counter ISIS conflict in, in Iraq and Syria, ISIS did horrible, horrible things, and most people have no idea.

558
00:59:29,650 --> 00:59:30,450
Right.

559
00:59:30,490 --> 00:59:46,170
And so, and just better, better showing those things can help in themselves to influence those groups, just like the New York Times article, talking about the Russians and delivery striking hospitals.

560
00:59:46,170 --> 00:59:46,730
Okay.

561
00:59:46,730 --> 00:59:48,790
They, they did that quietly.

562
00:59:48,990 --> 00:59:53,390
When the New York Times came out, Russia started to squirm, right?

563
00:59:53,390 --> 00:59:55,630
Because they don't want to be known for that.

564
00:59:55,630 --> 00:59:58,890
They want to do it for various interests.

565
00:59:58,890 --> 01:00:07,310
But when you start putting the spotlight on them, that can help, that can help influence them in themselves.

566
01:00:07,310 --> 01:00:15,250
We should also be thinking about how can we better use that information to really deter them from doing those terrible things.

567
01:00:15,250 --> 01:00:19,030
So just wanted to kind of give that overall framework for war crimes.

568
01:00:20,190 --> 01:00:29,970
So Larry, another comment that somebody wrote in that I think is a really interesting one is a question of whether, right, so DOD has these AI policies.

569
01:00:30,570 --> 01:00:32,770
Should, should those be codified in law?

570
01:00:32,770 --> 01:00:36,610
Because it's, it's really easy to change norms and guidelines.

571
01:00:36,830 --> 01:00:38,690
What would you say to that?

572
01:00:43,620 --> 01:00:57,700
So, so I remember actually having a conversation with Eric Schmidt about, he's part of the, the National Security Commission for AI.

573
01:00:57,700 --> 01:01:02,820
And so there was this discussion about, you know, law versus policy.

574
01:01:02,820 --> 01:01:19,240
And, you know, the tricky thing is that it is so early that oftentimes we don't even really know what we're talking about, you know, and, and these kinds of conversations are so valuable because we're, we're, we're trying to figure out what is really important.

575
01:01:19,240 --> 01:01:22,660
And that, that's one reason why we've been asking these questions too, right?

576
01:01:22,660 --> 01:01:24,060
You know, what do you think is important?

577
01:01:24,060 --> 01:01:26,380
Because we want to hear from, from all of you.

578
01:01:27,040 --> 01:01:33,760
And, but, but trying to make a law too soon has risks, right?

579
01:01:33,760 --> 01:01:38,020
Because then you've sort of codified your limited understanding.

580
01:01:38,380 --> 01:01:43,720
And it's really, it's harder to change law than it is to change policy.

581
01:01:43,720 --> 01:01:51,160
So, so in these early stages and really, you know, DoD has not done much.

582
01:01:51,160 --> 01:01:52,560
They've done a lot of talking.

583
01:01:52,560 --> 01:01:54,780
There really has not been a lot.

584
01:01:55,240 --> 01:02:02,500
Diane mentioned the, the DoD directive on, on autonomous weapons that was created in 2012.

585
01:02:02,500 --> 01:02:10,080
And it calls for the senior level review for additional scrutiny for systems that would meet certain requirements.

586
01:02:10,080 --> 01:02:14,060
They haven't done a single one because they're just not there yet.

587
01:02:14,100 --> 01:02:21,800
So I think... And just, they haven't done a single one because, because nobody's made it, nobody's tried, nobody's asked, right?

588
01:02:21,800 --> 01:02:23,560
Just to, just to clarify that.

589
01:02:23,680 --> 01:02:24,460
Yeah.

590
01:02:26,320 --> 01:02:26,980
Yeah.

591
01:02:27,020 --> 01:02:38,060
So I think, you know, it's, it's kind of easy to say, you know, yeah, there's so much that can be done, but in reality, militaries move slowly.

592
01:02:38,060 --> 01:02:39,760
They're built to move slowly.

593
01:02:39,760 --> 01:02:46,240
They're these huge bureaucratic structures that are designed to reduce risk.

594
01:02:46,240 --> 01:02:47,860
And so they move really slowly.

595
01:02:47,860 --> 01:02:52,900
And so I think at this point, we, we don't necessarily need a law.

596
01:02:52,900 --> 01:02:56,800
We need to actually figure out what the important things are.

597
01:02:57,200 --> 01:02:59,380
And we're not, we're not even there yet.

598
01:02:59,380 --> 01:03:07,260
So, so that's why we really appreciate this discussion today to try to help figure out what are the things we really need to be worried about.

599
01:03:08,060 --> 01:03:08,860
Yeah, thanks.

600
01:03:08,860 --> 01:03:22,500
And, and I'm tracking, just looking through the chat here, I think there's, there's, there's one more comment that I'm seeing that I don't think we've, we've addressed yet that I'll read off here, which is just that I think the person who recommended this,

601
01:03:22,500 --> 01:03:29,680
this report that you talked about earlier, so this, so they, they raised a few issues from that report.

602
01:03:30,000 --> 01:03:33,000
This was the vulnerabilities, AI vulnerabilities issue.

603
01:03:33,000 --> 01:03:41,760
They also raised the point that there's a dynamic where offense is easier than defense in some of the space.

604
01:03:41,760 --> 01:03:53,880
And so I think that, that one thing I would, you know, I would say two things there, right, which is that, you know, one is that I don't, I don't think that's special of the AI.

605
01:03:54,140 --> 01:04:07,360
And I would also say that, you know, we did some research a little while back looking at lethal autonomy, you know, potential dynamics or uses within, hypothetical uses within the military.

606
01:04:07,700 --> 01:04:26,740
And, you know, back to this, a drone swarm attacking a ship example, I would say just don't discount that, that, you know, certainly it's right that in a lot of contexts, the offense is easier than defense, but there are still really important defensive contexts where technologies like this can be really helpful.

607
01:04:26,740 --> 01:04:31,060
And, and those are cases where we're talking about, you know, protecting the lives of our own soldiers.

608
01:04:31,060 --> 01:04:39,140
So they're ones that we, you know, I think as a country and certainly our military regards very highly and is very important.

609
01:04:40,280 --> 01:04:45,620
If I can just kind of add to that, the, the context thing is really important.

610
01:04:45,620 --> 01:04:52,720
So, you know, I was, I was part of the US delegation to the UN and in some of these talks on lethal autonomous systems.

611
01:04:52,840 --> 01:04:59,020
And one thing that was often lost is that the context really is important.

612
01:04:59,020 --> 01:05:13,620
So, so you have some, some people would talk about, well, how will you use a lethal autonomous weapons system in an urban environment against, you know, against groups that aren't wearing uniforms and they're, you know, hiding within the population.

613
01:05:14,160 --> 01:05:20,960
And then, you know, basically the answer of, you know, the US and Australia and the UK was like, we won't.

614
01:05:21,220 --> 01:05:31,360
And what, what we really, what they kind of want this stuff for is, is for missile defense in, in a maritime setting, right?

615
01:05:31,360 --> 01:05:43,000
So there's no civilians anywhere around but they have hypersonic missiles coming in and they want, they want to have capability that helps them to, to better defend against those kinds of things.

616
01:05:43,000 --> 01:05:54,980
So, so just, just recognizing that there are some contexts where you can use some of this technology with a much lower risk to civilians than others.

617
01:05:55,300 --> 01:06:00,640
And just having a more nuanced conversation where we recognize that.

618
01:06:00,660 --> 01:06:03,460
And, and, you know, militaries need to recognize that too, right?

619
01:06:03,460 --> 01:06:14,300
They do internal reviews where they try to recognize risk and the context is really important for those, those decisions on whether they think it's okay or not to use them.

620
01:06:14,900 --> 01:06:15,940
Yeah, thanks.

621
01:06:15,940 --> 01:06:21,140
And so one more person wrote in a good comment that, that I'll read off here, right?

622
01:06:21,140 --> 01:06:23,320
Which is the concerns about, right?

623
01:06:23,320 --> 01:06:27,440
AI needs big data a lot of times to, to work.

624
01:06:27,440 --> 01:06:30,500
And so what about data security there?

625
01:06:31,760 --> 01:06:34,260
So do you have any, any comments on that, Larry?

626
01:06:34,340 --> 01:06:43,940
So I think this is, this is, I mean, we talked about, there are some risks that, that, you know, DoD is like, is really handling well.

627
01:06:43,940 --> 01:06:47,200
And then there are some that I think they're behind.

628
01:06:47,200 --> 01:06:49,220
And I think this is one of them.

629
01:06:49,620 --> 01:06:54,680
And it's, you know, we talked about AI and applications.

630
01:06:54,680 --> 01:07:03,700
I think this is one reason why, you know, DoD is not as, as making as much progress as people may think.

631
01:07:04,100 --> 01:07:10,200
Because there are all these enablers that they need to have enabled to use AI effectively.

632
01:07:10,200 --> 01:07:14,000
And the data situation with DoD is really problematic.

633
01:07:14,560 --> 01:07:17,580
They, they, they have lots of different networks.

634
01:07:17,860 --> 01:07:20,620
You know, they're not necessarily secure.

635
01:07:20,620 --> 01:07:22,300
They don't necessarily talk to each other.

636
01:07:22,300 --> 01:07:28,440
Well, they're, they're not always formatted in a consistent way or quality control.

637
01:07:28,440 --> 01:07:34,260
And so that makes it really difficult to actually use that data in a, in a constructive way.

638
01:07:34,820 --> 01:07:40,000
So, so DoD is going to have to get a handle on that.

639
01:07:40,000 --> 01:07:45,100
And part of, part of that is going to be trying to make data more secure.

640
01:07:45,100 --> 01:07:58,160
I will say in the few serious AI applications that, that I'm aware of, there has been kind of this recognition, both of they need to be careful about data poisoning.

641
01:07:58,560 --> 01:08:06,840
And, and that has implications both on storage, but also on who you get to do your, your data processing, right?

642
01:08:06,840 --> 01:08:15,480
You can't just necessarily outsource this to some commercial organization that may, may or may not be the most trustworthy.

643
01:08:16,040 --> 01:08:17,500
And then the storage.

644
01:08:17,500 --> 01:08:19,020
So it's a great question.

645
01:08:19,020 --> 01:08:19,700
I think.

646
01:08:21,860 --> 01:08:22,740
Yeah.

647
01:08:23,340 --> 01:08:23,620
Okay.

648
01:08:23,620 --> 01:08:25,980
Well, thank you guys so much.

649
01:08:25,980 --> 01:08:30,320
We really appreciate hearing your, your thoughts and comments.

650
01:08:30,320 --> 01:08:34,680
I'll reiterate what Larry said a minute ago, which is the, please feel free to reach out to us.

651
01:08:34,680 --> 01:08:39,960
You can either do it through discord or through our email addresses that we put up a few minutes ago.

652
01:08:39,980 --> 01:08:53,840
We would love to hear, you know, if there's, if there's more that you want to talk about that we didn't get to, again, we would, you know, we, we don't have all the answers, but we are really interested in hearing your thoughts and concerns, even if they're not things that are,

653
01:08:53,840 --> 01:08:55,480
can be perfectly answered today.

654
01:08:58,000 --> 01:09:01,760
So, so thank you again, Ethics Village for hosting us.

655
01:09:02,400 --> 01:09:06,280
And I really appreciate everybody's, everybody's participation here.

656
01:09:08,520 --> 01:09:09,480
Awesome.

657
01:09:09,480 --> 01:09:15,880
We'd like to, again, thank you to our speakers, Dan and Larry for our very first talk of day two.

658
01:09:16,160 --> 01:09:22,040
For those of you that are here for the rest of the few hours, stay tuned for our next few talks.

659
01:09:22,040 --> 01:09:22,980
Our next talk will be from our guest speaker, Dan.

660
01:09:22,980 --> 01:09:22,980
So, Dan, thank you so much for