1
00:00:00,600 --> 00:00:01,820
All right, welcome.

2
00:00:01,820 --> 00:00:16,840
This is the AI Village panel on AI ethics, where we're going to discuss the ethical implications of AI technologies and especially their deployment both in general and also specifically in data science and computer security.

3
00:00:16,840 --> 00:00:22,820
We have four lovely panelists from a variety of backgrounds here to discuss the topic with us.

4
00:00:22,820 --> 00:00:31,140
Catherine is a HackerMOMA scholar and artist who focuses on feminist technology, data literacy, and civic engagement.

5
00:00:31,480 --> 00:00:40,900
She wrote a 2020 book called Data Feminism, co-authored with Lauren Klein, where she charts a course for more ethical and empowering data science practices.

6
00:00:41,160 --> 00:00:51,900
Catherine is an assistant professor of urban science and planning at the Department of Urban Studies and Planning at MIT, where she's also the director of their Data Plus Feminism Lab.

7
00:00:52,120 --> 00:00:59,700
Ram is a data cowboy within the trustworthy ML team at Microsoft, working on the intersection of machine learning and security.

8
00:00:59,700 --> 00:01:04,700
His team's goal is to empower engineers to develop and deploy ML systems securely.

9
00:01:04,740 --> 00:01:12,320
Ram is also an affiliate at the Berkman Klein Center at Harvard University, where he explores the societal implications of adversarial AI.

10
00:01:12,900 --> 00:01:22,180
Ai-Lin is an assistant professor of computer science at George Washington University and at the Institute for Data, Democracy, and Politics.

11
00:01:22,180 --> 00:01:30,940
She is interested in AI ethics, bias in AI, machine learning, and the implications of machine learning on privacy, security, fairness, and society.

12
00:01:30,940 --> 00:01:39,340
Much of her research is on methods to interpret AI and gain insight into the bias in both artificial and natural intelligence.

13
00:01:39,680 --> 00:01:49,860
And she has worked to demonstrate how human-like biases can crop up in natural language analysis due to the statistical regularities of language.

14
00:01:50,080 --> 00:01:56,860
And finally, Anima is the Bren Professor at Caltech and the Director of ML Research at NVIDIA.

15
00:01:56,860 --> 00:02:01,220
She was previously a principal scientist at Amazon Web Services.

16
00:02:03,040 --> 00:02:05,860
She's received several honors, such as the Alfred P.

17
00:02:05,860 --> 00:02:12,820
Salone Fellowship, NSF Career Award, Young Investigator Awards from DoD, and several more.

18
00:02:12,820 --> 00:02:23,880
She is part of the World Economic Forum's Expert Network and is passionate about designing principled AI algorithms and applying them to interdisciplinary outcomes.

19
00:02:23,880 --> 00:02:33,160
She has been outspoken on AI bias and the need for democratization and focuses her research on unsupervised AI optimization and tensor methods.

20
00:02:33,620 --> 00:02:42,540
They are all wonderful people and I can personally vouch for the fact that super cool research and I'm more than thrilled to talk with them all.

21
00:02:42,600 --> 00:02:53,900
So to kind of get us started, there's been a lot of conversation, at least recently, about AI bias and misuse of AI in the news in the United States recently.

22
00:02:53,900 --> 00:03:03,040
A lot of tech companies have talked about how they're going to stop using facial recognition algorithms or stop providing them to police to use.

23
00:03:03,040 --> 00:03:19,420
And there's also been a lot of conversation both in the academic literature and also in the national news media about biased AI, the AIs that learn to be racist because they look at racist data or learn to be sexist because they look at sexist data.

24
00:03:19,780 --> 00:03:22,600
But AI ethics is kind of a much broader topic than that.

25
00:03:22,600 --> 00:03:30,140
So to kick us off, can each of you say a little bit about what the scope of the field is to you and how it relates to the work that you do?

26
00:03:30,660 --> 00:03:32,800
Ram, why don't you give us a start?

27
00:03:33,040 --> 00:03:34,440
Oh, thank you.

28
00:03:34,440 --> 00:03:38,100
First of all, I'd like to say that I'm not an AI ethics expert.

29
00:03:38,100 --> 00:03:43,280
I'm so glad that Anima, Catherine, Eileen are here along with Stella.

30
00:03:43,460 --> 00:03:48,180
So you should be getting like solid like stellar advice from them.

31
00:03:48,180 --> 00:04:00,120
So whatever I'm going to talk is through my lived experience and interacting with folks like Kendra Albert, Jonathan Penny from Berkman and along with Mary Gray and a whole host of other folks.

32
00:04:00,120 --> 00:04:04,400
So that's my, you know, that's my preface for this.

33
00:04:04,520 --> 00:04:12,740
So for me, like I've been looking at Berkman at the, you know, what is adversarial ML?

34
00:04:12,740 --> 00:04:15,780
How does that impact the experience of citizenship?

35
00:04:16,040 --> 00:04:34,220
So one of the things that I, you know, for me, like how I, you know, how we've been interpreting this topic is moving away from just like an, you know, like the constraint of like ethics and thinking about power structures, especially in the context of adversarial machine learning.

36
00:04:34,220 --> 00:04:47,560
So turns out that, you know, machine learning is political in a Langdon Winner sense and security has a lot of like implications, like, you know, if you think of the moral character of cryptography by Rod Wave.

37
00:04:48,160 --> 00:04:55,600
And adversarial machine learning, which is what I work on as a researcher , sits in the intersection of these two topics.

38
00:04:55,600 --> 00:05:01,880
So thinking through who is securing like the ML systems?

39
00:05:01,880 --> 00:05:03,960
What is securing?

40
00:05:03,960 --> 00:05:05,200
Who are we securing it against?

41
00:05:05,200 --> 00:05:06,540
Who is the attacker?

42
00:05:06,540 --> 00:05:12,360
Are we, is the attacker just like anybody who compromises the confidentiality, integrity, and availability?

43
00:05:12,720 --> 00:05:16,980
So those are kind of the broader questions that I'm looking at at Berkman.

44
00:05:18,040 --> 00:05:19,780
All right, Ellen.

45
00:05:22,140 --> 00:05:24,800
Oh, Stella, thank you for the introduction.

46
00:05:24,800 --> 00:05:33,240
And I'm very excited to be part of this amazing panel so that I will get to hear all of your ideas and vision on this topic.

47
00:05:33,240 --> 00:05:46,660
My research focuses on, working on the implications of machine learning on privacy and security.

48
00:05:46,660 --> 00:05:57,240
Since machine learning or AI technologies are double-edged swords, when they have security enhancing properties, they usually end up infringing upon our privacy.

49
00:05:57,240 --> 00:06:13,040
And starting from there, I started slowly seeing the link between privacy and fairness and how when our data is exploited, abused, or collected by third parties, it can lead to fairness concerns at the group level.

50
00:06:13,100 --> 00:06:32,360
And nowadays I'm focusing on understanding how AI systems learn bias or essentially how systems can learn some form of intelligence or learning ability and how that relates to bias in natural and artificial intelligence.

51
00:06:32,360 --> 00:06:57,300
And when we looked at natural language systems, for example, word embeddings or today's neural language models that can generate text that is almost human level, we see that these systems are learning human-like biases from society very accurately, especially when compared to validated social psychological methods such as implicit association tests.

52
00:06:57,380 --> 00:07:07,420
And then these systems are deployed at the world scale impacting billions of people every single day with their consequential decisions or biased outputs.

53
00:07:07,420 --> 00:07:16,480
And this ends up turning into this AI bias lifecycle, which is also a feedback cycle of AI ethics as well.

54
00:07:16,480 --> 00:07:42,200
And I'm interested in understanding all of the components in this AI ethics lifecycle, the role humans play in this lifecycle, how we can discover biases or harmful side effects of these AI systems, and what we can do to mitigate these harmful effects and prepare for upcoming problems that might have already been caused by AI systems.

55
00:07:42,200 --> 00:07:56,820
And during today's panel, we will go into the details of all of these single components, starting from the most technical atomic details of machine learning, going up to probably some philosophical questions at the high level abstraction.

56
00:07:58,500 --> 00:08:00,340
Thank you Aylin.

57
00:08:00,520 --> 00:08:01,700
Anima?

58
00:08:01,900 --> 00:08:06,640
Yeah, I completely agree with Aylin and Ram.

59
00:08:06,640 --> 00:08:22,940
You know, it's to me exciting times to be an AI researcher, but also, right, I'm filled with dread that where the world is moving and how fast we are, you know, moving in directions that I find problematic.

60
00:08:22,940 --> 00:08:30,020
And to me, the source of it all comes to the foundations under which this deep learning revolution is built on, right?

61
00:08:30,020 --> 00:08:39,120
So I'd like to emphasize when I'm giving talks on AI about this trinity, you know, the data, algorithms, and computer infrastructure coming together.

62
00:08:39,120 --> 00:08:46,040
And that's what allowed us to have this fast progress in deep learning.

63
00:08:46,160 --> 00:08:50,040
But most of that progress has been with supervised learning, right?

64
00:08:50,040 --> 00:08:56,440
The availability of large-scale label data starting from the ImageNet data set was the start of this revolution.

65
00:08:56,660 --> 00:08:59,400
And in a way, that is quite straightforward.

66
00:08:59,400 --> 00:09:07,280
Once you know that you want to optimize for accuracy and you have the label data, you can kind of quickly build these systems.

67
00:09:07,280 --> 00:09:11,960
But the problem is sometimes we want not just the accuracy, but like robustness, right?

68
00:09:11,960 --> 00:09:16,720
Like what Ram was saying, what are the different changes beyond the training data?

69
00:09:16,720 --> 00:09:24,460
And when our training data is biased, how does this affect on minority populations that are not represented in training data?

70
00:09:24,460 --> 00:09:29,360
And part of it is not just the data imbalance, but also the algorithmic bias.

71
00:09:29,500 --> 00:09:38,480
You know, if you are on Twitter like me, and if you follow the Twitter storm on the presence of algorithmic bias on deep learning, you know what I'm talking about.

72
00:09:39,460 --> 00:09:48,840
And just to the gist of it, the idea is I think some proponents of deep learning do not want anything negative being said around deep learning.

73
00:09:48,840 --> 00:09:54,840
But I do think these very large-scale models, highly nonlinear models, are the source of the problem.

74
00:09:54,840 --> 00:10:01,220
Because the more nonlinear it gets, the more brittle they are, the more they amplify the bias, right?

75
00:10:01,220 --> 00:10:02,620
There is no free lunch.

76
00:10:02,620 --> 00:10:13,360
You know, you cannot have it all in one mechanism unless we are thoughtful in terms of the objectives we build in, the kind of data we use, and also the kind of testing we do.

77
00:10:13,520 --> 00:10:16,320
And so what I work on is unsupervised learning.

78
00:10:16,320 --> 00:10:22,660
And I think to me, this is the core for us to move away from these highly biased models.

79
00:10:22,660 --> 00:10:30,500
You know, to take a simple example, you know, even before the deep learning revolution, I was working on topic models, right?

80
00:10:30,500 --> 00:10:37,800
Where we wanted to categorize large corpus of documents and there's no label data.

81
00:10:37,820 --> 00:10:43,140
And so now with these unsupervised learning, we are suggesting these are possible topics, right?

82
00:10:43,140 --> 00:10:45,940
And these are the words representing the topics.

83
00:10:45,940 --> 00:10:53,720
And then it's up to the human to interpret and judge what are relevant topics, you know, because there are weird correlations in text.

84
00:10:53,720 --> 00:10:56,940
So there can be lots of topics that are not human interpretable.

85
00:10:56,940 --> 00:11:06,280
And so human in the loop with unsupervised learning, I think is the key to get out of this place where we are stuck with biased models.

86
00:11:06,400 --> 00:11:10,200
And, you know, the current revolution now is unsupervised.

87
00:11:10,200 --> 00:11:14,840
Now we're seeing a lot of progress in self-supervised learning and robustness.

88
00:11:14,840 --> 00:11:17,200
Hopefully that'll lead us the way forward.

89
00:11:17,360 --> 00:11:18,360
Thank you.

90
00:11:19,280 --> 00:11:20,040
All right.

91
00:11:20,340 --> 00:11:21,220
Thank you.

92
00:11:21,220 --> 00:11:25,900
And finally, Catherine, where are you coming from with this?

93
00:11:26,360 --> 00:11:27,040
Yeah.

94
00:11:27,040 --> 00:11:28,220
Thanks so much.

95
00:11:28,220 --> 00:11:29,900
Thanks, Stella, for the invitation.

96
00:11:29,900 --> 00:11:32,760
It's a super pleasure to be here with you all.

97
00:11:34,080 --> 00:11:34,380
Yeah.

98
00:11:34,380 --> 00:11:51,980
So I guess where I'm coming from is not as much of a focus on AI, although some of the things that we say, many of the things we say in our book, Data Feminism, apply, but we sort of enlarge the scope more broadly to be data writ large.

99
00:11:52,680 --> 00:12:05,220
And thinking about also, you know, one of the things that's happened as this conversation has grown around fairness, accountability, transparency, bias, ethics, and so on.

100
00:12:05,840 --> 00:12:11,400
You know, there's been a lot of pushback on just like AI and data are great.

101
00:12:11,400 --> 00:12:13,300
They're working great for everybody.

102
00:12:14,100 --> 00:12:31,580
And our contribution with Data Feminism is to say, hey, actually, feminism as a kind of body of knowledge, as a kind of intellectual heritage, which has theorized, philosophized, and also done a lot of writing and sort of action, like social movements,

103
00:12:31,580 --> 00:12:42,860
action around the topic of inequality has a lot to offer us in terms of conceptual models for how we can think about this current state of affairs.

104
00:12:42,860 --> 00:12:50,620
And so that was sort of the premise of Data Feminism is to say, hey, let's look over here at feminism.

105
00:12:50,620 --> 00:13:09,420
What do we, what can we bring from feminism to understand how we might work with data science in a more ethical way, and specifically taking structural inequality, which is basically like power differences between social groups as kind of our main locus of attention.

106
00:13:09,420 --> 00:13:24,660
And so when you look at that, and you look at things like, you know, one of the things I guess that was irritating, I guess I'll say as we were writing the book, which I'm sure folks here are also irritated by, is that often, you know, when we have a biased,

107
00:13:24,660 --> 00:13:37,780
whatever, algorithm or something like this, like a resume screener, let's say, that is favoring men over women, you know, the media reports on it with this thing of like surprise, like, oh my gosh, like, how could this be possible?

108
00:13:37,840 --> 00:13:41,240
Like, how could there be bias in a machine?

109
00:13:42,020 --> 00:13:46,540
And we're like, oh my gosh, Data Feminism has so much to say about this.

110
00:13:46,540 --> 00:13:51,380
And so ultimately, like what the book does is it kind of backs us up.

111
00:13:51,380 --> 00:13:53,500
It's in a way less in the details.

112
00:13:53,500 --> 00:13:59,820
It's not providing you with a way of like systematically fixing your data set or anything like that.

113
00:13:59,820 --> 00:14:07,240
But it backs us up, like way further back in the processing pipeline to say, well, like, here's why these things surface.

114
00:14:07,240 --> 00:14:09,720
They surface because of hundreds of years of inequality.

115
00:14:09,840 --> 00:14:18,800
They surface because of really flawed models of people like women not being considered fully human or people of color not being considered fully human.

116
00:14:18,800 --> 00:14:29,980
And to really take, you know, if we aspire to build something that is ethical, we have to look and we have to undertake that with a different kind of time horizon.

117
00:14:29,980 --> 00:14:32,940
And that has to be built into everything that we do.

118
00:14:32,940 --> 00:14:37,180
So we're trying to kind of both say like, hey, let's look back at the root causes.

119
00:14:37,180 --> 00:14:38,880
What causes structural inequality?

120
00:14:38,880 --> 00:14:46,360
Like, why do we end up with biased, you know, crime data sets in the first place before we start trying to like fix them or something?

121
00:14:46,760 --> 00:14:53,720
And then how do we use that information to decide which problems do we work on?

122
00:14:53,780 --> 00:14:55,260
Who do we work with?

123
00:14:55,260 --> 00:14:57,360
Who do we want to benefit with our work?

124
00:14:57,360 --> 00:15:00,660
And who do we want to just like refuse to work with entirely?

125
00:15:00,660 --> 00:15:03,940
Because there are some really bad actors out there.

126
00:15:04,180 --> 00:15:12,320
And so that's... and then we talk a lot about democratizing data, data literacy, and AI as well.

127
00:15:12,320 --> 00:15:17,660
Because that, I think, is really part of the key of how we're gonna get through this.

128
00:15:17,660 --> 00:15:25,140
So we're trying to balance being not only doom and gloom with like, how do we move forward?

129
00:15:25,140 --> 00:15:32,440
And we propose a set of seven principles for how we move forward with a kind of feminist approach to data science.

130
00:15:33,360 --> 00:15:34,780
All right, thank you.

131
00:15:34,780 --> 00:15:52,180
So I think a common thread in what all of y'all said is that there are problems both... problems in machine learning exist both in the pipeline process, where that is coming from, how it's processed, but also... but that that isn't all of the...

132
00:15:52,180 --> 00:15:53,800
all of the problems.

133
00:15:53,800 --> 00:16:07,220
A lot of the time in the media and in conversations, people have this idea that data is like some kind of objective truth, and that when you come up with biased results or incorrect results, it's because the analyst messed up somewhere.

134
00:16:07,800 --> 00:16:10,760
So, you know, do you think that's true?

135
00:16:10,760 --> 00:16:13,720
Spoiler alert, I'm pretty sure you're all going to say no.

136
00:16:14,960 --> 00:16:21,420
But, you know, what would you say to someone who thinks that way?

137
00:16:21,420 --> 00:16:24,980
And like, what is... what are the problems with thinking about data that way?

138
00:16:27,260 --> 00:16:30,140
Whoever wants to kind of get started.

139
00:16:30,600 --> 00:16:33,960
I can quickly start with a positive note.

140
00:16:34,060 --> 00:16:41,100
Data certainly reflects the state of the society, but it does it in a defective way.

141
00:16:41,100 --> 00:16:50,940
So we are not getting a perfect representation of society, especially with regards to minorities, because they are underrepresented.

142
00:16:50,940 --> 00:17:06,400
But when we have such data sets, at the same time, one advantage is that we can also study the historical injustices these minorities, underrepresented or disadvantaged groups have been going through, because it's reflected in the data set.

143
00:17:06,400 --> 00:17:23,620
As I already mentioned, since these data sets are a reflection of society, and then when we start building models on these systems, then we have data that is reflecting these biases.

144
00:17:23,620 --> 00:17:34,600
But at the same time, it also reflects some facts about the world, such as statistical biases, inductive biases, or some kind of facts that can also be useful.

145
00:17:34,600 --> 00:17:45,020
But that does not justify the fact that there are all these harmful information that gets very accurately embedded in these systems.

146
00:17:45,020 --> 00:17:53,000
So this is my double-edged view on this data topic for now.

147
00:17:53,000 --> 00:17:55,760
I can follow up with that.

148
00:17:55,760 --> 00:17:57,380
I completely agree with Arlene.

149
00:17:57,380 --> 00:18:06,620
I think the problem is how the data on the internet is reflecting the society, but only in very balanced ways.

150
00:18:06,620 --> 00:18:10,740
And this gets amplified when you want large data sets.

151
00:18:10,740 --> 00:18:15,620
The latest one in this aspect is the GPT-3.

152
00:18:15,960 --> 00:18:18,500
It's trained on links from Reddit.

153
00:18:18,640 --> 00:18:20,800
I'm almost never on Reddit.

154
00:18:20,800 --> 00:18:24,580
I've only been a victim of trolling and gun threats on Reddit.

155
00:18:24,580 --> 00:18:25,520
I stay away.

156
00:18:25,520 --> 00:18:30,420
I don't know any woman in machine learning that I'm aware of who is active on Reddit.

157
00:18:30,420 --> 00:18:38,320
So you can see that even among all the imbalance in the internet, the Reddit is even further amplified.

158
00:18:38,320 --> 00:18:59,600
So the decision to go for Reddit links and claim that's what we want to build the largest language model on, we can spend tens of millions of dollars on this data set and nothing else, is to me extremely disappointing and disheartening that the VCs, the decision makers,

159
00:18:59,600 --> 00:19:07,260
they're always going for the worst of the toxic forms of data that's available on the internet.

160
00:19:07,260 --> 00:19:08,920
And I really can't justify this.

161
00:19:08,920 --> 00:19:11,500
I don't know how they sleep every night.

162
00:19:11,500 --> 00:19:16,780
I don't know how they can say with a straight face that this is the path towards AGI.

163
00:19:16,780 --> 00:19:18,660
I mean, this is nonsense, right?

164
00:19:18,660 --> 00:19:21,160
So I get frustrated all the time.

165
00:19:21,160 --> 00:19:24,660
I know Aileen was trying to paint a more positive picture.

166
00:19:24,760 --> 00:19:30,220
I'm just extremely disappointed that I'm sorry for being open about that.

167
00:19:31,160 --> 00:19:34,360
Yeah, well, maybe I'll pop in on the negative note.

168
00:19:36,420 --> 00:19:43,240
I mean, what happens that I think we're pointing to here is that data are very resource intensive, right?

169
00:19:43,240 --> 00:20:00,980
And so it's only certain actors, it's something we say in the book, it's only certain actors in the world that have the resources to collect, store, maintain, utilize, analyze, deploy in some product or whatever, these kinds of data.

170
00:20:00,980 --> 00:20:07,140
And so at root, you only have certain actors in the world that are able to actually do that work.

171
00:20:07,140 --> 00:20:14,620
And because data are so expensive, we see all these kinds of shortcuts, like exactly like this, where it's like, oh, like, where can we get a big data set?

172
00:20:14,620 --> 00:20:21,860
Okay, Reddit, or like, oh, you know, where can we get images to train image, you know, face recognition algorithm?

173
00:20:21,860 --> 00:20:24,200
Oh, okay, like celebrity faces or whatever.

174
00:20:24,200 --> 00:20:32,280
And just like not connecting the dots of again, like, in what ways like who is not represented in those data sets.

175
00:20:32,280 --> 00:20:50,280
And there's like all these multiple points of kind of where inequality kind of enters that pipeline and like not catching it at any of those points, because then the people that are working with the data are often they are the white elite, non-immigrant men,

176
00:20:50,280 --> 00:20:54,660
you know, and they are representative of that demographic that are celebrities, right?

177
00:20:54,660 --> 00:21:00,080
So like, you know, that you have these kind of multiple stages at which those things don't get caught.

178
00:21:00,080 --> 00:21:09,460
But I mean, at the root of it, I often think of the constraint in a lot of ways is, it's financial, like folks are looking for a shortcut.

179
00:21:13,120 --> 00:21:22,100
I mean, I'm so glad, you know, Catherine, you brought the issue of like celebrity data set.

180
00:21:22,100 --> 00:21:41,140
So in the news event, you know, you've heard about Fox, which is kind of like a cloaking tool, you know, for privacy, it uses like very fundamental techniques and adversarial machine learning so that, you know, people can kind of, you know, like save their privacy from like terrible tools like Clearview AI.

181
00:21:41,660 --> 00:21:47,740
And Kashmiri Hill, kind of like the New York Times reporter, tried it out.

182
00:21:47,860 --> 00:21:52,640
It was so weird, you know, she got she got some really weird photos.

183
00:21:52,680 --> 00:21:58,540
And when she asked the PI of that project, like, hey, why are these photos looking really awry?

184
00:21:58,540 --> 00:22:02,820
I thought the whole point is to keep the face same in the cloaked picture.

185
00:22:03,040 --> 00:22:13,360
And he said something to the effect, and I'm perhaps going to misquote him, he said, Oh, yeah, women get mustaches and like, you know, men get eyelashes.

186
00:22:13,360 --> 00:22:15,380
That's how you kind of like cloak.

187
00:22:15,680 --> 00:22:17,140
You know, that's kind of like an outcome.

188
00:22:17,140 --> 00:22:25,500
I'm sure there's a whole like, semester of SDS gender studies to unpack there, because they use like celebrity faces.

189
00:22:25,720 --> 00:22:30,840
And like, how many transgenders are represented in like celebrity data sets?

190
00:22:31,240 --> 00:22:35,760
My first thought was, have you seen the winner of Eurovision 2019?

191
00:22:35,800 --> 00:22:36,940
Like, shame on you.

192
00:22:36,940 --> 00:22:38,900
Have you seen like RuPaul's Drag Race?

193
00:22:38,900 --> 00:22:39,860
Shame on you.

194
00:22:40,000 --> 00:22:42,620
But, but yeah, it is.

195
00:22:42,620 --> 00:22:46,900
And I'm so sorry, Eileen, that, you know, we're not like raining on your parade.

196
00:22:47,120 --> 00:23:01,440
But even though, even though somebody starts with really good intent, and I'm sure like the Fox team did it with very good intent of like, I am going to, you know, forward privacy, I'm going to like, you know, empower people.

197
00:23:01,440 --> 00:23:14,060
But like you said, like when they don't connect the dots, or like how Anima said, when they take like shortcuts, yeah, it doesn't lead to less than like harmful outcomes.

198
00:23:14,600 --> 00:23:21,520
So just as a quick reminder for the people listening, Fox is the paper that we discussed yesterday at Journal Club.

199
00:23:22,200 --> 00:23:24,540
So if you were there for that, awesome.

200
00:23:24,540 --> 00:23:26,520
If not, you can read about the paper.

201
00:23:26,520 --> 00:23:33,100
You can listen to me and a couple others talk about the paper I was talking about on our YouTube channel, where it should be up by now.

202
00:23:33,160 --> 00:23:34,920
Sorry, Eileen, did I cut you off?

203
00:23:35,260 --> 00:23:38,600
Oh, I saved my pessimistic comments to the end.

204
00:23:41,420 --> 00:23:44,460
But now we have our feedback loop completed.

205
00:23:44,460 --> 00:23:59,660
So I'm really curious about all the data practices that are going on out there, or the data impractices or abuse, because as I said, I'm coming from a privacy background, and how people's data is abused, so that they can be de-anonymized, they can be tracked,

206
00:23:59,660 --> 00:24:07,160
and essentially, entities that hold power can use this for surveillance or for large scale manipulation.

207
00:24:07,240 --> 00:24:18,080
And unfortunately, as citizens, or as human beings living in the world, we don't consent to our data being collected, but we don't have an option to opt out.

208
00:24:18,080 --> 00:24:36,780
So all of our data, under the premise of free applications, free products, or just walking on the streets in country X, is being collected by these entities that hold power to build large databases of any kind of information.

209
00:24:36,840 --> 00:24:42,200
And in some cases, this data is not being even used, because they think that we don't have a way to use it right now.

210
00:24:42,200 --> 00:24:50,180
But in the future, we will figure it out, figure out a way to make use of this data to make one type of consequential decision again.

211
00:24:50,180 --> 00:25:02,040
And this is extremely problematic, because the citizen is not informed about these practices and what the long term effects might be on individuals and society.

212
00:25:02,040 --> 00:25:21,800
And on the other hand, when we look at other power holders, such as proprietary software, or all companies that are providing these services for the entire world that are free, supposedly free, are making so much money because they are making use of the data that we are providing them.

213
00:25:21,800 --> 00:25:35,540
And after that, they keep building stronger machine learning models so that they can keep amplifying these biases in their direction with their targeted ads, or whatever their incentive is.

214
00:25:35,540 --> 00:25:40,660
And here there is a serious problem about the incentives of these systems.

215
00:25:40,660 --> 00:25:54,080
And both Catherine and Anima mentioned that, for example, we have these objective functions optimizing for accuracy in these systems that actually are helping these power entities.

216
00:25:54,080 --> 00:26:03,040
But the objective function could also be how can we use these data sets for social good, as opposed to optimizing for accuracy.

217
00:26:03,040 --> 00:26:15,040
So there are so many different faceted and different levels of abstraction of problems involved in this data ecosystem again.

218
00:26:15,040 --> 00:26:24,080
And I guess we can keep discussing these in more details, but today the goal is kind of give a glimpse of what is going on in this area.

219
00:26:24,480 --> 00:26:27,320
So Aileen, you picked out... guess wrong?

220
00:26:27,380 --> 00:26:31,240
Oh sorry, I just have like a quick follow-up for Aileen and Anima.

221
00:26:31,940 --> 00:26:37,020
I know that both of you have kind of revealed the shortcomings of ML systems.

222
00:26:37,020 --> 00:26:40,920
Anima, you've done extremely awesome work about how GPT-3 is flawed.

223
00:26:40,920 --> 00:26:44,560
And Aileen, your groundbreaking work on word embeddings.

224
00:26:45,540 --> 00:26:52,500
And I know that both of you have communicated that research to the community in a very approachable fashion.

225
00:26:52,980 --> 00:27:02,960
But I don't know about you, Aileen, but I visibly see the kind of like negative, rancorous feedback that you get.

226
00:27:03,240 --> 00:27:12,240
And from very well-intentioned research, I mean I don't know well-intentioned, from like people who are in the ML community, why do you think that happens?

227
00:27:12,240 --> 00:27:23,960
And Catherine, like from a feminist perspective, why are people from the own educated community like unloading on these researchers?

228
00:27:23,960 --> 00:27:26,660
I would love to get your perspectives on this.

229
00:27:26,680 --> 00:27:29,360
Sorry, Stella, I didn't mean to like... No, that's fine.

230
00:27:29,360 --> 00:27:31,620
That's very close to where I was going to go anyway.

231
00:27:31,820 --> 00:27:35,560
I mean, just maybe I can get started on this, right?

232
00:27:35,560 --> 00:27:38,500
I mean, to me, I think it's at the root of human psychology.

233
00:27:38,500 --> 00:27:40,620
Like no one wants to be the bad guy.

234
00:27:40,720 --> 00:27:43,560
No one wants to think of themselves as the bad guys.

235
00:27:43,560 --> 00:27:49,020
So they believe, right, they are going towards AGI through this GPT-3 model, for instance, right?

236
00:27:49,020 --> 00:27:50,980
I mean, I think they truly believe.

237
00:27:50,980 --> 00:27:55,520
So let's assume that.

238
00:27:55,520 --> 00:28:01,400
And I think the thing is they really don't want to also hear anything that may harm that, right?

239
00:28:01,400 --> 00:28:06,360
So they believe anything that may harm that path is that are their enemies.

240
00:28:06,360 --> 00:28:09,640
People who are voicing opinions against it are their enemies.

241
00:28:09,640 --> 00:28:19,700
And so that's why we see this vicious pushback on social media platforms like Twitter and, you know, and then they have huge follower counts.

242
00:28:19,700 --> 00:28:21,020
So that builds up, right?

243
00:28:21,020 --> 00:28:30,780
And all they have to do is do it, just kind of give their followers a sense to go after people who are bringing up these difficult issues.

244
00:28:30,860 --> 00:28:35,280
And honestly, because there's also a lot of insecurity, they don't know the answers to this.

245
00:28:35,280 --> 00:28:37,620
They don't know a way out.

246
00:28:37,620 --> 00:28:46,160
And they worry that their entire existence is in question, which I think that's why, you know, to me, I feel like we are on the right track.

247
00:28:46,160 --> 00:28:51,220
So we have to keep pushing this on and increasing awareness, have more allies.

248
00:28:51,360 --> 00:28:54,300
I think we will get more support.

249
00:28:54,300 --> 00:28:57,960
And it's already been amazing over the last few years, right?

250
00:28:57,960 --> 00:29:03,020
I mean, it's happened so quickly, even just the mention of AI bias was not there.

251
00:29:03,020 --> 00:29:06,240
It was completely dismissed in machine learning conferences.

252
00:29:06,340 --> 00:29:11,560
People working on it were considered not technical, you know, were not respected.

253
00:29:11,860 --> 00:29:17,580
There was a lot of gaslighting, you know, because it happens to be women, it happens to be minorities.

254
00:29:17,620 --> 00:29:25,140
So there was even more of all that racism and sexism coming into play when such research was reported.

255
00:29:25,440 --> 00:29:34,200
But now, you know, especially with the Black Lives Matter movement, that's when these companies finally said, we won't be selling face recognition to law enforcement, right?

256
00:29:34,200 --> 00:29:39,860
We've been asking for this, you know, we've signed letters, we've gone to the press, we've been doing this for a while.

257
00:29:39,860 --> 00:29:43,700
And it's been led by black women, which is amazing.

258
00:29:43,700 --> 00:29:48,860
But, you know, it took a broader global movement to make this happen.

259
00:29:48,860 --> 00:29:54,960
And I think we are building this momentum up, because it's not just about AI, it's not just about data ethics.

260
00:29:54,960 --> 00:29:58,000
It's a much broader question for humanity.

261
00:29:58,100 --> 00:30:01,260
And I think we're, you know, I have to, I remain hopeful.

262
00:30:01,960 --> 00:30:23,260
So since Anima brought this up, I wanted to take a moment and ask you guys, do any of you, as yourselves personally, or the companies or institutions that you represent, have any policies about what types of organizations or what types of tasks you will not do ML research or data research for?

263
00:30:23,260 --> 00:30:28,660
You know, be it police applications, military applications are two of the ones I know are really popular.

264
00:30:29,520 --> 00:30:41,980
I mean, speaking personally, disclaimer, sorry, before I say this, my company has a large number of lawyers who'd like me to tell you that my reviews on the stream represent myself and in no way represent my company at all.

265
00:30:42,260 --> 00:30:45,120
I work for a US government contractor.

266
00:30:45,300 --> 00:30:52,360
And a lot of the people who work at my company do ML research for the Department of Defense and the US military.

267
00:30:52,780 --> 00:30:59,320
And that's a very contentious topic, both within my company and I know within the ML community at large.

268
00:30:59,520 --> 00:31:06,380
So, you know, I have been personally trying to decide how to navigate that line myself.

269
00:31:06,420 --> 00:31:10,620
But I was wondering if any of you guys had particular stances you've developed?

270
00:31:13,880 --> 00:31:21,860
I mean, I'll just say, for my own work, and I'm not speaking about MIT, because obviously MIT is all wound up with all sorts of different things.

271
00:31:21,860 --> 00:31:29,360
But at least as a professor, you have autonomy to do what you want.

272
00:31:29,360 --> 00:31:38,260
And so at least from my perspective, I've decided to only work with folks who are doing some kind of counter hegemonic work.

273
00:31:38,260 --> 00:31:45,660
So activists, journalists, social movements, community based organizations.

274
00:31:45,660 --> 00:31:57,080
That's also kind of the, like, that's also in my research interest is to think about, like, how do we think about building data literacy in those kinds of sectors as well.

275
00:31:57,080 --> 00:31:58,600
So it's like a really nice fit.

276
00:31:58,600 --> 00:32:00,680
And that happens to be where my interests are.

277
00:32:00,720 --> 00:32:11,480
But I'm sympathetic to the stance that like, yeah, when you work in a large organization, as many of these places are, you have a lot less autonomy and thinking about, well, like, how do you navigate that as an individual?

278
00:32:11,480 --> 00:32:13,420
How do you navigate that as a team?

279
00:32:13,420 --> 00:32:17,100
And how do you navigate that at the level of the institution?

280
00:32:17,320 --> 00:32:25,940
But I will say, like, I am really energized by all of the organizing and activism that's happened within large companies right now as well.

281
00:32:25,940 --> 00:32:32,960
And so I think that is a huge, potentially, is a huge force for change as well.

282
00:32:33,800 --> 00:32:35,640
Yeah, I agree.

283
00:32:35,640 --> 00:32:39,400
And I want to add what Stella said earlier, right?

284
00:32:39,400 --> 00:32:44,160
I am not representing NVIDIA or Celtic for that matter.

285
00:32:44,160 --> 00:32:49,480
But I see a lot of support building up from the grassroots in these organizations.

286
00:32:49,500 --> 00:32:57,180
And NVIDIA being the platform for AI computing, you know, we are not directly selling these kind of AI applications, right?

287
00:32:57,180 --> 00:32:59,760
So it's really our customers and partners.

288
00:32:59,900 --> 00:33:06,080
And so there is less of a control in sense of everything the others do, but there's always periodic reviews.

289
00:33:06,080 --> 00:33:16,820
And, you know, there have been discussions internally in terms of how to make sure that our platforms are being used for good causes rather than bad ones.

290
00:33:17,000 --> 00:33:20,060
But I agree, there is a lot more work to be done.

291
00:33:20,060 --> 00:33:35,340
But I'm very energized by activism internally at NVIDIA, and also at Celtic, especially the students, everything, you know, from the recent Black Lives Matter, how it was really the students who have been energizing this movement.

292
00:33:35,340 --> 00:33:37,440
And so that gives me a lot of hope.

293
00:33:38,240 --> 00:33:39,480
Yeah, and I'm happy.

294
00:33:39,480 --> 00:33:39,920
Oh, sorry.

295
00:33:39,920 --> 00:33:40,940
Go ahead, Aileen.

296
00:33:41,220 --> 00:33:42,020
Please.

297
00:33:42,020 --> 00:33:50,380
I was just going to say that I'm not representing anyone except my humble opinion that might not be very well formed.

298
00:33:50,380 --> 00:34:01,100
But I try to build my own ethical framework when approaching these issues, since we don't have the AI ethics framework that is widely accepted by anyone.

299
00:34:01,100 --> 00:34:05,020
I basically try not to work on any confidential project.

300
00:34:05,020 --> 00:34:09,280
I don't try, I haven't worked on any confidential or secret projects.

301
00:34:09,280 --> 00:34:30,120
And other than that, I try to get involved in pro bono activities to help companies, entities, research their systems to understand what kind of negative side effects they might have, especially with regards to bias, can we come up with new forms of ways to measure bias,

302
00:34:30,120 --> 00:34:31,860
develop methods.

303
00:34:31,860 --> 00:34:35,840
And that has been my approach to this issue right now.

304
00:34:35,840 --> 00:34:46,140
But if we had a well accepted code of ethics among researchers, I think everyone would benefit from such a framework.

305
00:34:49,720 --> 00:34:51,480
I mean, I totally agree.

306
00:34:51,480 --> 00:35:02,020
I think one of the things that's kind of like helped me personally, again, you know, boilerplate, boilerplate, I'm only representing myself.

307
00:35:02,020 --> 00:35:06,620
When Anima said like reviews that kind of like really resonated with me.

308
00:35:07,140 --> 00:35:16,560
There is like, within Microsoft, there's this like committee called the AI Ethics and Research Organization.

309
00:35:16,620 --> 00:35:18,000
It's called ETHER.

310
00:35:18,000 --> 00:35:20,520
I'm perhaps butchering the acronym.

311
00:35:20,720 --> 00:35:30,400
But one of the things that they do, and it is like, you know, folks from MSR, you know, and it's just not the ML folks, but MSR is such a vibrant community.

312
00:35:30,400 --> 00:35:49,480
So, you know, you've got like sociologists, you know, folks from like, who think about bias from MSR New York, in Eric Horowitz's organization, kind of like, you know, making sure for critical like ML applications, you know, you need to get ETHER's sign off.

313
00:35:49,540 --> 00:35:52,840
So that has been a, you know, a guiding light.

314
00:35:52,900 --> 00:36:05,440
The second thing for me is like, you know, it's, I completely agree with Eileen, if there's some like, like a framework, and I, you know, that way you can meet it out to engineering engineers.

315
00:36:05,780 --> 00:36:18,180
The thing that's kind of worked for me is that the standards of business conduct at Microsoft is surprisingly good, which is a very, which is a very good thing to say, because they make it, they shoot it like Netflix style videos.

316
00:36:18,240 --> 00:36:21,480
So it's not the boring training that you have to kind of go through.

317
00:36:21,480 --> 00:36:28,960
I mean, I mean, I've worked in Microsoft, like, in 2013, it was just like the most like boring thing, but you still have to do it.

318
00:36:28,960 --> 00:36:30,360
It's like a read through it.

319
00:36:30,360 --> 00:36:36,220
But now it's like an actual like Netflix series, they've got the same actors coming in different years.

320
00:36:36,220 --> 00:36:59,640
But I mean, long story short, there was a module about how this guy tries to train his model on this, like private customer data, and how he gets into such so much of, you know, like, like a big, like big soup, he can't get out of it, you know, he's trying to demo it and he has to like talk to his manager and managers like,

321
00:36:59,640 --> 00:37:02,300
oh my gosh, you're gonna like, you know, lose your job.

322
00:37:02,300 --> 00:37:07,120
So they really rain it down on you in a more relatable fashion.

323
00:37:07,120 --> 00:37:30,420
So all of this to say that, you know, having this, you know, this committee to kind of like guide for critical like applications, and kind of having this like relatable, you know, ethics discussion has kind of like helped at least me and my team personally a lot,

324
00:37:30,420 --> 00:37:34,140
quickly ask Ram, so can they release this to the public?

325
00:37:34,140 --> 00:37:38,100
I think immensely from this as well.

326
00:37:38,100 --> 00:37:44,600
Yeah, I really hope that they do because it's, it's really captivating, like you're not bored.

327
00:37:45,280 --> 00:37:47,220
You know, when you're watching that.

328
00:37:48,020 --> 00:38:02,260
So I have a question from the discord that is something you guys have all touched on a little bit, but I think it's worth like directly addressing, which is that so we've been talking about how we were, we all represent large companies, or major universities,

329
00:38:02,260 --> 00:38:06,140
you know, institutions that have power and money and resources.

330
00:38:06,320 --> 00:38:10,940
And, you know, how, what is it?

331
00:38:11,120 --> 00:38:23,240
So what the question asked on discord was, was, you know, if, what's the role that these large institutions is powerful and franchise institutions play?

332
00:38:23,240 --> 00:38:32,320
And, you know, if the ability to collect large data sets or train these very large expensive models were somehow democratized, would that improve the situation?

333
00:38:32,980 --> 00:38:37,320
Ram, I think we should throw this over to you, because I know you've recently written a paper on this topic.

334
00:38:39,840 --> 00:38:44,260
Yeah, that's a really hard question for me.

335
00:38:44,500 --> 00:38:58,400
You know, just because, you know, again, from, from a, from a, from a winner sense, like, is machine learning inherently authoritarian, you know, because like, or is it inherently democratic?

336
00:38:58,580 --> 00:39:15,240
And, you know, there, there are arguments to be made on both sides, like, you know, like Anima started off with data compute, and the algorithms, that knowledge is kind of centralized in a lot of, you know, in a lot of aspects, right?

337
00:39:15,240 --> 00:39:30,960
You, you know, like, $10 million or so with their TPU training pods, which, you know, like a mom and pop shop, like an individual researcher is not going to have.

338
00:39:31,200 --> 00:39:42,000
And data, as we all know, like Catherine kind of like outlines in her book beautifully, like how it is really, it's situated with power, almost.

339
00:39:42,520 --> 00:39:50,540
So, you know, in a, in a, in a, in a land and winner sense, it really does, you know, one may almost say that, you know, machine learning is authoritarian.

340
00:39:50,540 --> 00:40:06,300
On the other hand, actually, one of the other papers that we read was Anima's Born Again Neural Nets paper, where she shows that, you know, a student can actually in certain like conditions outperform the teacher.

341
00:40:06,860 --> 00:40:20,920
And, and if you think about the revolution that transfer learning is bringing along, you know, you, a company can put out this like pre-trained model, and then you can just retrain the last layer for your like custom, like applications.

342
00:40:21,240 --> 00:40:33,520
So, and, and now with like, you know, systems like fast.ai and like, you know, they're bringing like machine learning, like two programmers, which again is not making it democratic.

343
00:40:33,520 --> 00:40:43,460
So it's almost like, you know, you kind of trying to install solar panels in your, in your house, as opposed to depending on like a nuclear power station.

344
00:40:43,480 --> 00:40:46,840
So I completely struggle with this question.

345
00:40:46,840 --> 00:40:49,120
I keep vacillating back and forth.

346
00:40:49,120 --> 00:40:56,140
And unfortunately I do not have like, I'm not, I'm not, I'm not like settled on an answer.

347
00:40:56,140 --> 00:41:00,800
And perhaps like other panelists will guide us, will guide us through this cave.

348
00:41:02,380 --> 00:41:05,140
Thank you Ram for bringing up the paper.

349
00:41:05,140 --> 00:41:09,960
I think to me, I, you know, I again pin my hope back on the algorithms, right?

350
00:41:09,960 --> 00:41:15,340
So I think we are just at the infancy in terms of algorithmic development for deep learning.

351
00:41:15,340 --> 00:41:18,220
We've been using the very basic algorithms.

352
00:41:18,220 --> 00:41:21,320
The question is, you know, where to go next?

353
00:41:21,320 --> 00:41:25,380
You know, I mentioned unsupervised learning earlier as a broad topic.

354
00:41:25,380 --> 00:41:29,500
I think within that, what we want is more controllability, right?

355
00:41:29,500 --> 00:41:31,620
Like for instance, generating text.

356
00:41:31,620 --> 00:41:43,100
If you could control what needs to be generated, then you can try, you know, head towards removing these biases, at least in what's generated, even if it's inherent in the model.

357
00:41:43,240 --> 00:41:46,100
And in fact, that's what we need in practical applications.

358
00:41:46,100 --> 00:41:48,660
You know, we don't want it to write weird poetry.

359
00:41:48,800 --> 00:41:56,280
We want it to maybe, right, give us knowledge facts or ultimately program, you know, do program synthesis and so on.

360
00:41:56,280 --> 00:41:58,420
And so that's something we've been working on.

361
00:41:58,420 --> 00:42:00,740
Same for text, same for images, right?

362
00:42:00,740 --> 00:42:07,880
Ultimately the GAN should be able to have controllable generation and it can control various attributes, right?

363
00:42:07,880 --> 00:42:13,760
And that way then it's really at the hands of the human on how to use these models.

364
00:42:13,760 --> 00:42:17,500
And currently we don't have good frameworks for that.

365
00:42:17,500 --> 00:42:28,860
And I think unsupervised learning is the key to enable that because you have to learn what are the hidden variations in data and then how to control and change that and be able to generate new examples.

366
00:42:29,360 --> 00:42:32,180
Yeah, so I think that's the direction to go forward.

367
00:42:33,100 --> 00:42:36,020
I completely agree with Anima, by the way.

368
00:42:36,020 --> 00:42:38,400
She has been mentioning unsupervised learning.

369
00:42:38,400 --> 00:42:58,840
I also think that the future is probably in unsupervised, self-supervised learning that is a bit maybe more intelligent, depends on how you define intelligent, because again, this is semantics and a vague term, but it seems like once we get there so that the system is able to figure out this universal learning pattern somehow,

370
00:42:58,840 --> 00:43:08,400
and then starts figuring out different kinds of tests by itself, we might be able to somehow incorporate some kind of awareness to the system.

371
00:43:08,400 --> 00:43:11,520
And one option would be a human in the loop, for example.

372
00:43:11,520 --> 00:43:19,420
And again, Anima mentioned this a few minutes ago, saying that human psychology and awareness plays a very important role here.

373
00:43:19,420 --> 00:43:27,320
Like, we are not able to agree on many of these questions, because we are dealing with ethical questions.

374
00:43:27,320 --> 00:43:33,800
Everyone has their own understanding, their own answer, and their own desired fairness notion.

375
00:43:34,140 --> 00:43:38,100
And we can't satisfy all of these at the same time.

376
00:43:38,100 --> 00:43:47,560
However, if we are able to bring some kind of awareness in these systems, then we might be able to change actions and decisions.

377
00:43:47,560 --> 00:43:59,980
Because, the evolution of how people are reacting to biases depends on how aware we are of our biases.

378
00:43:59,980 --> 00:44:03,320
It's not like people are trying to discriminate other groups.

379
00:44:03,320 --> 00:44:06,680
It's mostly caused by favoring the in-group.

380
00:44:06,680 --> 00:44:12,000
Like, there are no harms that are explicitly trying to be imposed upon other groups.

381
00:44:12,000 --> 00:44:16,020
But this unintentional bias ends up causing all of these harms.

382
00:44:16,020 --> 00:44:29,060
So, in an unsupervised way, we can figure out some kind of an efficient learning pattern, where we can also incorporate ethical mechanisms, then we will start getting there.

383
00:44:29,060 --> 00:44:43,400
But of course, all of these will require a lot of informed discussions from many different perspectives, and many different disciplines as well, because this is affecting all disciplines, all the sciences, and every social fabric.

384
00:44:45,700 --> 00:44:54,140
And yeah, maybe I'll just, I mean, I think the question of, you know, is machine learning authoritarian or not is a super interesting one.

385
00:44:54,140 --> 00:45:00,580
But ultimately, the way that we would answer it from a feminist perspective is, in whose hands is it?

386
00:45:00,580 --> 00:45:05,960
So, you know, so feminism, basically, what feminism brings is who questions.

387
00:45:05,960 --> 00:45:12,660
So, like, who is using it, for whom, to support whom, to oppress whom, and so on.

388
00:45:12,660 --> 00:45:20,520
And so, I mean, I think that, for me, is the, those are the questions to be asking when we're thinking about, like, democratizing AI.

389
00:45:20,520 --> 00:45:29,440
Like, I think it's lovely to think about making it easier to use models, democratizing access to data and computational power.

390
00:45:29,440 --> 00:45:33,880
Those are all great things as, like, a kind of infrastructure of democratization.

391
00:45:33,880 --> 00:45:45,620
But on their own, they won't create a democratic kind of usage of AI, because we also need to do the bridge to the groups that could really actually benefit from AI.

392
00:45:45,620 --> 00:45:54,840
And for me, like the, again, like I'm saying, like, I'm interested in sort of counter-hegemonic groups and thinking about, like, what are groups that actually hold power accountable?

393
00:45:54,840 --> 00:46:00,820
They're groups like lawyers and legal scholars, journalists, community organizations, social movements.

394
00:46:00,820 --> 00:46:11,660
Like, these are the folks who need to be kind of trained and brought to the table, if we're actually talking about a comprehensive effort to democratize AI.

395
00:46:11,660 --> 00:46:16,740
Because those are the folks that you want to understand, like, what are actually the creative, creative potentials.

396
00:46:16,740 --> 00:46:31,000
And there are many, like, some of the journalists that have started to discover the power of machine learning have been doing amazing things, like detecting, using image recognition to detect amber mining from satellite imagery in the Ukraine, for example.

397
00:46:31,000 --> 00:46:40,620
Like, you can do this, like, kind of amazing accountability work, but not every person is going to have that idea, because they haven't been exposed to working in this way.

398
00:46:40,620 --> 00:46:42,560
So, yeah.

399
00:46:43,140 --> 00:46:54,440
So, if we're talking about, you know, AI algorithms being developed by institutions of power, and the fact that, like, who is using algorithms matters a lot , is there a role in, like, adversarial AI?

400
00:46:54,440 --> 00:47:01,980
Adversarial AI is usually cast as a bad thing, because machine learning engineers do good, and those pesky evildoers are trying to ruin it.

401
00:47:01,980 --> 00:47:08,280
But if machine learning algorithms are being used for bad, you know, does that mean that adversarial AI techniques can be used for good?

402
00:47:08,700 --> 00:47:09,480
Oh, I love that.

403
00:47:09,480 --> 00:47:10,300
Totally.

404
00:47:11,120 --> 00:47:28,880
I totally think so, yes, because, like, there are so many reasons why groups or individuals would need to be protecting themselves, like, from, you know, we were just talking about Reddit, for example, you know, like, there's so many reasons why, you know,

405
00:47:28,880 --> 00:47:35,200
like, an act of, kind of, an adversarial act is not necessarily bad.

406
00:47:35,200 --> 00:47:57,500
Like, for example, I have a friend who, as a way of, kind of, fooling the just rampant data capture that's done by promotional ads and things in the browser, and so, like, the browser is always sending the IP address and whatnot, and what it does is, instead of,

407
00:47:57,500 --> 00:48:03,580
like, sending what it clicks on, the browser clicks on, it actually just sends every link on the page.

408
00:48:03,980 --> 00:48:14,280
So it's, like, hacking it through obfuscation, you know, and so it's, like, I think that's a brilliant technique for then preserving privacy.

409
00:48:14,280 --> 00:48:20,040
So you're obfuscating what the user is actually doing, and screwing with the ad systems or whatever.

410
00:48:20,040 --> 00:48:31,460
And so I think, especially in situations where power is not equal, which is every situation right now, like, we also need, like, good, like, white hat adversarial ML, basically.

411
00:48:32,100 --> 00:48:34,320
I love this, Catherine.

412
00:48:34,320 --> 00:48:38,420
I got so excited for the ad blocker research.

413
00:48:39,400 --> 00:48:51,900
I mean, right now, in the field of adversarial machine learning, the use of adversarial machine learning to enhance the experience of citizenship is still nascent, but, you know, there's fantastic work that's happening.

414
00:48:51,900 --> 00:48:53,260
I'm so excited for this.

415
00:48:53,280 --> 00:49:02,060
Bogdan Kulinich, I'm sorry if I'm pronouncing his last name incorrectly, developed this paper called POTS, which is Protective Optimization Techniques.

416
00:49:02,060 --> 00:49:04,320
We're into, kind of, like, empower.

417
00:49:04,380 --> 00:49:21,560
So what they do is, it's really cool, it turns out that, obviously, credit card scoring is biased against minority population, and they show, like, how you can poison the system tactically, like, do it, literally launch, like, a poisoning attack to, kind of,

418
00:49:21,560 --> 00:49:26,880
like, you know, make it better for the target population.

419
00:49:27,000 --> 00:49:42,840
There's really another cool work from some researchers who found that Facebook's ads platform is micro-targeting, and they, again, use adversarial ML to show, like, you know, faults in that.

420
00:49:42,840 --> 00:49:50,640
So that opens up, like, exactly, like, how Catherine says, white hat adversarial examples.

421
00:49:50,660 --> 00:49:57,060
Unfortunately, you know, it's not the dominant thinking.

422
00:49:57,060 --> 00:50:08,860
In fact, DARPA, when it released its guard challenge, which is, you know, securing against, like, AI challenge, they said, we need to protect ML system from attackers.

423
00:50:08,860 --> 00:50:11,920
And they do not define who attackers are.

424
00:50:12,020 --> 00:50:17,420
So that's... And I completely agree with Ram and Catherine.

425
00:50:17,420 --> 00:50:28,600
And I think there's also, right, because we don't know who are the attackers, I feel like there's also people are going in all kinds of weird directions with adversarial AI research, right?

426
00:50:28,600 --> 00:50:34,120
Because they don't know what kind of attack to, you know, and they're like, okay, maybe I'll do this norm bound or some other.

427
00:50:34,380 --> 00:50:36,620
And then it's like, okay, why does this matter?

428
00:50:36,620 --> 00:50:37,740
We don't know, right?

429
00:50:37,740 --> 00:50:42,040
So I think perceptual attacks for images are much more meaningful.

430
00:50:42,040 --> 00:50:51,120
I mean, you know, if those that kind of preserve the for humans, the perception, but, you know, change the statistics of data, right.

431
00:50:51,120 --> 00:50:59,800
And to me, I think the next step is to really have the regulatory agencies, you know, take on these tests.

432
00:51:00,060 --> 00:51:01,360
I mean, we have a long way to go.

433
00:51:01,360 --> 00:51:09,020
Because if you look at the NIST, which is the standards data sets for face recognition, that itself is heavily biased against minorities.

434
00:51:09,280 --> 00:51:18,240
But ultimately, the hope is regulatory agencies in each domain will be using these as tests, you know, especially for autonomous driving, this is a must, right?

435
00:51:18,240 --> 00:51:31,760
I mean, given the long tail, the only way to do is also have very heavy testing would need a lot of compute, but only then certify that this is roadworthy and otherwise be very careful about that.

436
00:51:32,580 --> 00:51:37,840
And Anima, to my understanding, NIST has been tasked to develop these AI standards.

437
00:51:37,840 --> 00:51:45,180
So they are especially working on those given the example with the face recognition or computer vision data sets they had as well.

438
00:51:45,180 --> 00:51:48,540
So that was a great point that you identified.

439
00:51:48,540 --> 00:51:59,880
And again, there are all of these privacy enhancing or fairness enhancing adversarial applications of machine learning, but they might not be very common right now.

440
00:51:59,880 --> 00:52:10,920
But the most simplistic version we can think of is synthetically generating data to oversample minority populations so that we can improve their representation.

441
00:52:10,920 --> 00:52:19,200
Because at least in real world statistics, it's well known that oversampling minority groups is a helpful approach.

442
00:52:19,200 --> 00:52:26,600
But at the same time, again, this calls for the need of some standardization in this process.

443
00:52:26,600 --> 00:52:32,760
And again, this ties back into the question of is only data bias or algorithms also biased?

444
00:52:32,760 --> 00:52:34,840
Why are we creating these algorithms?

445
00:52:34,840 --> 00:52:37,260
What are the goals of these algorithms?

446
00:52:37,420 --> 00:52:44,300
Who are the bias power holders implementing these algorithms?

447
00:52:44,300 --> 00:52:46,580
And again, so it really matters.

448
00:52:46,580 --> 00:52:49,500
And this might bring us to the Kant question here.

449
00:52:49,500 --> 00:52:52,200
Should we have this ontological approach?

450
00:52:52,200 --> 00:52:55,040
Or should we go with utilitarianism?

451
00:52:55,040 --> 00:52:55,760
And so on.

452
00:52:55,760 --> 00:53:05,120
So there are many interesting questions in this direction as well that we are starting to just scratch the surface nowadays, because it's becoming a very hot topic.

453
00:53:05,120 --> 00:53:07,660
So why are we even building these systems?

454
00:53:07,660 --> 00:53:10,120
Why are they going to empower at the end?

455
00:53:11,100 --> 00:53:16,540
Absolutely, I just want to quickly add to Alan's point on the oversampling for minorities.

456
00:53:16,600 --> 00:53:20,760
I think, yeah, if you don't have more data, I think that's a quick fix right?

457
00:53:20,760 --> 00:53:22,160
That's the starting point.

458
00:53:22,160 --> 00:53:26,180
But that itself can also create problems because GANs have mode collapse.

459
00:53:26,180 --> 00:53:30,840
So you just have very less diversity among, say, the minority class, right?

460
00:53:30,840 --> 00:53:42,840
For instance, in fact, it was shown when the images that are generated based on the celeb data set, the celebrity data set, black women tended to have all the makeup on, right?

461
00:53:42,840 --> 00:53:49,260
So it was and with these face recognition systems, when black women didn't have makeup, it was even worse.

462
00:53:49,260 --> 00:53:56,980
You know, so then there is kind of further bias even within the class because of the lack of diversity and variation.

463
00:53:57,300 --> 00:54:07,140
And that's why I think we need to disentangle all these factors and do unsupervised learning and learn, is there also enough diversity within the class in terms of these variations?

464
00:54:08,020 --> 00:54:11,380
Yeah, go ahead Stella, sorry.

465
00:54:11,380 --> 00:54:15,260
So that sort of feeds into something that someone in the Slack asked about as well.

466
00:54:15,260 --> 00:54:27,440
And it's been in the academic discussion of AI bias a lot recently, which is how models can exacerbate or create biases out of imperfections.

467
00:54:27,440 --> 00:54:36,340
Even things that aren't normally thought of as imperfections, just the choice of your loss function implicitly decides how you're going to weigh different samples.

468
00:54:36,340 --> 00:54:52,740
And this is crude, but if 90% of the people in a data set are white and 10% of the people in the data set are black, then an algorithm that gets a little gain in performance at the cost of a little loss on a white person, at the cost of a little loss on a black person,

469
00:54:52,740 --> 00:54:59,080
is going to perform better on average, if it just mostly pays attention to white people.

470
00:55:00,520 --> 00:55:05,900
So, you know, lost the rest of my train of thought.

471
00:55:07,220 --> 00:55:18,200
How, so it's not, it's not just that when we talk about bias in algorithms, we're not just talking about the data coming in, but kind of algorithms can create their own sorts of little biases.

472
00:55:18,800 --> 00:55:21,120
Like, what does it look like to combat that?

473
00:55:21,120 --> 00:55:23,280
And kind of how does that happen?

474
00:55:23,660 --> 00:55:28,480
You know, from the perspective of a data scientist who's looking at something, it's like, you know, how do you know that's happening?

475
00:55:28,480 --> 00:55:30,080
And how do you combat that?

476
00:55:32,020 --> 00:55:35,260
Just a quick anecdote, because I don't have an answer to your question.

477
00:55:35,940 --> 00:55:48,820
A quick anecdote was one of the resume screening algorithms, it wasn't Amazon, but it was like a different company, when they reverse engineered it to see, like, why was it preferring men over women?

478
00:55:48,820 --> 00:55:53,580
One of the factors that they found, one of the strongest predictors was like, the name.

479
00:55:53,580 --> 00:56:02,400
So if somebody was named Jared, in particular, it was a very strong predictor of success, as well as if they were a lacrosse player or not on their resume.

480
00:56:03,020 --> 00:56:12,220
And so it's like, just a sort of very interesting thing, where, in a way, what happens there, like, they're not even they're considering, well, I don't know, I don't really know the model.

481
00:56:12,220 --> 00:56:16,740
But like, I don't think they were directly considering race or gender in that case.

482
00:56:16,740 --> 00:56:24,180
But if you look at those things, like Jared is a very white, like kind of Anglo name.

483
00:56:25,160 --> 00:56:32,460
Lacrosse is a very elite, white, sort of private school sport in the United States, despite the fact that it was invented by Native Americans, actually.

484
00:56:32,720 --> 00:56:39,920
And so like, those things sort of, they seep in, and then they become these factors, and they become predictive factors.

485
00:56:39,920 --> 00:56:46,980
But like, behind those is always like, is race and gender and all of those things are like, kind of still underlying.

486
00:56:48,640 --> 00:56:58,500
Yeah, these systems learn the associations, and you were both asking what kind of metrics we should, for example, develop to understand how to improve these systems.

487
00:56:58,500 --> 00:57:12,040
So I was wondering, maybe this exists in literature, but do we have, for example, diversity dispersion measurement metrics to understand how diverse a group is, so that we can try to improve diversity within one social group?

488
00:57:12,040 --> 00:57:26,940
And then the other thing is, Stella, you mentioned objective functions, or machine learning algorithms that are trying to optimize based on the data representation, where people are categorized into groups, racial groups, or gender.

489
00:57:26,940 --> 00:57:29,860
And it's not even clear how ethical this approach is.

490
00:57:29,860 --> 00:57:32,260
Should people be represented in categories?

491
00:57:32,260 --> 00:57:38,640
Is this the correct proxy to a person's identity or being?

492
00:57:38,640 --> 00:57:44,180
And how should we deal with system design when we are developing new machine learning algorithms?

493
00:57:44,180 --> 00:57:52,140
Because each algorithm depends on these data types, the way they are represented, numeric values, ordinal, or whatever.

494
00:57:52,140 --> 00:58:03,340
And should we come up with more creative ways to design new algorithms for machine learning to encounter, to counter balance, these problems?

495
00:58:04,800 --> 00:58:20,580
And Stella, I mean, I want to rewrite this question back to Catherine, because when I was reading a chapter from her book, Catherine, you spoke about concepts which maintain power, like fairness, bias, and ethics.

496
00:58:20,580 --> 00:58:24,220
And you give this amazing framework about concepts of challenge power.

497
00:58:24,220 --> 00:58:26,100
I didn't even think about that.

498
00:58:27,200 --> 00:58:29,880
Is that something you want to talk about?

499
00:58:29,880 --> 00:58:36,400
I feel it's a really cool framework for engineers to think beyond the fact star or fact framework that we're used to.

500
00:58:36,940 --> 00:58:37,620
Sure.

501
00:58:37,620 --> 00:58:37,980
Yeah.

502
00:58:37,980 --> 00:58:41,980
I think that's in chapter one or two.

503
00:58:43,200 --> 00:58:44,600
I can't remember.

504
00:58:45,600 --> 00:59:00,940
But yeah, I mean, so one of the things I think that Lauren and I were frustrated by is that we just felt like fairness, and this is not, I guess I should say a little bit of a disclaimer , because I actually think the work that's going on in the fact star community is super exciting.

505
00:59:01,160 --> 00:59:06,620
And so like, this is not to say that, like, no one should do that work, because I am really happy that work is being done.

506
00:59:07,280 --> 00:59:14,300
But that's just some of the organizing framing concepts for what we're talking about aren't, they don't go far enough.

507
00:59:14,300 --> 00:59:30,540
And so it's like, instead of fairness, we propose justice, or no, instead of ethics, we propose justice, like, we're really thinking about what does it mean to make a just system, which means repairing past harms, like harm has been done in the world,

508
00:59:30,540 --> 00:59:49,140
we come to the world in the bodies, in which harm has been inflicted at the group level, for some of us for centuries, you know, and so like, how do we build systems that not only kind of treat us all equally as if we're all the same, but actually take that harm into account.

509
00:59:49,180 --> 00:59:54,200
And so we propose other kinds of like, justice, concepts like...

510
00:59:55,380 --> 00:59:57,420
My favorite one is transgression.

511
00:59:58,060 --> 00:59:59,340
That's what I was gonna say.

512
00:59:59,340 --> 01:00:08,640
Yeah, like, kind of like, why settle for accountability, because often accountability is after the fact, you know, like, why can't we build systems that liberate us?

513
01:00:08,640 --> 01:00:11,060
Why can't we build systems that heal us?

514
01:00:11,060 --> 01:00:16,560
Like, I feel like we, we have the power to imagine just much more expansively.

515
01:00:16,560 --> 01:00:25,620
So like, it doesn't always have to be after the fact that we're trying to, like, you know, repair things or whatever, we can really, we can really aspire to do much more.

516
01:00:25,620 --> 01:00:31,160
So co-liberation means, it's this idea that none of us are free until all of us are free.

517
01:00:31,160 --> 01:00:46,840
So we work with the most marginalized groups and center, like, this is a case where we would like, build a data set around whoever is most marginalized in whatever the application area is, and we would work with them directly, and they would be part of the process.

518
01:00:47,040 --> 01:00:54,080
And whatever is built, the system that is built would like, actively serve their needs in a really culturally grounded way, you know.

519
01:00:54,420 --> 01:00:57,480
And so I think that's totally feasible.

520
01:00:57,480 --> 01:00:59,600
It's just not the way that things are done right now.

521
01:01:00,440 --> 01:01:01,920
I fully agree.

522
01:01:01,920 --> 01:01:04,380
I think these are great points, right?

523
01:01:04,380 --> 01:01:07,600
Ultimately, we have to kind of rework the whole framework.

524
01:01:07,800 --> 01:01:15,720
But I think to begin with, even, right, like, the way these systems are now just kind of put out with almost no testing, right?

525
01:01:15,720 --> 01:01:32,060
And, and then only kind of somebody like, kind of, right, brings up the problem, usually a minority researcher, and then the company is doing everything to push back and use all the resources to discredit the researcher, I think, at least we should change that conversation.

526
01:01:32,060 --> 01:01:49,800
And I think that can happen, especially, you know, with the viewers of this panel, we need more hackers into these AI systems, go hack every AI system out there, reveal all the vulnerabilities, take it down, rebuild a new one.

527
01:01:49,800 --> 01:01:54,440
Yeah, you can have like a brigade of like algorithmic bias assessors.

528
01:01:54,440 --> 01:01:58,300
I mean, good trolls, good trolls.

529
01:01:59,840 --> 01:02:17,280
To kind of move this towards, you know, advice or takeaways from the audience, because this has been a question popping up in the chat a lot, you know, so you guys, you know, you guys are talking about how there's definitely steps that people can take to make progress in the way that they design their algorithms to be more inclusive and stuff like that.

530
01:02:17,280 --> 01:02:31,160
And one thing that kind of stood out to me is being implicitly present in a lot of what you guys said, but I don't think anyone said this explicitly, is that kind of the default view of AI is it's supposed to be descriptive, you take a description of the world as it was,

531
01:02:31,160 --> 01:02:36,020
and you put that through an algorithm, and the role of the algorithm is to make the future look like the past.

532
01:02:36,020 --> 01:02:40,120
And so in a very real sense, AI is intrinsically regressive.

533
01:02:40,460 --> 01:02:55,200
But at the same time, nobody, there's nobody saying you have to do that, you know, you can decide that you want to choose to value certain things more than others, and you want to kind of bring about certain impacts, and you can proscriptively put those into your algorithms.

534
01:02:56,580 --> 01:03:06,520
Personally, I have tried to advocate for this, and largely speaking, failed, because people are scandalized by the idea that you might tell the data how it's supposed to look.

535
01:03:09,300 --> 01:03:28,400
But you know, so if you're a, if you're a data scientist, or a machine learning research engineer, listening to this chat, you know, can, can, how can they proscriptively design their, their algorithms for equity, for justice, for liberation, for whatever all these very nice words are,

536
01:03:28,400 --> 01:03:32,160
that are better than bias, or fairness, I suppose.

537
01:03:32,660 --> 01:03:36,440
You know, what does that, what does that look like to design an algorithm in that fashion?

538
01:03:41,100 --> 01:03:44,120
I mean, I'll start, and then hopefully others can build.

539
01:03:44,120 --> 01:03:53,160
But like, I mean, I think what it would look like is to do, to really think about where, where what your work is doing is being applied, and on whom, right?

540
01:03:53,160 --> 01:04:07,180
And like, who you're making it for, like, who's your boss, and then like, who are ultimately like the end users, or who is kind of impacted, who are the decisions being made about because of the system, and to do basically like a power analysis of that system.

541
01:04:07,780 --> 01:04:24,820
And then think about what is, that application area, who has been systematically disenfranchised, and then work with the groups who are working towards a better future in that domain to co-create, like what the metrics should be for the future.

542
01:04:24,820 --> 01:04:33,680
You know, I mean, I think that we're not asking, I don't, I don't think it's like appropriate that like, like a machine learning engineer would like invent the metrics out of their head or something.

543
01:04:33,680 --> 01:04:34,220
You know what I mean?

544
01:04:34,220 --> 01:04:49,780
Like in a domain space, they may not know very much about, but I think it is feasible that like a company could work really well with a group of folks, say, working on prison abolition or something like that, and could craft like here are the metrics for what the world that we would like to see.

545
01:04:49,780 --> 01:04:52,740
And this is like the kind of outcomes that we want to see.

546
01:04:52,740 --> 01:04:56,520
And so that's, that would be my kind of generalized recommendation.

547
01:04:56,520 --> 01:05:05,500
But that said, and this is like one of the major tensions of this space, is that often people are building tools that are more generic, right?

548
01:05:05,500 --> 01:05:13,060
Like you're looking for generalizability, you're looking for abstraction and like that could be applied in a wide variety of different application domains.

549
01:05:13,060 --> 01:05:14,640
And that I think is a tension.

550
01:05:14,660 --> 01:05:22,700
I actually think that is like a major tension for AI doing good in the world and being culturally grounded and informed.

551
01:05:22,700 --> 01:05:23,860
I don't know how to resolve that.

552
01:05:23,860 --> 01:05:31,840
And I would love to hear what like the computer scientists and researchers have to say about if they see that tension as well or not.

553
01:05:32,740 --> 01:05:34,440
I fully agree.

554
01:05:34,440 --> 01:05:39,020
And I think to me that foundations of that goes back to the education, right?

555
01:05:39,020 --> 01:05:47,420
So computer science and engineering has had the military background, you know, it started as a way to get into the military and that's there today.

556
01:05:47,420 --> 01:05:50,580
Like, you know, we are told to keep our emotions aside.

557
01:05:50,620 --> 01:05:56,880
In fact, that was one of a lot of gaslighting on Twitter is about, oh, why are we emotionally charged about machine learning?

558
01:05:56,880 --> 01:05:58,940
Just focus on the math, you know?

559
01:05:58,940 --> 01:06:05,560
And so that's something that's pervasive in the educational system and then beyond in the companies, right?

560
01:06:05,560 --> 01:06:07,360
Like, you don't worry about this.

561
01:06:07,360 --> 01:06:09,400
You do, you know, you derive math.

562
01:06:09,400 --> 01:06:10,960
You just, you know, program.

563
01:06:10,960 --> 01:06:14,520
So you're asked to become a robot without any emotions.

564
01:06:14,520 --> 01:06:18,200
And I think that's just really dangerous, right?

565
01:06:18,200 --> 01:06:21,120
Because it's not every engineer who's the problem.

566
01:06:21,120 --> 01:06:34,900
I mean, there's absolutely a decision maker who is probably, right, kind of, you know, may or may not understand the implications, but it's kind of directing all this and leading to the pervasive bias and injustice.

567
01:06:35,240 --> 01:06:37,140
I think we have to start with education.

568
01:06:37,140 --> 01:06:38,740
That's where I would pin my hopes.

569
01:06:38,740 --> 01:06:43,360
For instance, at Caltech, we have a course on ethics for AI.

570
01:06:43,360 --> 01:06:45,720
I think every university is now doing that.

571
01:06:45,720 --> 01:06:48,500
Ultimately more humanistic education.

572
01:06:48,680 --> 01:06:51,080
You know, we are kind of, you know, I went to technical school.

573
01:06:51,080 --> 01:06:53,280
Now I'm in a technical school, right?

574
01:06:53,280 --> 01:06:57,440
The role of humanities there needs to be even further emphasized.

575
01:06:57,440 --> 01:07:04,200
You know, it's much better than when I went to school, but it's still a long way to go to integrate humanities and engineering together.

576
01:07:04,600 --> 01:07:06,700
I completely agree with Anima.

577
01:07:06,700 --> 01:07:08,580
I'm also teaching bias in AI.

578
01:07:08,580 --> 01:07:23,360
I think it needs to be part of the curriculum, not for computer scientists only, but also engineers, humanities, social scientists, probably as well, because data scientists or essentially anyone that is interested in computing is using these tools.

579
01:07:23,360 --> 01:07:28,700
So they need to understand that all of these tools exist within a context .

580
01:07:28,700 --> 01:07:33,520
They don't just exist in an abstract, generalizable way, as Catherine mentioned.

581
01:07:33,520 --> 01:07:43,720
It's correct that these tools or algorithms were developed so that they can be applied to hypothetical data sets so that the theory or foundations are presented.

582
01:07:43,720 --> 01:07:55,360
But then when we take social data and apply these tools for machine learning tests in the social domain, then we are dealing with all kinds of human complications.

583
01:07:55,360 --> 01:08:10,040
So people need to always keep in mind that these systems, yes, they might look objective and neutral, but they are not in the social context because they exist with all the problems or side effects that human decision making brings into the game.

584
01:08:10,040 --> 01:08:17,460
But moreover, AI also brings its own kinds of problems, such as new types of biases we are not even aware of yet.

585
01:08:17,460 --> 01:08:20,540
And these keep getting amplified in society.

586
01:08:20,540 --> 01:08:42,220
So before we start introducing more of those, we need to understand what we are bringing to the world and what we should be careful about, because all of these systems are deployed right now without understanding what side effects they might have or what the implications are on society,

587
01:08:42,220 --> 01:08:44,920
economy, democracy, and so on.

588
01:08:44,920 --> 01:08:50,860
And now we would need to deal with all of these consequences on top of all the other things.

589
01:08:53,780 --> 01:09:09,800
It seems like a major takeaway you guys all advocate for is to kind of broaden the scope of who gets included in the conversation about AI design, about AI data collection, you know, bringing in activists in the relevant spaces, bringing in other stakeholders.

590
01:09:10,880 --> 01:09:19,900
And, you know, one thing that I would like to add to that is that the notion of stakeholder is really politically fraught.

591
01:09:19,980 --> 01:09:35,820
In industry context, stakeholders are generally conceived of as like, you know, so I suppose I'm at a ML company developing an algorithm that I'm going to sell to another company to use for financial profits.

592
01:09:35,820 --> 01:09:48,120
You know, the stakeholders in this process are generally conceived as being me and my team, my superiors at my company, the people I'm selling the algorithm to, and the stockholders of their company.

593
01:09:48,700 --> 01:09:55,140
And then I think that misses a really crucial point, which is the stakeholders are the people whose data is going to be analyzed.

594
01:09:55,600 --> 01:10:04,780
And their voice is very, very frequently just completely left out of any discussion about how things are designed, why things are designed, and where they come from.

595
01:10:04,800 --> 01:10:17,260
And I mean, to an extent, partially by design, because a lot of companies exist out of the fact, because of the fact that they can make money analyzing user data, like that's their sole reason, that's their reason for existing.

596
01:10:17,260 --> 01:10:18,620
There are lots of these things.

597
01:10:19,700 --> 01:10:29,120
But broadening, I think broadening the notion of a stakeholder , in particular, viewing anyone whose data is going to be processed by an AI as a stakeholder is really important.

598
01:10:32,960 --> 01:10:35,120
Yeah, I just I totally agree with that.

599
01:10:35,120 --> 01:10:52,540
We have a chapter in our book, which is about the principle embrace pluralism, which is basically this idea that, you know, we create more complete knowledge, like the path to objectivity is not through individual genius, but it's through bringing multiple different perspectives to the table.

600
01:10:52,820 --> 01:11:01,680
But I agree with you, I think like the it's important to think about like, because we can say, oh, it was participatory, like that company with all the money and the other company with all the money.

601
01:11:02,580 --> 01:11:12,180
So I do think and this is because participation, I mean, there's this whole world, you know, participatory design and participatory methods.

602
01:11:12,700 --> 01:11:24,980
And so like, there's even folks have talked about participation washing, where, you know, you're like, oh, yeah, we did a, you know, this also happens in urban planning, like, oh, yeah, we did a public meeting and it was at like 10pm and there was no childcare.

603
01:11:27,460 --> 01:11:31,440
So like, what's the quality and the character of the participation?

604
01:11:31,440 --> 01:11:32,660
Like who shows up?

605
01:11:32,660 --> 01:11:34,340
Who are these stakeholders?

606
01:11:34,340 --> 01:11:37,740
And who are the stakeholders that are not convenient to have?

607
01:11:37,740 --> 01:11:39,680
And how do you make folks listen to them?

608
01:11:39,680 --> 01:11:45,480
I mean, I think that's a big, that's a big question when ultimately, like the goal is to optimize profit.

609
01:11:45,480 --> 01:11:47,760
That to me is a conundrum.

610
01:11:50,090 --> 01:11:53,930
I think how do you make people listen, I think is a really interesting question.

611
01:11:53,930 --> 01:12:06,930
You know, a lot of people, I imagine a lot of people feel like they would like to do something good, you know, they would like to be more just and do their design the right way.

612
01:12:07,090 --> 01:12:19,310
But you know, how do you convince your boss or your boss's boss who controls the money and the funding, or the money and the decision making power, and who's maybe more interested in what's profitable?

613
01:12:19,350 --> 01:12:23,510
You know, how do you do any of you guys have advice about how to approach that conversation?

614
01:12:23,890 --> 01:12:37,330
Or how to how to advocate for justice in AI design, in context where that input is not necessarily the primary concern of the people who pull the strings?

615
01:12:38,530 --> 01:13:00,950
I'm interested in that as a research question, if we can figure out a way to simulate AI systems to show their long term gains and losses for all the parties that are involved, I think we would be able to understand how this affects economy in very from various perspectives.

616
01:13:00,950 --> 01:13:03,490
But again, this is a hypothetical research question.

617
01:13:03,490 --> 01:13:06,610
I'm sure there's already some great research on this yet.

618
01:13:06,610 --> 01:13:15,430
But maybe long term goals might be able to convince people that are profit driven as well.

619
01:13:17,230 --> 01:13:41,190
I mean, especially if there's concern, I really like the idea a lot, especially if there's concern, I mean, as there's this growing sort of industry of sort of impact assessment, or algorithmic bias assessment, and things like this, this is like, bringing folks in and doing a more participatory design process is a preventative for what could happen later,

620
01:13:41,190 --> 01:13:49,990
and so I think maybe that's the, like, if the case needs to be economic rather than moral, I mean, you can make the moral case, but probably what that listens to is the economic case, right?

621
01:13:50,010 --> 01:14:06,230
But if the case needs to be economic, I think that, you know, it would be lovely to have some quantifiable research that you could cite to say that like, yeah, there's this like, very strong economic case that this will reduce the number of lawsuits later,

622
01:14:06,230 --> 01:14:10,070
that you'll be fully compliant with all federal regulations, but you know what I mean?

623
01:14:10,070 --> 01:14:24,410
Like this, these kinds of things, I think that that might be the path for convincing the employers, and like, ultimately, like, you've invested in prevention, so that you don't have to clean up a big mess after the fact.

624
01:14:27,970 --> 01:14:33,070
That's why academia and the independent researchers have such an important role, right?

625
01:14:33,070 --> 01:14:39,410
Because they can talk openly about these issues, like, you know, what happened with the studies, right?

626
01:14:39,410 --> 01:14:50,810
Even though internally, there was a lot of conversation at Amazon, when I was there, you know, having that external light into it, and then the wider public part spreading in it, I think is critical.

627
01:14:51,570 --> 01:14:56,650
Because, you know, like being in these companies, it's just the whole system is so short term driven, right?

628
01:14:56,650 --> 01:15:02,690
The Wall Street expects quarterly results, that's not delivered, the company won't even exist, right?

629
01:15:02,690 --> 01:15:06,690
So the whole system is not built for the long term implications.

630
01:15:06,690 --> 01:15:09,310
And I think that's what creates these challenges.

631
01:15:09,790 --> 01:15:26,090
And so somehow, maybe then having more participation from the public, right, can show that also, you know, bad AI can lead to bad PR and loss of profits, even in the short term, I think that activism and awareness is critical to change this.

632
01:15:28,630 --> 01:15:30,210
Ram, you want to jump in?

633
01:15:30,210 --> 01:15:36,850
Yeah, I was just framing about what Anima said.

634
01:15:37,010 --> 01:15:51,330
But the way I think about this is, at least again, through lived experience, I can say that what has changed for me is that if you think of like inclusion, as after the fact, that's kind of like really weird.

635
01:15:51,410 --> 01:15:57,810
One of the things that I'm very lucky is that inclusion is part of my core priorities that I set.

636
01:15:57,930 --> 01:16:01,210
Like, it's just not for me, it's for everybody in the company.

637
01:16:01,450 --> 01:16:04,010
Like, you've got to set that, what is your inclusion priority?

638
01:16:04,010 --> 01:16:09,990
For me, it might be like, you know, I'm going to take care of Pride Month, like, you know, talks or, you know, whatever things, right?

639
01:16:09,990 --> 01:16:12,130
So obviously, it doesn't solve all problems.

640
01:16:12,130 --> 01:16:24,370
So first of all, I fundamentally believe that if your organization is questioning why you're doing something ethical, that itself is a big red flag for me.

641
01:16:24,990 --> 01:16:33,650
Right, like, and for the business case, I do agree, like, you know, there's this, like, you know, how Catherine said, there could be some financial motivation.

642
01:16:33,650 --> 01:16:42,810
I saw like an HPR article about like, how diversity leads to more surplus, because you're selling your product to more people.

643
01:16:43,030 --> 01:16:45,330
That could be like an other way.

644
01:16:45,630 --> 01:16:50,110
The PR angle, like, I mean, everybody is afraid of like bad marketing.

645
01:16:50,150 --> 01:16:51,830
I'd never actually thought about that.

646
01:16:51,830 --> 01:16:53,370
So thanks for bringing that up.

647
01:16:53,430 --> 01:16:59,310
But for me, intrinsically, you know, I have never been told you can't publish this.

648
01:16:59,310 --> 01:17:02,310
I mean, but again, it's a very different setting, like MSR.

649
01:17:02,310 --> 01:17:04,650
And like, you know, product setting is very different.

650
01:17:05,150 --> 01:17:15,830
But yeah, I would be very uncomfortable if I have to make a case to my manager as to why I'm doing something ethical in the first place.

651
01:17:15,830 --> 01:17:22,770
But like Anima pointed out, like, business, I've only worked in one organization for like, my working life.

652
01:17:22,770 --> 01:17:24,770
I do not know how others operate.

653
01:17:24,770 --> 01:17:28,370
I know some of the CEOs out there question why they should use pronouns.

654
01:17:28,610 --> 01:17:36,530
So or like, why they should like, you know, why they do not like fundamentally care about democracy and just want to dismantle it.

655
01:17:36,530 --> 01:17:40,110
So I don't know, I do not know how things work there from my lived experience.

656
01:17:40,290 --> 01:17:42,870
I don't know if this might actually be helpful for others.

657
01:17:45,840 --> 01:17:52,600
All right, so it seems like the kind of two major points you guys have raised is that you can view it as a risk mitigation factor.

658
01:17:53,600 --> 01:17:57,260
Because it will stop things from it can stop things from going wrong in the future.

659
01:17:57,260 --> 01:17:59,500
And it's in some sense insurance on that.

660
01:17:59,500 --> 01:18:04,160
And it's also a PR thing, which is partially a risk mitigation factor.

661
01:18:04,160 --> 01:18:13,320
You really don't want the New York Times running an article saying so and so's company develops algorithms systematically discriminates against women in hiring.

662
01:18:13,780 --> 01:18:16,240
Oh, I actually saw a headline.

663
01:18:16,240 --> 01:18:16,620
Yeah.

664
01:18:16,620 --> 01:18:17,660
Oh, sorry.

665
01:18:17,660 --> 01:18:18,160
Yeah.

666
01:18:18,160 --> 01:18:19,340
I actually saw an email.

667
01:18:19,340 --> 01:18:24,040
I only saw it through your tweet about gentrify or genderify.

668
01:18:24,400 --> 01:18:35,440
Yeah, another thing after all the right outcry got withdrawn, right?

669
01:18:35,440 --> 01:18:43,140
The fact that it even got launched is just appalling the kind of lack of awareness these companies.

670
01:18:43,500 --> 01:18:44,920
Yeah, it was shocking.

671
01:18:44,920 --> 01:18:47,240
You know, there was another company.

672
01:18:49,100 --> 01:18:51,320
I don't know who alerted me to it.

673
01:18:51,320 --> 01:19:03,060
But essentially, like their solution for more representation on stock photos was to kind of like artificially generate like photos using GANs.

674
01:19:03,060 --> 01:19:06,620
So you will not actually pay money to hire a black model.

675
01:19:06,620 --> 01:19:12,020
Oh, let's just use StyleGAN and like to kind of like, like, how do you get through?

676
01:19:12,020 --> 01:19:17,960
Like, who's funding this and goes back to an email like initial comment about VCs getting excited.

677
01:19:17,960 --> 01:19:20,860
So that I feel is a bigger like thing.

678
01:19:20,860 --> 01:19:32,240
I would be very less less worried about managers and more worried about like plutocratic like Silicon Valley folks to then like actually managers.

679
01:19:32,660 --> 01:19:44,100
So to kind of contextualize this for people in the audience who aren't aware, Genderify is a product that existed for all of about four hours.

680
01:19:46,900 --> 01:19:56,960
What their business model was, was that you can send them user data, you can send them photos, you can send them people's names, and they will

681
01:20:00,440 --> 01:20:20,620
they say they will determine the gender of the people that you in that you sell to or the people that are in your data set with an aim of doing better predictive marketing and stuff based on the... I'm having trouble coming up with a word other than assumed gender of the of the people in your data set.

682
01:20:21,240 --> 01:20:40,240
And this was this was funded by some VCs and it went online and everybody including lots of people in the ML space who don't really care very much, who don't talk very much publicly about about ML bias lost their shit because it was a completely absurd idea.

683
01:20:40,620 --> 01:20:47,140
And just to give an example, right, the word professor was only 1.9% females.

684
01:20:47,160 --> 01:20:50,980
So that's the assigned percentage for me in terms of gender.

685
01:20:50,980 --> 01:20:53,600
So I think that that was the problematic thing.

686
01:20:53,600 --> 01:21:01,340
Like how could you just type any name, like any word, not just name, any word in the vocabulary and assign a gender to that?

687
01:21:01,340 --> 01:21:04,140
I mean, and only binary genders, right?

688
01:21:04,140 --> 01:21:06,680
It's just appalling the whole concept.

689
01:21:06,760 --> 01:21:14,180
Yeah, my friend was saying the only time that's allowed it's in French, any other language, like, you know, because, you know, nouns have gender.

690
01:21:14,260 --> 01:21:17,820
Any other time table doesn't have like a male or female.

691
01:21:17,820 --> 01:21:18,600
Yes.

692
01:21:18,700 --> 01:21:19,560
Anyway.

693
01:21:19,700 --> 01:21:23,620
Yeah, but I'll know that like there are like this product is insane.

694
01:21:23,620 --> 01:21:30,120
But there are other products like there's all these gender detection libraries out there, both for faces.

695
01:21:30,140 --> 01:21:44,760
So like facial detection, which the researcher Oz Keys has written this amazing paper called Misrecognition Machines, which is about how harmful face like gender detection of faces is for trans folks.

696
01:21:44,860 --> 01:21:53,340
But then there's also all sorts of libraries that infer gender from name using various kinds of like name databases.

697
01:21:53,340 --> 01:22:04,600
And like, maybe I'll be like, sort of a naysayer here, because like, in general, I feel like these applications are very well, again, it depends on the application domain.

698
01:22:04,600 --> 01:22:17,840
So like, if you're doing this for predictive marketing, or like Oz talks about how they're doing it for bathroom access, which is like, disgusting, you know, like, are you the right gender to get into the bathroom or whatever?

699
01:22:18,480 --> 01:22:20,800
Those are super harmful and awful.

700
01:22:20,800 --> 01:22:22,840
And like those kind of applications shouldn't exist.

701
01:22:22,840 --> 01:22:41,480
At the same time, these gender detection libraries are helpful for doing like, bibliometric studies, where you're looking across like a huge corpus of say, like, I don't know, like several million authors and trying to see like, what's the over representation of men in that data set or something like that,

702
01:22:41,480 --> 01:22:50,340
you know, and so like, that that's why like, again, I think it's hard to just say categorically, hmm, something is bad or good.

703
01:22:50,340 --> 01:23:01,600
Although I do have certain things that I do think that way about, but like, that could enable really interesting research that monitors back on the patriarchy, for example.

704
01:23:01,620 --> 01:23:06,000
And so like, again, it's like, who's using it?

705
01:23:06,000 --> 01:23:06,740
For what?

706
01:23:07,740 --> 01:23:09,760
Yeah, like, that's what matters.

707
01:23:09,760 --> 01:23:15,860
And just to give an example, we have used the gender detector, but that was to learn about the MeToo conversation.

708
01:23:15,860 --> 01:23:23,520
So we, you know, collect lots of MeToo data from Twitter, men talking about it, how are women talking about it?

709
01:23:24,160 --> 01:23:26,500
How are Democrats, Republicans?

710
01:23:26,800 --> 01:23:34,040
So yeah, so that kind of, so you know, and this is in collaboration with Professor in Sociology, Mike Alvarez.

711
01:23:34,040 --> 01:23:36,840
So you know, like, the context certainly matters.

712
01:23:36,840 --> 01:23:45,820
We can't just ban all applications of gender detection, but really contextualize and look at whether it's for harm or for good.

713
01:23:45,860 --> 01:23:46,120
Yeah.

714
01:23:46,120 --> 01:24:03,100
And in medical applications, we especially need very accurate biased representations, because we know that for certain disease, the diagnosis and prognosis depends on ethnic factors, gender, and socioeconomic background as well.

715
01:24:03,100 --> 01:24:08,820
Of course, there are biases representing injustices in these medical data sets.

716
01:24:08,820 --> 01:24:16,580
But at the same time, there are biases that we need to learn from for more accurate medical treatment.

717
01:24:16,580 --> 01:24:26,600
And I guess that's why, for example, in the medical domain, machine decision making is becoming much more accurate than human decision making.

718
01:24:26,600 --> 01:24:32,220
But again, yet we need to deal with all the other disadvantages that come with this.

719
01:24:32,220 --> 01:24:39,680
So again, I guess everyone keeps mentioning this, it really depends on the context where we are using these things.

720
01:24:39,680 --> 01:24:45,820
And name-based gender detection is a kind of an outrageous example.

721
01:24:45,820 --> 01:25:04,500
But then there is a supercharged version of this in country X, where machine learning is used to predict the criminality of a person from their face, or actually give people social scores so that they can be sent to workers camps for their entire lives,

722
01:25:04,500 --> 01:25:06,480
because the machine said so.

723
01:25:06,480 --> 01:25:12,380
So there's a spectrum of these effects that we can discuss.

724
01:25:12,380 --> 01:25:19,660
And then what foundations should we be building on, given these examples that already exist in the world?

725
01:25:21,980 --> 01:25:23,780
Well, thanks for sharing that.

726
01:25:23,780 --> 01:25:26,720
I was just, I completely dismissed it, Catherine.

727
01:25:26,720 --> 01:25:31,620
And thank you, Anima and Eileen for, you know, correcting this.

728
01:25:31,620 --> 01:25:32,660
Thank you.

729
01:25:33,180 --> 01:25:35,720
I didn't mean to correct anything, by the way.

730
01:25:35,720 --> 01:25:38,520
Oh no, it was like, you know, I learned something.

731
01:25:38,520 --> 01:25:40,180
So thanks for telling me about this.

732
01:25:40,180 --> 01:25:40,920
Yes.

733
01:25:41,280 --> 01:25:42,280
I appreciate it.

734
01:25:42,280 --> 01:25:44,140
What did you learn, Ram?

735
01:25:44,220 --> 01:25:48,180
Well, I just completely dismissed like gender detection.

736
01:25:48,180 --> 01:25:51,160
I was like, oh, this is like, has literally no use.

737
01:25:51,160 --> 01:25:56,620
If I want to ask somebody's gender, I will just ask them if it's part of my business.

738
01:25:56,620 --> 01:25:57,900
I'll say it's none of my business.

739
01:25:57,900 --> 01:25:58,900
I'm not going to ask.

740
01:25:58,900 --> 01:26:04,900
But when, I was like, okay, that kind of makes sense.

741
01:26:04,920 --> 01:26:08,260
And when Eileen spoke about the medical contacts, that makes sense.

742
01:26:08,320 --> 01:26:12,680
So yeah, this was, like I said, I already learned something from this panel.

743
01:26:12,680 --> 01:26:16,680
And within like two more minutes for the panel to get over, I learned something.

744
01:26:16,680 --> 01:26:17,920
I learned a lot of things.

745
01:26:20,960 --> 01:26:30,660
Yeah, the example that Eileen brought up of criminality prediction is unfortunately running rampant in the AI literature right now.

746
01:26:31,160 --> 01:26:45,940
I feel like there is a half dozen papers that have been published and then promptly retracted after a massive public outcry, because they're basically trying to like reinvent phrenology, but doing it with machine learning, and therefore it's good.

747
01:26:47,000 --> 01:26:55,120
There's a whole lot that can be said about this that could and actually almost was its own panel at DEF CON.

748
01:26:56,100 --> 01:27:16,240
But there's a really good paper on this called Criminality from Facial Illusion, which I'll send out on Discord and on Twitch to people interested in reading about this problem that does a really good job of picking apart a lot of the issues with both morally and also technologically,

749
01:27:16,240 --> 01:27:22,500
because a lot of these papers are fundamentally technologically unsound with these kinds of works.

750
01:27:23,300 --> 01:27:26,880
So we're approaching an hour and a half.

751
01:27:27,540 --> 01:27:29,460
This is the last event of the day.

752
01:27:29,460 --> 01:27:31,480
I'm enjoying talking to you guys a lot.

753
01:27:31,480 --> 01:27:35,760
I don't know if you have somewhere you need to be, but I understand that.

754
01:27:35,760 --> 01:27:41,600
But if you guys are interested in continuing to talk, I am more than happy to do so.

755
01:27:42,580 --> 01:27:45,480
I have to go make dinner for my family.

756
01:27:49,740 --> 01:27:54,440
And it's actually sunny for a very short period of time in Seattle.

757
01:27:56,360 --> 01:27:59,560
Trying to sneak in like a run.

758
01:27:59,560 --> 01:28:06,660
Stella, before we all disperse, I first want to thank you so much for organizing this.

759
01:28:06,660 --> 01:28:10,320
You know, you have put so much thought and intention into this.

760
01:28:10,320 --> 01:28:14,940
Like, I remember like asking you like, Hey, I don't want to be the male in this.

761
01:28:14,940 --> 01:28:18,760
Like, have you thought about inclusive panel?

762
01:28:18,760 --> 01:28:19,940
I don't want to be a male.

763
01:28:19,940 --> 01:28:25,740
You wrote like, perhaps the most comprehensive male, you know, about this.

764
01:28:25,740 --> 01:28:28,980
So thank you so much for, you know, moving with intention, Stella.

765
01:28:28,980 --> 01:28:32,420
It clearly shows when you organize this panel.

766
01:28:32,420 --> 01:28:33,260
Thank you.

767
01:28:34,680 --> 01:28:35,380
Thank you.

768
01:28:35,380 --> 01:28:35,860
Second.

769
01:28:35,860 --> 01:28:36,560
Thank you.

770
01:28:36,560 --> 01:28:37,520
This is amazing.

771
01:28:37,520 --> 01:28:41,420
I learned so much and it's great to connect with all of you.

772
01:28:41,420 --> 01:28:42,540
And thanks, Stella.

773
01:28:42,540 --> 01:28:44,840
I enjoyed discussing with all of you.

774
01:28:44,840 --> 01:28:46,260
I think this was productive.

775
01:28:46,260 --> 01:28:49,400
And now I have more research ideas that I can go on.

776
01:28:51,560 --> 01:29:03,460
So quickly before you guys run, I want to take advantage of the fact that I have several experts here to get your thoughts on something I've been working on recently, which we've, we've almost touched on this several times.

777
01:29:03,460 --> 01:29:09,140
And it's that a lot of times machine learning algorithms have a underlying ontology.

778
01:29:09,140 --> 01:29:20,600
So Catherine mentioned, you know, that some gender algorithms exclude transgender people, you know, but even when you're talking about like all sorts of other applications you know, I work a lot with U.S.

779
01:29:20,600 --> 01:29:22,080
government data scientists.

780
01:29:22,080 --> 01:29:24,420
And so they're working with data sets that come from the U.S.

781
01:29:24,420 --> 01:29:24,780
census.

782
01:29:24,780 --> 01:29:25,120
And the U.S.

783
01:29:25,120 --> 01:29:33,260
census is a very specific notion of what races, and there isn't an option to use anything else because you're, you know, you're the U.S.

784
01:29:33,260 --> 01:29:34,600
government and you're using U.S.

785
01:29:34,600 --> 01:29:35,600
census data.

786
01:29:36,260 --> 01:29:50,340
But kind of how you, how you define your categories, you know, what, how you choose who gets labeled as white, who gets labeled as Asian, really makes a big impact on what your algorithm does and how it behaves.

787
01:29:50,340 --> 01:29:55,120
And whether you include certain categories, you know, if you include transgender people or not.

788
01:29:55,220 --> 01:30:09,260
I mean, if transgender, if non-binary transgender people, people who don't identify as, as, as a man or as a woman is not a possible label, then you have a guaranteed 100% incorrect assignment rate for them.

789
01:30:10,660 --> 01:30:17,660
So, you know, do you have any, any thoughts or advice about how to kind of tackle, you know, how to, how to approach this problem?

790
01:30:17,660 --> 01:30:24,740
Obviously, originally, we can go and talk to the impacted communities and try to figure out what the right buckets are.

791
01:30:24,740 --> 01:30:30,720
But oftentimes we start from a position where our data has been pre-collected for us by someone else.

792
01:30:30,720 --> 01:30:39,520
And so if you have data that has this kind of problem in the way the questions were asked originally, you know, is there something you can do with it still?

793
01:30:40,300 --> 01:30:43,180
I think that's a great question, right?

794
01:30:43,180 --> 01:30:46,640
And a difficult one because, you know, the systems are already built.

795
01:30:46,640 --> 01:30:48,680
Like you were saying, the categories are there.

796
01:30:48,680 --> 01:30:50,340
This is the label data.

797
01:30:50,620 --> 01:31:00,380
So I think towards getting to solutions for this, I think an important step is being able to trust the system in terms of confidence levels, right?

798
01:31:00,380 --> 01:31:04,620
Like, can the system say truly how confident it is?

799
01:31:04,620 --> 01:31:07,620
And the standard neural networks don't have this property.

800
01:31:07,620 --> 01:31:10,180
So it can make high confidence errors.

801
01:31:10,900 --> 01:31:14,680
You know, it can be like 95% confident and be wrong, right?

802
01:31:14,680 --> 01:31:23,500
And that especially happens on these kind of hard examples, which hasn't seen in training data because it also gets the calibration wrong.

803
01:31:23,520 --> 01:31:26,860
In one of our papers, we call it angular visual hardness.

804
01:31:26,860 --> 01:31:39,060
We proposed a new angular measure for detecting these hard examples that was more robust than the standard accuracy scores and the confidence scores that the systems are built on.

805
01:31:39,060 --> 01:31:42,400
And we also have some mathematical intuitions for that.

806
01:31:42,400 --> 01:31:50,600
But the outcome of this is we can now look at, you know, examples which are hard for the current classifier, right?

807
01:31:50,600 --> 01:31:59,280
So in this gender detection example, it would be most likely the transgender people who would fall as hard examples because it's not getting this accurate, right?

808
01:31:59,280 --> 01:32:08,540
No matter what binary gender the person is assigned to and if they don't kind of, you know, display that gender, right?

809
01:32:08,540 --> 01:32:13,780
So there is that issue that can be detected through these mathematical means.

810
01:32:13,840 --> 01:32:17,880
And then I think at least there is now the problem that has come up, right?

811
01:32:17,880 --> 01:32:24,020
That not every example is equally treated and there are hard examples for this classifier.

812
01:32:24,020 --> 01:32:27,440
We can go back and then design a system to correct that.

813
01:32:27,860 --> 01:32:39,380
And on a related note, I will give a vague answer to this before I actually ask an even more daunting or deeper question.

814
01:32:40,140 --> 01:32:49,360
Maybe an email, for example, idea of using more unsupervised approaches might be some kind of a solution for future generations of AI algorithms.

815
01:32:49,360 --> 01:33:01,300
Maybe you wouldn't need to categorize yourself as a person, but instead some kind of more unsupervised continuous representation of social entities might be a way to approach this problem.

816
01:33:01,300 --> 01:33:13,200
This is not going to happen anytime soon, so I don't really have an answer for today's problems because we know that gender is reflected to other features, all other features as proxies as well.

817
01:33:13,200 --> 01:33:18,680
Because when we think about gender, that's the largest social group in society.

818
01:33:18,680 --> 01:33:22,080
So it's one of the biggest proxies in the data sets.

819
01:33:22,080 --> 01:33:30,260
But my more concerning question is semantics, gender, and race is defined in categories.

820
01:33:30,260 --> 01:33:47,820
So it's a proxy, but when you think about it, essentially race doesn't really exist or gender might not even exist, but we have it in our language, in our linguistic system, and that's the way we are constructed to deal with the world.

821
01:33:47,820 --> 01:33:56,960
So how are we going to change these problems that are social constructs in AI systems that we are building on these social constructs?

822
01:33:56,960 --> 01:34:06,080
So it's a longer-term problem that might require longer-term solutions, which might not be very helpful for your question.

823
01:34:07,100 --> 01:34:17,980
Yeah, that really gets us something unfortunate, and a lot of people in the Discord are asking applied questions that I haven't raised because I know that the answer is, we don't know how to do that.

824
01:34:18,040 --> 01:34:20,780
We would love to know how to do that too.

825
01:34:21,500 --> 01:34:33,440
So one example question is, if you can't bring in experts or social scientists or stakeholders, is there an algorithmic way to kind of pick up on the bias?

826
01:34:33,440 --> 01:34:37,780
And the answer is, well, there's 50 and we don't know which ones work better in different contexts.

827
01:34:38,080 --> 01:34:43,720
I mean, but there is an algorithmic way to educate yourself, and that's like search engines, right?

828
01:34:44,380 --> 01:34:45,360
That's what?

829
01:34:45,560 --> 01:34:52,920
That's like, that could be search engines, just like, you know, go type, like, you know, if you want to learn about bias, go read Aileen's papers.

830
01:34:52,920 --> 01:34:58,320
If you want to learn about, like, you know, bias, go watch, like, Timnit Gupru's, like, workshops.

831
01:34:58,400 --> 01:35:10,020
So I think a lot of this hands-on things, people, smart people have already put out content, and it's just like a matter for us to spend, like, cycles to educate ourselves on.

832
01:35:11,060 --> 01:35:22,660
Yeah, and as a, like, very basic kind of check, like, you can look at the output of your system, if it has to do with people, and just look across intersectional groups.

833
01:35:22,660 --> 01:35:41,040
So, like, you do, like, assuming that you have gender data and race data, maybe you have income data, or maybe you have age, or maybe you have, like, some other kinds of, you know, dimensions that describe people, is just looking at disparate outcomes within each thing,

834
01:35:41,040 --> 01:35:46,960
and then combining those in various different ways to see if there are disparate outcomes amongst the categories.

835
01:35:46,960 --> 01:35:56,140
Because, like, the theory of intersectionality from black feminism says that, like, these categories compound in, like, a non-linear way, right?

836
01:35:56,140 --> 01:36:04,380
So being, experiencing sexism as a white woman is fundamentally very different from experiencing sexism and racism as a black woman.

837
01:36:04,380 --> 01:36:16,100
So, I mean, I think that's a very basic kind of self-audit that you can do when you're working with data that's about people, that, like, you don't need all that much, like, special training, necessarily.

838
01:36:16,100 --> 01:36:17,860
But, I mean, I think other people are working on this, right?

839
01:36:17,860 --> 01:36:30,680
I mean, I feel like you guys would know this better than me, but, like, folks have been working on things, like, where you would substitute out, like, you do, like, a sort of counterfactual, and you look at, well, what if this data point, like, they have this outcome,

840
01:36:30,680 --> 01:36:34,980
but what if they had, their race had been white, would they have had the same outcome with my system?

841
01:36:34,980 --> 01:36:38,360
So I think people are working on those kinds of methods.

842
01:36:38,360 --> 01:36:41,940
I think just a really basic one is just do a kind of an intersectional audit.

843
01:36:44,480 --> 01:36:49,860
Well, Stella, I, for one, look forward to reading your work when it comes out.

844
01:36:50,280 --> 01:36:51,800
Thank you so much for joining.

845
01:36:51,980 --> 01:36:53,920
Thank you guys so much for joining us.

846
01:36:53,920 --> 01:37:02,660
We've gone horribly over, but I've had a wonderful time talking to all of you, and I know that the people listening really enjoyed it.

847
01:37:02,660 --> 01:37:17,660
If anyone wants to continue this conversation who's listening, if you head over to the DEF CON Discord, there's a voice channel where I'm happy to continue to talk to anyone who wants to ask questions or get references to papers that people brought up and such.

848
01:37:17,880 --> 01:37:20,780
Thank you again to all of our wonderful panelists.

849
01:37:20,780 --> 01:37:23,480
This has been amazing.

850
01:37:23,480 --> 01:37:24,900
So thank you very much.

851
01:37:24,900 --> 01:37:25,860
Thank you.

852
01:37:25,860 --> 01:37:26,540
Thank you.

853
01:37:27,100 --> 01:37:29,740
Wonderful to talk with all of you.

854
01:37:29,780 --> 01:37:30,480
All right.

855
01:37:30,480 --> 01:37:31,560
Bye.

856
01:37:33,020 --> 01:37:45,900
I was also going to say we've also, we're going to have a journal club focused on ethics and privacy with AI starting on the 18th.

857
01:37:45,900 --> 01:37:54,220
It's organized by Suhaka on Thing, and you can check out our website for a link to a description of the whole thing.