start	end	text
600	6700	All right, welcome. This is the AI Village panel on AI ethics, where we're going to discuss the
6700	11800	ethical implications of AI technologies and especially their deployment both in general
12340	18580	and also specifically in data science and computer security. We have four lovely panelists
18580	24780	from a variety of backgrounds here to discuss the topic with us. Catherine is a HackerMOMA
24780	31140	scholar and artist who focuses on feminist technology, data literacy, and civic engagement.
31480	37020	She wrote a 2020 book called Data Feminism, co-authored with Lauren Klein, where she
37020	42220	charts a course for more ethical and empowering data science practices. Catherine is an assistant
42780	48300	professor of urban science and planning at the Department of Urban Studies and Planning at MIT,
48300	54100	where she's also the director of their Data Plus Feminism Lab. Ram is a data cowboy within the
54100	59700	trustworthy ML team at Microsoft, working on the intersection of machine learning and security.
59700	64700	His team's goal is to empower engineers to develop and deploy ML systems securely.
64740	69220	Ram is also an affiliate at the Berkman Klein Center at Harvard University, where he explores
69220	75640	the societal implications of adversarial AI. Ai-Lin is an assistant professor of computer
75640	82180	science at George Washington University and at the Institute for Data, Democracy, and Politics.
82180	87780	She is interested in AI ethics, bias in AI, machine learning, and the implications of machine
87780	93000	learning on privacy, security, fairness, and society. Much of her research is on methods to
93000	99340	interpret AI and gain insight into the bias in both artificial and natural intelligence.
99680	106180	And she has worked to demonstrate how human-like biases can crop up in natural language analysis
106700	113920	due to the statistical regularities of language. And finally, Anima is the Bren Professor at Caltech
113920	119640	and the Director of ML Research at NVIDIA. She was previously a principal scientist at Amazon
119640	126820	Web Services. She's received several honors, such as the Alfred P. Salone Fellowship,
126820	134540	NSF Career Award, Young Investigator Awards from DoD, and several more. She is part of the
134540	138940	World Economic Forum's Expert Network and is passionate about designing principled AI
138940	145900	algorithms and applying them to interdisciplinary outcomes. She has been outspoken on AI bias and
145900	151640	the need for democratization and focuses her research on unsupervised AI optimization and
151640	156940	tensor methods. They are all wonderful people and I can personally vouch for the fact that
156940	162540	super cool research and I'm more than thrilled to talk with them all.
162600	169700	So to kind of get us started, there's been a lot of conversation, at least recently, about AI bias
169700	175280	and misuse of AI in the news in the United States recently. A lot of tech companies have talked
175280	180760	about how they're going to stop using facial recognition algorithms or stop providing them
180760	190740	to police to use. And there's also been a lot of conversation both in the academic literature and
190740	196200	also in the national news media about biased AI, the AIs that learn to be racist because they look
196200	201460	at racist data or learn to be sexist because they look at sexist data. But AI ethics is kind of a
201460	206940	much broader topic than that. So to kick us off, can each of you say a little bit about what the
206940	212800	scope of the field is to you and how it relates to the work that you do? Ram, why don't you give us a start?
213040	219940	Oh, thank you. First of all, I'd like to say that I'm not an AI ethics expert. I'm so glad that Anima,
219940	227080	Catherine, Eileen are here along with Stella. So you should be getting like solid like stellar
227080	232940	advice from them. So whatever I'm going to talk is through my lived experience and interacting
232940	238940	with folks like Kendra Albert, Jonathan Penny from Berkman and along with Mary Gray and a whole
238940	245900	host of other folks. So that's my, you know, that's my preface for this. So for me, like
246460	253960	I've been looking at Berkman at the, you know, what is adversarial ML? How does that impact the
253960	261800	experience of citizenship? So one of the things that I, you know, for me, like how I, you know,
261800	267140	how we've been interpreting this topic is moving away from just like an, you know, like the
267140	273220	constraint of like ethics and thinking about power structures, especially in the context of adversarial
273220	279240	machine learning. So turns out that, you know, machine learning is political in a Langdon Winner
279240	285600	sense and security has a lot of like implications, like, you know, if you think of the moral character
285600	292620	of cryptography by Rod Wave. And adversarial machine learning, which is what I work on as a
292620	300840	researcher, sits in the intersection of these two topics. So thinking through who is securing like
300840	306540	the ML systems? What is securing? Who are we securing it against? Who is the attacker?
306540	311460	Are we, is the attacker just like anybody who compromises the confidentiality, integrity,
311460	316980	and availability? So those are kind of the broader questions that I'm looking at at Berkman.
318040	319780	All right, Ellen.
322140	328340	Oh, Stella, thank you for the introduction. And I'm very excited to be part of this amazing panel
328340	341440	so that I will get to hear all of your ideas and vision on this topic. My research focuses on,
341440	346660	working on the implications of machine learning on privacy and security.
346660	352200	Since machine learning or AI technologies are double-edged swords, when they have security
352200	358080	enhancing properties, they usually end up infringing upon our privacy. And starting from
358080	365020	there, I started slowly seeing the link between privacy and fairness and how when our data is
365020	371800	exploited, abused, or collected by third parties, it can lead to fairness concerns at the group
371800	382180	level. And nowadays I'm focusing on understanding how AI systems learn bias or essentially how
382180	389420	systems can learn some form of intelligence or learning ability and how that relates to bias in
389420	395100	natural and artificial intelligence. And when we looked at natural language systems,
395100	400640	for example, word embeddings or today's neural language models that can generate text that is
400640	408280	almost human level, we see that these systems are learning human-like biases from society
408280	414860	very accurately, especially when compared to validated social psychological methods such as
414860	421120	implicit association tests. And then these systems are deployed at the world scale impacting
421120	427420	billions of people every single day with their consequential decisions or biased outputs.
427420	434040	And this ends up turning into this AI bias lifecycle, which is also a feedback cycle
434040	440360	of AI ethics as well. And I'm interested in understanding all of the components in this AI
440360	450320	ethics lifecycle, the role humans play in this lifecycle, how we can discover biases or harmful
450320	456900	side effects of these AI systems, and what we can do to mitigate these harmful effects and prepare
456900	463380	for upcoming problems that might have already been caused by AI systems. And during today's panel,
463380	468660	we will go into the details of all of these single components, starting from the most technical
468660	474940	atomic details of machine learning, going up to probably some philosophical questions at the
474940	486640	high level abstraction. Thank you Aylin. Anima? Yeah, I completely agree with Aylin and Ram.
486640	493320	You know, it's to me exciting times to be an AI researcher, but also, right, I'm filled with
493320	501740	dread that where the world is moving and how fast we are, you know, moving in directions that I find
501740	507920	problematic. And to me, the source of it all comes to the foundations under which this deep learning
507920	513920	revolution is built on, right? So I'd like to emphasize when I'm giving talks on AI about
513920	519120	this trinity, you know, the data, algorithms, and computer infrastructure coming together.
519120	527280	And that's what allowed us to have this fast progress in deep learning. But most of that
527280	532560	progress has been with supervised learning, right? The availability of large-scale label data starting
532560	538560	from the ImageNet data set was the start of this revolution. And in a way, that is quite
538560	544000	straightforward. Once you know that you want to optimize for accuracy and you have the label data,
544000	549840	you can kind of quickly build these systems. But the problem is sometimes we want not just the
549840	555000	accuracy, but like robustness, right? Like what Ram was saying, what are the different changes
555000	561340	beyond the training data? And when our training data is biased, how does this affect on minority
561340	567060	populations that are not represented in training data? And part of it is not just the data imbalance,
567060	572260	but also the algorithmic bias. You know, if you are on Twitter like me, and if you follow the
572260	578480	Twitter storm on the presence of algorithmic bias on deep learning, you know what I'm talking about.
579460	585660	And just to the gist of it, the idea is I think some proponents of deep learning do not want
585660	591360	anything negative being said around deep learning. But I do think these very large-scale models,
591360	597220	highly nonlinear models, are the source of the problem. Because the more nonlinear it gets,
597220	602720	the more brittle they are, the more they amplify the bias, right? There is no free lunch. You know,
602720	608740	you cannot have it all in one mechanism unless we are thoughtful in terms of the objectives we build
608740	615580	in, the kind of data we use, and also the kind of testing we do. And so what I work on is unsupervised
615580	622660	learning. And I think to me, this is the core for us to move away from these highly biased models.
622660	628140	You know, to take a simple example, you know, even before the deep learning revolution,
628140	635100	I was working on topic models, right? Where we wanted to categorize large corpus of documents
635700	641420	and there's no label data. And so now with these unsupervised learning, we are suggesting these
641420	646560	are possible topics, right? And these are the words representing the topics. And then it's up
646560	651460	to the human to interpret and judge what are relevant topics, you know, because there are
651460	657240	weird correlations in text. So there can be lots of topics that are not human interpretable. And so
657240	663780	human in the loop with unsupervised learning, I think is the key to get out of this place where
663780	670340	we are stuck with biased models. And, you know, the current revolution now is unsupervised. Now
670340	675860	we're seeing a lot of progress in self-supervised learning and robustness. Hopefully that'll lead
675860	683760	us the way forward. Thank you. All right. Thank you. And finally, Catherine,
683760	688840	where are you coming from with this? Yeah. Thanks so much. Thanks, Stella,
688840	696000	for the invitation. It's a super pleasure to be here with you all. Yeah. So I guess
696580	703300	where I'm coming from is not as much of a focus on AI, although some of the things that we say,
703300	708700	many of the things we say in our book, Data Feminism, apply, but we sort of enlarge the
708700	716940	scope more broadly to be data writ large. And thinking about also, you know, one of the things
716940	722920	that's happened as this conversation has grown around fairness, accountability, transparency,
722920	730940	bias, ethics, and so on. You know, there's been a lot of pushback on just like AI and data are
730940	738440	great. They're working great for everybody. And our contribution with Data Feminism is to say,
738440	744960	hey, actually, feminism as a kind of body of knowledge, as a kind of intellectual heritage,
744960	750320	which has theorized, philosophized, and also done a lot of writing and sort of action,
750320	757560	like social movements, action around the topic of inequality has a lot to offer us in terms of
757560	763780	conceptual models for how we can think about this current state of affairs. And so that was sort of
763780	771280	the premise of Data Feminism is to say, hey, let's look over here at feminism. What do we,
771280	776760	what can we bring from feminism to understand how we might work with data science in a more
776760	783420	ethical way, and specifically taking structural inequality, which is basically like power
783420	790700	differences between social groups as kind of our main locus of attention. And so when you look at
790700	796200	that, and you look at things like, you know, one of the things I guess that was irritating, I guess
796200	802680	I'll say as we were writing the book, which I'm sure folks here are also irritated by, is that often,
802680	808140	you know, when we have a biased, whatever, algorithm or something like this, like a resume
808140	813800	screener, let's say, that is favoring men over women, you know, the media reports on it with this
813800	819080	thing of like surprise, like, oh my gosh, like, how could this be possible? Like, how could there be
819080	826540	bias in a machine? And we're like, oh my gosh, Data Feminism has so much to say about this.
826540	832740	And so ultimately, like what the book does is it kind of backs us up. It's in a way less in the
832740	839180	details. It's not providing you with a way of like systematically fixing your data set or anything
839180	845280	like that. But it backs us up, like way further back in the processing pipeline to say, well,
845280	849720	like, here's why these things surface. They surface because of hundreds of years of inequality.
849840	856020	They surface because of really flawed models of people like women not being considered fully human
856020	862100	or people of color not being considered fully human. And to really take, you know, if we aspire
862100	868580	to build something that is ethical, we have to look and we have to undertake that with a different
868580	873760	kind of time horizon. And that has to be built into everything that we do. So we're trying to
873760	878880	kind of both say like, hey, let's look back at the root causes. What causes structural inequality?
878880	884580	Like, why do we end up with biased, you know, crime data sets in the first place before we start
884580	891700	trying to like fix them or something? And then how do we use that information to decide which
891700	897780	problems do we work on? Who do we work with? Who do we want to benefit with our work? And who do
897780	903940	we want to just like refuse to work with entirely? Because there are some really bad actors out there.
904180	912320	And so that's... and then we talk a lot about democratizing data, data literacy, and AI as well.
912320	918060	Because that, I think, is really part of the key of how we're gonna get through this. So we're
918060	925940	trying to balance being not only doom and gloom with like, how do we move forward? And we propose
926160	932440	a set of seven principles for how we move forward with a kind of feminist approach to data science.
933360	939820	All right, thank you. So I think a common thread in what all of y'all said is that
939820	946960	there are problems both... problems in machine learning exist both in the pipeline process,
946960	952180	where that is coming from, how it's processed, but also... but that that isn't all of the...
952180	958260	all of the problems. A lot of the time in the media and in conversations, people have this idea
958260	964300	that data is like some kind of objective truth, and that when you come up with biased results or
964300	970140	incorrect results, it's because the analyst messed up somewhere. So, you know, do you think that's
970140	978420	true? Spoiler alert, I'm pretty sure you're all going to say no. But, you know, what would you say
978420	983780	to someone who thinks that way? And like, what is... what are the problems with thinking about
983780	992860	data that way? Whoever wants to kind of get started. I can quickly start with a positive
992860	1001380	note. Data certainly reflects the state of the society, but it does it in a defective way. So
1001700	1008080	we are not getting a perfect representation of society, especially with regards to minorities,
1008080	1014180	because they are underrepresented. But when we have such data sets, at the same time,
1014180	1020180	one advantage is that we can also study the historical injustices these minorities,
1020180	1025180	underrepresented or disadvantaged groups have been going through, because it's reflected in
1025180	1033880	the data set. As I already mentioned, since these data sets are a reflection of society,
1034240	1041980	and then when we start building models on these systems, then we have data that is reflecting
1041980	1047940	these biases. But at the same time, it also reflects some facts about the world, such as
1047940	1054600	statistical biases, inductive biases, or some kind of facts that can also be useful.
1054600	1061420	But that does not justify the fact that there are all these harmful information that gets
1061420	1069560	very accurately embedded in these systems. So this is my double-edged view on this
1070440	1077780	data topic for now. I can follow up with that. I completely agree with Arlene. I think
1078360	1085440	the problem is how the data on the internet is reflecting the society, but only in very
1085440	1091400	balanced ways. And this gets amplified when you want large data sets. The latest
1092060	1100800	one in this aspect is the GPT-3. It's trained on links from Reddit. I'm almost never on Reddit.
1100800	1105920	I've only been a victim of trolling and gun threats on Reddit. I stay away. I don't know
1105920	1110420	any woman in machine learning that I'm aware of who is active on Reddit.
1110420	1116300	So you can see that even among all the imbalance in the internet,
1116300	1121160	the Reddit is even further amplified. So the decision to go for Reddit links and claim
1121160	1127320	that's what we want to build the largest language model on, we can spend tens of millions of dollars
1127840	1134800	on this data set and nothing else, is to me extremely disappointing and disheartening that
1135920	1144780	the VCs, the decision makers, they're always going for the worst of the toxic forms of
1144780	1149840	data that's available on the internet. And I really can't justify this. I don't know how they
1149840	1155000	sleep every night. I don't know how they can say with a straight face that this is the path
1155000	1161640	towards AGI. I mean, this is nonsense, right? So I get frustrated all the time. I know Aileen
1161640	1167860	was trying to paint a more positive picture. I'm just extremely disappointed that I'm sorry for
1167860	1174360	being open about that. Yeah, well, maybe I'll pop in on the negative note.
1176420	1182940	I mean, what happens that I think we're pointing to here is that data are very resource intensive,
1182940	1187360	right? And so it's only certain actors, it's something we say in the book, it's only certain
1187360	1196820	actors in the world that have the resources to collect, store, maintain, utilize, analyze, deploy
1196820	1203660	in some product or whatever, these kinds of data. And so at root, you only have certain actors in
1203660	1209900	the world that are able to actually do that work. And because data are so expensive, we see all
1209900	1214020	these kinds of shortcuts, like exactly like this, where it's like, oh, like, where can we get a big
1214020	1220400	data set? Okay, Reddit, or like, oh, you know, where can we get images to train image, you know,
1220400	1225100	face recognition algorithm? Oh, okay, like celebrity faces or whatever. And just like not
1225100	1232280	connecting the dots of again, like, in what ways like who is not represented in those data sets.
1232280	1238260	And there's like all these multiple points of kind of where inequality kind of enters that
1238260	1243240	pipeline and like not catching it at any of those points, because then the people that are working
1243240	1251320	with the data are often they are the white elite, non-immigrant men, you know, and they
1251320	1256080	are representative of that demographic that are celebrities, right? So like, you know, that you
1256080	1260800	have these kind of multiple stages at which those things don't get caught. But I mean, at the root
1260800	1268080	of it, I often think of the constraint in a lot of ways is, it's financial, like folks are looking
1268080	1281200	for a shortcut. I mean, I'm so glad, you know, Catherine, you brought the issue of like celebrity
1281200	1288420	data set. So in the news event, you know, you've heard about Fox, which is kind of like a cloaking
1288420	1294200	tool, you know, for privacy, it uses like very fundamental techniques and adversarial machine
1294200	1299380	learning so that, you know, people can kind of, you know, like save their privacy from like terrible
1299380	1307740	tools like Clearview AI. And Kashmiri Hill, kind of like the New York Times reporter, tried it out.
1307860	1314000	It was so weird, you know, she got she got some really weird photos. And when she asked the PI
1314000	1319500	of that project, like, hey, why are these photos looking really awry? I thought the whole point is
1319500	1326260	to keep the face same in the cloaked picture. And he said something to the effect, and I'm
1326260	1332800	perhaps going to misquote him, he said, Oh, yeah, women get mustaches and like, you know, men get
1332800	1337400	eyelashes. That's how you kind of like cloak. You know, that's kind of like an outcome. I'm sure
1337400	1344240	there's a whole like, semester of SDS gender studies to unpack there, because they use like
1344240	1350840	celebrity faces. And like, how many transgenders are represented in like celebrity data sets?
1351240	1357160	My first thought was, have you seen the winner of Eurovision 2019? Like, shame on you. Have you
1357160	1364700	seen like RuPaul's Drag Race? Shame on you. But, but yeah, it is. And I'm so sorry, Eileen, that,
1364700	1371180	you know, we're not like raining on your parade. But even though, even though somebody starts with
1371180	1376180	really good intent, and I'm sure like the Fox team did it with very good intent of like, I am going
1376180	1382920	to, you know, forward privacy, I'm going to like, you know, empower people. But like you said, like
1382920	1387300	when they don't connect the dots, or like how Anima said, when they take like shortcuts,
1389040	1394060	yeah, it doesn't lead to less than like harmful outcomes.
1394600	1400000	So just as a quick reminder for the people listening, Fox is the paper that we discussed
1400000	1406100	yesterday at Journal Club. So if you were there for that, awesome. If not, you can read about the
1406100	1410600	paper. You can listen to me and a couple others talk about the paper I was talking about on our
1410600	1414920	YouTube channel, where it should be up by now. Sorry, Eileen, did I cut you off?
1415260	1418600	Oh, I saved my pessimistic comments to the end.
1421420	1427400	But now we have our feedback loop completed. So I'm really curious about all the data practices
1427400	1432440	that are going on out there, or the data impractices or abuse, because as I said,
1432440	1437620	I'm coming from a privacy background, and how people's data is abused, so that they can be
1437800	1444180	de-anonymized, they can be tracked, and essentially, entities that hold power can use this for
1444180	1451700	surveillance or for large scale manipulation. And unfortunately, as citizens, or as human beings
1451700	1457160	living in the world, we don't consent to our data being collected, but we don't have an option to
1457160	1466040	opt out. So all of our data, under the premise of free applications, free products, or just walking
1466040	1473780	on the streets in country X, is being collected by these entities that hold power to build large
1473780	1479520	databases of any kind of information. And in some cases, this data is not being even used,
1479520	1483340	because they think that we don't have a way to use it right now. But in the future, we will
1483340	1488920	figure it out, figure out a way to make use of this data to make one type of consequential
1488920	1495480	decision again. And this is extremely problematic, because the citizen is not informed about these
1495480	1503100	practices and what the long term effects might be on individuals and society. And on the other hand,
1503100	1510400	when we look at other power holders, such as proprietary software, or all companies that are
1510400	1517240	providing these services for the entire world that are free, supposedly free, are making so much
1517240	1523840	money because they are making use of the data that we are providing them. And after that, they keep
1523840	1530620	building stronger machine learning models so that they can keep amplifying these biases in their
1530620	1536900	direction with their targeted ads, or whatever their incentive is. And here there is a serious
1536900	1544420	problem about the incentives of these systems. And both Catherine and Anima mentioned that,
1544420	1548880	for example, we have these objective functions optimizing for accuracy in these systems
1549380	1556660	that actually are helping these power entities. But the objective function could also be how can
1556660	1564660	we use these data sets for social good, as opposed to optimizing for accuracy. So there are so many
1564660	1571600	different faceted and different levels of abstraction of problems involved in this
1573180	1578200	data ecosystem again. And I guess we can keep discussing these in more details,
1578200	1584080	but today the goal is kind of give a glimpse of what is going on in this area.
1584480	1589740	So Aileen, you picked out... guess wrong? Oh sorry, I just have like a quick follow-up for
1589740	1597020	Aileen and Anima. I know that both of you have kind of revealed the shortcomings of ML systems.
1597020	1602440	Anima, you've done extremely awesome work about how GPT-3 is flawed. And Aileen, your groundbreaking
1602440	1610080	work on word embeddings. And I know that both of you have communicated that research to the
1610080	1617320	community in a very approachable fashion. But I don't know about you, Aileen, but I visibly see
1617320	1625340	the kind of like negative, rancorous feedback that you get. And from very well-intentioned
1625340	1630400	research, I mean I don't know well-intentioned, from like people who are in the ML community,
1630400	1636280	why do you think that happens? And Catherine, like from a feminist perspective, why are people from
1636280	1645220	the own educated community like unloading on these researchers? I would love to get your
1645220	1650240	perspectives on this. Sorry, Stella, I didn't mean to like... No, that's fine. That's very close
1650240	1655820	to where I was going to go anyway. I mean, just maybe I can get started on this, right? I mean,
1655820	1660620	to me, I think it's at the root of human psychology. Like no one wants to be the bad guy.
1660720	1665620	No one wants to think of themselves as the bad guys. So they believe, right, they are going
1665620	1670980	towards AGI through this GPT-3 model, for instance, right? I mean, I think they truly believe.
1670980	1679640	So let's assume that. And I think the thing is they really don't want to also hear anything
1679640	1685720	that may harm that, right? So they believe anything that may harm that path is that are
1685720	1690320	their enemies. People who are voicing opinions against it are their enemies. And so that's why
1690320	1697840	we see this vicious pushback on social media platforms like Twitter and, you know, and then
1697840	1702380	they have huge follower counts. So that builds up, right? And all they have to do is
1702960	1709720	do it, just kind of give their followers a sense to go after people who are bringing up these
1709720	1714200	difficult issues. And honestly, because there's also a lot of insecurity, they don't know the
1714200	1720640	answers to this. They don't know a way out. And they worry that their entire existence is in
1720640	1726500	question, which I think that's why, you know, to me, I feel like we are on the right track. So we
1726500	1732880	have to keep pushing this on and increasing awareness, have more allies. I think we will
1732880	1738720	get more support. And it's already been amazing over the last few years, right? I mean, it's
1738720	1744320	happened so quickly, even just the mention of AI bias was not there. It was completely dismissed
1744320	1750180	in machine learning conferences. People working on it were considered not technical, you know,
1750180	1755320	were not respected. There was a lot of gaslighting, you know, because it happens to be women, it
1755320	1762320	happens to be minorities. So there was even more of all that racism and sexism coming into play when
1762320	1768540	such research was reported. But now, you know, especially with the Black Lives Matter movement,
1768540	1774060	that's when these companies finally said, we won't be selling face recognition to law enforcement,
1774060	1778140	right? We've been asking for this, you know, we've signed letters, we've gone to the press,
1778140	1783700	we've been doing this for a while. And it's been led by black women, which is amazing.
1783700	1790040	But, you know, it took a broader global movement to make this happen. And I think we are building
1790040	1795600	this momentum up, because it's not just about AI, it's not just about data ethics. It's a much
1795600	1801260	broader question for humanity. And I think we're, you know, I have to, I remain hopeful.
1801960	1807360	So since Anima brought this up, I wanted to take a moment and ask you guys, do any of you,
1807360	1814600	as yourselves personally, or the companies or institutions that you represent, have any
1814600	1821700	policies about what types of organizations or what types of tasks you will not do ML research
1821700	1826700	or data research for? You know, be it police applications, military applications are two of
1826700	1833300	the ones I know are really popular. I mean, speaking personally, disclaimer, sorry, before
1833300	1838480	I say this, my company has a large number of lawyers who'd like me to tell you that my reviews
1838480	1843820	on the stream represent myself and in no way represent my company at all. I work for a US
1843820	1850260	government contractor. And a lot of the people who work at my company do ML research for the
1850260	1855900	Department of Defense and the US military. And that's a very contentious topic, both within my
1855900	1862640	company and I know within the ML community at large. So, you know, I have been personally trying
1862640	1867640	to decide how to navigate that line myself. But I was wondering if any of you guys had
1868400	1876880	particular stances you've developed? I mean, I'll just say, for my own work,
1876880	1881340	and I'm not speaking about MIT, because obviously MIT is all wound up with all sorts of different
1881340	1889800	things. But at least as a professor, you have autonomy to do what you want. And so at least
1889800	1897060	from my perspective, I've decided to only work with folks who are doing some kind of counter
1897060	1905660	hegemonic work. So activists, journalists, social movements, community based organizations.
1905660	1911520	That's also kind of the, like, that's also in my research interest is to think about, like,
1911520	1917820	how do we think about building data literacy in those kinds of sectors as well. So it's like a
1917820	1923240	really nice fit. And that happens to be where my interests are. But I'm sympathetic to the stance
1923240	1927400	that like, yeah, when you work in a large organization, as many of these places are,
1927400	1930640	you have a lot less autonomy and thinking about, well, like, how do you navigate that as an
1930640	1935940	individual? How do you navigate that as a team? And how do you navigate that at the level of the
1935940	1942560	institution? But I will say, like, I am really energized by all of the organizing and activism
1942560	1947620	that's happened within large companies right now as well. And so I think that is a huge,
1948400	1957900	potentially, is a huge force for change as well. Yeah, I agree. And I want to add what Stella
1957900	1965780	said earlier, right? I am not representing NVIDIA or Celtic for that matter. But I see a lot of
1965780	1971900	support building up from the grassroots in these organizations. And NVIDIA being the platform for
1971900	1977460	AI computing, you know, we are not directly selling these kind of AI applications, right? So it's
1977460	1983100	really our customers and partners. And so there is less of a control in sense of everything the
1983100	1987920	others do, but there's always periodic reviews. And, you know, there have been discussions
1987920	1995260	internally in terms of how to make sure that our platforms are being used for good causes rather
1995260	2002480	than bad ones. But I agree, there is a lot more work to be done. But I'm very energized by activism
2003020	2008260	internally at NVIDIA, and also at Celtic, especially the students, everything, you know,
2008260	2013160	from the recent Black Lives Matter, how it was really the students who have been
2013660	2017440	energizing this movement. And so that gives me a lot of hope.
2018240	2022020	Yeah, and I'm happy. Oh, sorry. Go ahead, Aileen. Please.
2022020	2029040	I was just going to say that I'm not representing anyone except my humble opinion that might not be
2029040	2035680	very well formed. But I try to build my own ethical framework when approaching these issues,
2035680	2041620	since we don't have the AI ethics framework that is widely accepted by anyone. I basically
2041620	2047340	try not to work on any confidential project. I don't try, I haven't worked on any confidential
2047340	2054080	or secret projects. And other than that, I try to get involved in pro bono activities to help
2054080	2060780	companies, entities, research their systems to understand what kind of negative side effects
2060780	2069120	they might have, especially with regards to bias, can we come up with new forms of ways to measure
2069120	2076600	bias, develop methods. And that has been my approach to this issue right now. But if we had
2076760	2086140	a well accepted code of ethics among researchers, I think everyone would benefit from such a framework.
2089720	2097640	I mean, I totally agree. I think one of the things that's kind of like helped me personally,
2097640	2102940	again, you know, boilerplate, boilerplate, I'm only representing myself. When Anima said like
2102940	2110720	reviews that kind of like really resonated with me. There is like, within Microsoft,
2110720	2118000	there's this like committee called the AI Ethics and Research Organization. It's called ETHER.
2118000	2123520	I'm perhaps butchering the acronym. But one of the things that they do, and it is like,
2123520	2129800	you know, folks from MSR, you know, and it's just not the ML folks, but MSR is such a vibrant
2129800	2134460	community. So, you know, you've got like sociologists, you know, folks from like,
2134460	2142080	who think about bias from MSR New York, in Eric Horowitz's organization, kind of like,
2142080	2148320	you know, making sure for critical like ML applications, you know, you need to get ETHER's
2148320	2155820	sign off. So that has been a, you know, a guiding light. The second thing for me is like, you know,
2155820	2162540	it's, I completely agree with Eileen, if there's some like, like a framework, and I, you know,
2162540	2167240	that way you can meet it out to engineering engineers. The thing that's kind of worked for
2167240	2172860	me is that the standards of business conduct at Microsoft is surprisingly good, which is a very,
2172860	2178180	which is a very good thing to say, because they make it, they shoot it like Netflix style videos.
2178240	2183040	So it's not the boring training that you have to kind of go through. I mean, I mean, I've worked
2183040	2188740	in Microsoft, like, in 2013, it was just like the most like boring thing, but you still have to do
2188740	2193120	it. It's like a read through it. But now it's like an actual like Netflix series, they've got the
2193120	2201520	same actors coming in different years. But I mean, long story short, there was a module about how
2202180	2209620	this guy tries to train his model on this, like private customer data, and how he gets into such
2210800	2216340	so much of, you know, like, like a big, like big soup, he can't get out of it, you know,
2216340	2220440	he's trying to demo it and he has to like talk to his manager and managers like, oh my gosh,
2220440	2226160	you're gonna like, you know, lose your job. So they really rain it down on you in a more relatable
2226160	2233980	fashion. So all of this to say that, you know, having this, you know, this committee to kind of
2233980	2241680	like guide for critical like applications, and kind of having this like relatable, you know,
2241680	2250420	ethics discussion has kind of like helped at least me and my team personally a lot,
2250420	2254320	quickly ask Ram, so can they release this to the public? I think
2256080	2261060	immensely from this as well. Yeah, I really hope that they do because it's,
2261640	2267220	it's really captivating, like you're not bored. You know, when you're watching that.
2268020	2272380	So I have a question from the discord that is something you guys have all touched on a little
2272380	2276940	bit, but I think it's worth like directly addressing, which is that so we've been talking
2276940	2282700	about how we were, we all represent large companies, or major universities, you know,
2282700	2290940	institutions that have power and money and resources. And, you know, how, what is it?
2291120	2298020	So what the question asked on discord was, was, you know, if, what's the role that
2298700	2304380	these large institutions is powerful and franchise institutions play? And, you know,
2304380	2308780	if the ability to collect large data sets or train these very large expensive models
2308780	2314440	were somehow democratized, would that improve the situation? Ram, I think we should throw this over
2314440	2321780	to you, because I know you've recently written a paper on this topic. Yeah, that's a really
2322300	2330480	hard question for me. You know, just because, you know, again, from, from a, from a, from a
2330480	2336940	winner sense, like, is machine learning inherently authoritarian, you know, because like, or is it
2336940	2342880	inherently democratic? And, you know, there, there are arguments to be made on both sides,
2342880	2349900	like, you know, like Anima started off with data compute, and the algorithms, that knowledge is
2349900	2360460	kind of centralized in a lot of, you know, in a lot of aspects, right? You, you know,
2360460	2367780	like, $10 million or so with their TPU training pods, which, you know, like a mom and pop shop,
2367780	2373420	like an individual researcher is not going to have. And data, as we all know, like Catherine
2373420	2381180	kind of like outlines in her book beautifully, like how it is really, it's situated with power,
2381180	2386400	almost. So, you know, in a, in a, in a, in a land and winner sense, it really does,
2386400	2391280	you know, one may almost say that, you know, machine learning is authoritarian. On the other
2391280	2398120	hand, actually, one of the other papers that we read was Anima's Born Again Neural Nets paper,
2398120	2405200	where she shows that, you know, a student can actually in certain like conditions outperform
2405200	2411020	the teacher. And, and if you think about the revolution that transfer learning is bringing
2411020	2416740	along, you know, you, a company can put out this like pre-trained model, and then you can just
2416740	2424640	retrain the last layer for your like custom, like applications. So, and, and now with like,
2424640	2430540	you know, systems like fast.ai and like, you know, they're bringing like machine learning,
2430540	2434820	like two programmers, which again is not making it democratic. So it's almost like,
2434820	2440440	you know, you kind of trying to install solar panels in your, in your house, as opposed to
2440440	2447140	depending on like a nuclear power station. So I completely struggle with this question. I keep
2447140	2454000	vacillating back and forth. And unfortunately I do not have like, I'm not, I'm not, I'm not like
2454000	2459600	settled on an answer. And perhaps like other panelists will guide us, will guide us through
2459600	2467640	this cave. Thank you Ram for bringing up the paper. I think to me, I, you know, I again pin my
2467640	2473560	hope back on the algorithms, right? So I think we are just at the infancy in terms of algorithmic
2473560	2479640	development for deep learning. We've been using the very basic algorithms. The question is,
2479640	2485380	you know, where to go next? You know, I mentioned unsupervised learning earlier as a broad topic.
2485380	2490480	I think within that, what we want is more controllability, right? Like for instance,
2490480	2497440	generating text. If you could control what needs to be generated, then you can try, you know,
2497440	2502280	head towards removing these biases, at least in what's generated, even if it's inherent in the
2502280	2506820	model. And in fact, that's what we need in practical applications. You know, we don't want
2506820	2512380	it to write weird poetry. We want it to maybe, right, give us knowledge facts or ultimately
2512380	2518420	program, you know, do program synthesis and so on. And so that's something we've been working on.
2518420	2524380	Same for text, same for images, right? Ultimately the GAN should be able to have controllable
2524380	2530940	generation and it can control various attributes, right? And that way then it's really at the hands
2530940	2537500	of the human on how to use these models. And currently we don't have good frameworks for that.
2537500	2541800	And I think unsupervised learning is the key to enable that because you have to learn what are
2541800	2547500	the hidden variations in data and then how to control and change that and be able to generate
2547500	2554740	new examples. Yeah, so I think that's the direction to go forward. I completely agree
2554740	2559700	with Anima, by the way. She has been mentioning unsupervised learning. I also think that the
2559700	2565800	future is probably in unsupervised, self-supervised learning that is a bit maybe more intelligent,
2565800	2570820	depends on how you define intelligent, because again, this is semantics and a vague term, but
2570820	2576460	it seems like once we get there so that the system is able to figure out this universal
2576460	2583620	learning pattern somehow, and then starts figuring out different kinds of tests by itself,
2583620	2589120	we might be able to somehow incorporate some kind of awareness to the system. And one option would
2589120	2594220	be a human in the loop, for example. And again, Anima mentioned this a few minutes ago, saying
2594220	2601480	that human psychology and awareness plays a very important role here. Like, we are not able to
2601480	2607880	agree on many of these questions, because we are dealing with ethical questions. Everyone has
2607880	2615320	their own understanding, their own answer, and their own desired fairness notion. And we can't
2615320	2620920	satisfy all of these at the same time. However, if we are able to bring some kind of awareness
2621500	2631460	in these systems, then we might be able to change actions and decisions. Because,
2632200	2639980	the evolution of how people are reacting to biases depends on how aware we are of our biases.
2639980	2645860	It's not like people are trying to discriminate other groups. It's mostly caused by favoring the
2645860	2651580	in-group. Like, there are no harms that are explicitly trying to be imposed upon other
2651580	2657880	groups. But this unintentional bias ends up causing all of these harms. So, in an unsupervised way,
2657880	2665240	we can figure out some kind of an efficient learning pattern, where we can also incorporate
2665240	2670700	ethical mechanisms, then we will start getting there. But of course, all of these will require
2671200	2676480	a lot of informed discussions from many different perspectives, and many different disciplines as
2676480	2683400	well, because this is affecting all disciplines, all the sciences, and every social fabric.
2685700	2690940	And yeah, maybe I'll just, I mean, I think the question of, you know, is machine learning
2690940	2696260	authoritarian or not is a super interesting one. But ultimately, the way that we would answer it
2696260	2703060	from a feminist perspective is, in whose hands is it? So, you know, so feminism, basically,
2703060	2710160	what feminism brings is who questions. So, like, who is using it, for whom, to support whom,
2710160	2716740	to oppress whom, and so on. And so, I mean, I think that, for me, is the,
2716740	2720640	those are the questions to be asking when we're thinking about, like, democratizing AI. Like,
2720640	2727720	I think it's lovely to think about making it easier to use models, democratizing access to
2727720	2732080	data and computational power. Those are all great things as, like, a kind of infrastructure
2732080	2739460	of democratization. But on their own, they won't create a democratic kind of usage of AI,
2739460	2745620	because we also need to do the bridge to the groups that could really actually benefit from AI.
2745620	2750180	And for me, like the, again, like I'm saying, like, I'm interested in sort of counter-hegemonic
2750180	2754940	groups and thinking about, like, what are groups that actually hold power accountable? They're
2754940	2761180	groups like lawyers and legal scholars, journalists, community organizations, social movements. Like,
2761180	2768200	these are the folks who need to be kind of trained and brought to the table, if we're
2768200	2772480	actually talking about a comprehensive effort to democratize AI. Because those are the folks that
2772480	2777280	you want to understand, like, what are actually the creative, creative potentials. And there are
2777280	2781060	many, like, some of the journalists that have started to discover the power of machine learning
2781060	2788500	have been doing amazing things, like detecting, using image recognition to detect amber mining
2788500	2792820	from satellite imagery in the Ukraine, for example. Like, you can do this, like, kind of amazing
2792820	2797720	accountability work, but not every person is going to have that idea, because they haven't been
2797720	2806260	exposed to working in this way. So, yeah. So, if we're talking about, you know, AI algorithms
2806260	2811100	being developed by institutions of power, and the fact that, like, who is using algorithms matters a
2811100	2817080	lot, is there a role in, like, adversarial AI? Adversarial AI is usually cast as a bad thing,
2817080	2822400	because machine learning engineers do good, and those pesky evildoers are trying to ruin it. But
2822400	2826460	if machine learning algorithms are being used for bad, you know, does that mean that adversarial AI
2826460	2834060	techniques can be used for good? Oh, I love that. Totally. I totally think so, yes, because, like,
2834060	2841220	there are so many reasons why groups or individuals would need to be protecting themselves, like,
2841220	2846580	from, you know, we were just talking about Reddit, for example, you know, like, there's so many reasons
2846580	2855420	why, you know, like, an act of, kind of, an adversarial act is not necessarily bad. Like,
2855420	2864860	for example, I have a friend who, as a way of, kind of, fooling the just rampant data capture
2864860	2871240	that's done by promotional ads and things in the browser, and so, like, the browser is always sending
2871240	2879400	the IP address and whatnot, and what it does is, instead of, like, sending what it clicks on,
2879400	2886360	the browser clicks on, it actually just sends every link on the page. So it's, like,
2886360	2892060	hacking it through obfuscation, you know, and so it's, like, I think that's a brilliant technique
2892060	2898480	for then preserving privacy. So you're obfuscating what the user is actually doing, and screwing
2898480	2903720	with the ad systems or whatever. And so I think, especially in situations where power is not equal,
2903720	2910560	which is every situation right now, like, we also need, like, good, like, white hat adversarial ML,
2910560	2920400	basically. I love this, Catherine. I got so excited for the ad blocker research. I mean,
2921000	2925140	right now, in the field of adversarial machine learning, the use of adversarial machine learning
2925140	2931160	to enhance the experience of citizenship is still nascent, but, you know, there's fantastic work
2931160	2936800	that's happening. I'm so excited for this. Bogdan Kulinich, I'm sorry if I'm pronouncing his last
2936800	2942060	name incorrectly, developed this paper called POTS, which is Protective Optimization Techniques.
2942060	2947500	We're into, kind of, like, empower. So what they do is, it's really cool, it turns out that,
2947500	2954820	obviously, credit card scoring is biased against minority population, and they show, like, how
2954820	2960520	you can poison the system tactically, like, do it, literally launch, like, a poisoning attack
2960520	2968160	to, kind of, like, you know, make it better for the target population. There's really another
2968160	2976680	cool work from some researchers who found that Facebook's ads platform is micro-targeting,
2976680	2985600	and they, again, use adversarial ML to show, like, you know, faults in that. So that opens up, like,
2985600	2991650	exactly, like, how Catherine says, white hat adversarial examples. Unfortunately,
2992440	2999560	you know, it's not the dominant thinking. In fact, DARPA, when it released its
3000360	3005140	guard challenge, which is, you know, securing against, like, AI challenge,
3005140	3011160	they said, we need to protect ML system from attackers. And they do not define who attackers
3011160	3018920	are. So that's... And I completely agree with Ram and Catherine. And I think there's also,
3018920	3023900	right, because we don't know who are the attackers, I feel like there's also people
3023900	3029240	are going in all kinds of weird directions with adversarial AI research, right? Because they don't
3029240	3033420	know what kind of attack to, you know, and they're like, okay, maybe I'll do this norm bound or some
3033420	3039400	other. And then it's like, okay, why does this matter? We don't know, right? So I think perceptual
3039400	3044960	attacks for images are much more meaningful. I mean, you know, if those that kind of preserve the
3044960	3052240	for humans, the perception, but, you know, change the statistics of data, right. And to me, I think
3052240	3060200	the next step is to really have the regulatory agencies, you know, take on these tests. I mean,
3060200	3064520	we have a long way to go. Because if you look at the NIST, which is the standards data sets for
3064520	3070940	face recognition, that itself is heavily biased against minorities. But ultimately, the hope is
3070940	3076360	regulatory agencies in each domain will be using these as tests, you know, especially for autonomous
3076360	3082580	driving, this is a must, right? I mean, given the long tail, the only way to do is also have very
3082580	3088940	heavy testing would need a lot of compute, but only then certify that this is roadworthy and
3088940	3091760	otherwise be very careful about that.
3092580	3098300	And Anima, to my understanding, NIST has been tasked to develop these AI standards. So they
3098300	3103820	are especially working on those given the example with the face recognition or computer vision data
3103820	3110180	sets they had as well. So that was a great point that you identified. And again, there are all of
3110180	3116960	these privacy enhancing or fairness enhancing adversarial applications of machine learning,
3116960	3122700	but they might not be very common right now. But the most simplistic version we can think of is
3122700	3129380	synthetically generating data to oversample minority populations so that we can improve
3129380	3135700	their representation. Because at least in real world statistics, it's well known that oversampling
3135700	3143700	minority groups is a helpful approach. But at the same time, again, this calls for the need of some
3143700	3150580	standardization in this process. And again, this ties back into the question of is only data bias
3150580	3156100	or algorithms also biased? Why are we creating these algorithms? What are the goals of these
3156100	3164980	algorithms? Who are the bias power holders implementing these algorithms? And again,
3164980	3170480	so it really matters. And this might bring us to the Kant question here. Should we have this
3170480	3176900	ontological approach? Or should we go with utilitarianism? And so on. So there are many
3176900	3181820	interesting questions in this direction as well that we are starting to just scratch the surface
3181820	3187880	nowadays, because it's becoming a very hot topic. So why are we even building these systems? Why are
3187880	3194380	they going to empower at the end? Absolutely, I just want to quickly add to Alan's point on the
3194380	3200460	oversampling for minorities. I think, yeah, if you don't have more data, I think that's a quick fix
3200460	3205520	right? That's the starting point. But that itself can also create problems because GANs have mode
3205520	3211980	collapse. So you just have very less diversity among, say, the minority class, right? For instance,
3211980	3217880	in fact, it was shown when the images that are generated based on the celeb data set,
3217880	3223420	the celebrity data set, black women tended to have all the makeup on, right? So it was and
3223420	3229260	with these face recognition systems, when black women didn't have makeup, it was even worse.
3229260	3235240	You know, so then there is kind of further bias even within the class because of the lack of
3235240	3240480	diversity and variation. And that's why I think we need to disentangle all these factors and do
3240480	3245960	unsupervised learning and learn, is there also enough diversity within the class in terms of
3245960	3253740	these variations? Yeah, go ahead Stella, sorry. So that sort of feeds into something that someone
3253740	3259360	in the Slack asked about as well. And it's been in the academic discussion of AI bias a lot
3259360	3268520	recently, which is how models can exacerbate or create biases out of imperfections. Even things
3268520	3273820	that aren't normally thought of as imperfections, just the choice of your loss function implicitly
3273820	3280520	decides how you're going to weigh different samples. And this is crude, but if 90% of the
3280520	3285720	people in a data set are white and 10% of the people in the data set are black, then an algorithm
3285720	3291120	that gets a little gain in performance at the cost of a little loss on a white person, at the cost of
3291260	3297360	a little loss on a black person, is going to perform better on average, if it just mostly
3297360	3305900	pays attention to white people. So, you know, lost the rest of my train of thought.
3307220	3312340	How, so it's not, it's not just that when we talk about bias in algorithms, we're not just
3312340	3317320	talking about the data coming in, but kind of algorithms can create their own sorts of little
3317320	3324340	biases. Like, what does it look like to combat that? And kind of how does that happen? You know,
3324340	3327680	from the perspective of a data scientist who's looking at something, it's like, you know, how do
3327680	3333860	you know that's happening? And how do you combat that? Just a quick anecdote, because I don't have
3333860	3340940	an answer to your question. A quick anecdote was one of the resume screening algorithms,
3340940	3346500	it wasn't Amazon, but it was like a different company, when they reverse engineered it to see,
3346500	3351180	like, why was it preferring men over women? One of the factors that they found, one of the strongest
3351180	3356320	predictors was like, the name. So if somebody was named Jared, in particular, it was a very
3356320	3362400	strong predictor of success, as well as if they were a lacrosse player or not on their resume.
3363020	3368860	And so it's like, just a sort of very interesting thing, where, in a way, what happens there,
3368860	3372380	like, they're not even they're considering, well, I don't know, I don't really know the model. But
3372380	3377900	like, I don't think they were directly considering race or gender in that case. But if you look at
3377900	3387140	those things, like Jared is a very white, like kind of Anglo name. Lacrosse is a very elite,
3387140	3390400	white, sort of private school sport in the United States, despite the fact that it was
3390400	3396140	invented by Native Americans, actually. And so like, those things sort of, they seep in,
3396140	3401520	and then they become these factors, and they become predictive factors. But like, behind those
3401520	3406980	is always like, is race and gender and all of those things are like, kind of still underlying.
3408640	3414920	Yeah, these systems learn the associations, and you were both asking what kind of metrics we
3414920	3419680	should, for example, develop to understand how to improve these systems. So I was wondering,
3419680	3424700	maybe this exists in literature, but do we have, for example, diversity dispersion measurement
3424700	3430700	metrics to understand how diverse a group is, so that we can try to improve diversity within one
3430700	3436940	social group? And then the other thing is, Stella, you mentioned objective functions,
3436940	3442640	or machine learning algorithms that are trying to optimize based on the data representation,
3442640	3448400	where people are categorized into groups, racial groups, or gender. And it's not even clear how
3448400	3454220	ethical this approach is. Should people be represented in categories? Is this the correct proxy
3454220	3462440	to a person's identity or being? And how should we deal with system design when we are developing
3462440	3468360	new machine learning algorithms? Because each algorithm depends on these data types, the way
3468360	3473980	they are represented, numeric values, ordinal, or whatever. And should we come up with more creative
3473980	3481960	ways to design new algorithms for machine learning to encounter, to counter balance,
3481960	3489280	these problems? And Stella, I mean, I want to rewrite this question back to Catherine, because
3489840	3496400	when I was reading a chapter from her book, Catherine, you spoke about concepts which
3496400	3503040	maintain power, like fairness, bias, and ethics. And you give this amazing framework about concepts
3503040	3509880	of challenge power. I didn't even think about that. Is that something you want to talk about?
3509880	3514780	I feel it's a really cool framework for engineers to think beyond the fact star or fact framework
3514780	3525780	that we're used to. Sure. Yeah. I think that's in chapter one or two. I can't remember. But
3526400	3530360	yeah, I mean, so one of the things I think that Lauren and I were frustrated by
3530360	3536260	is that we just felt like fairness, and this is not, I guess I should say a little bit of a
3536260	3540200	disclaimer, because I actually think the work that's going on in the fact star community is
3540200	3544580	super exciting. And so like, this is not to say that, like, no one should do that work, because
3544580	3551120	I am really happy that work is being done. But that's just some of the organizing framing concepts
3551120	3555800	for what we're talking about aren't, they don't go far enough. And so it's like, instead of fairness,
3555800	3561820	we propose justice, or no, instead of ethics, we propose justice, like, we're really thinking
3561820	3569660	about what does it mean to make a just system, which means repairing past harms, like harm has
3569660	3575000	been done in the world, we come to the world in the bodies, in which harm has been inflicted at
3575000	3580280	the group level, for some of us for centuries, you know, and so like, how do we build systems that
3580280	3588000	not only kind of treat us all equally as if we're all the same, but actually take that harm into
3588000	3594200	account. And so we propose other kinds of like, justice, concepts like...
3595380	3597420	My favorite one is transgression.
3598060	3601920	That's what I was gonna say. Yeah, like, kind of like, why settle for accountability,
3601920	3606500	because often accountability is after the fact, you know, like, why can't we build
3606500	3612120	systems that liberate us? Why can't we build systems that heal us? Like, I feel like we,
3612820	3617900	we have the power to imagine just much more expansively. So like, it doesn't always have
3617900	3622820	to be after the fact that we're trying to, like, you know, repair things or whatever, we can really,
3622820	3628500	we can really aspire to do much more. So co-liberation means, it's this idea that
3628500	3634880	none of us are free until all of us are free. So we work with the most marginalized groups
3634880	3640880	and center, like, this is a case where we would like, build a data set around whoever is most
3640880	3645100	marginalized in whatever the application area is, and we would work with them directly, and they
3645100	3649920	would be part of the process. And whatever is built, the system that is built would like,
3649920	3655800	actively serve their needs in a really culturally grounded way, you know. And so I think that's
3656440	3659600	totally feasible. It's just not the way that things are done right now.
3660440	3666360	I fully agree. I think these are great points, right? Ultimately, we have to kind of rework the
3666360	3672320	whole framework. But I think to begin with, even, right, like, the way these systems are now just
3672320	3678520	kind of put out with almost no testing, right? And, and then only kind of somebody like, kind
3678520	3683080	of, right, brings up the problem, usually a minority researcher, and then the company is
3683080	3689040	doing everything to push back and use all the resources to discredit the researcher, I think,
3689040	3694380	at least we should change that conversation. And I think that can happen, especially, you know,
3694380	3701200	with the viewers of this panel, we need more hackers into these AI systems, go hack every AI
3701200	3709800	system out there, reveal all the vulnerabilities, take it down, rebuild a new one.
3709800	3714440	Yeah, you can have like a brigade of like algorithmic bias assessors.
3714440	3718300	I mean, good trolls, good trolls.
3719840	3724520	To kind of move this towards, you know, advice or takeaways from the audience, because
3724520	3729120	this has been a question popping up in the chat a lot, you know, so you guys, you know,
3729120	3733620	you guys are talking about how there's definitely steps that people can take to make progress in the
3733620	3737740	way that they design their algorithms to be more inclusive and stuff like that. And one thing that
3737740	3743320	kind of stood out to me is being implicitly present in a lot of what you guys said, but I don't think
3743320	3747380	anyone said this explicitly, is that kind of the default view of AI is it's supposed to be
3747380	3752560	descriptive, you take a description of the world as it was, and you put that through an algorithm,
3752560	3757820	and the role of the algorithm is to make the future look like the past. And so in a very real sense,
3757820	3763360	AI is intrinsically regressive. But at the same time, nobody, there's nobody saying you have to
3763360	3768900	do that, you know, you can decide that you want to choose to value certain things more than others,
3768900	3773660	and you want to kind of bring about certain impacts, and you can proscriptively put those
3773660	3781620	into your algorithms. Personally, I have tried to advocate for this, and largely speaking,
3781620	3785780	failed, because people are scandalized by the idea that you might tell the data how it's supposed to
3785780	3793880	look. But you know, so if you're a, if you're a data scientist, or a machine learning research
3794460	3801400	engineer, listening to this chat, you know, can, can, how can they proscriptively design their,
3801400	3807680	their algorithms for equity, for justice, for liberation, for whatever all these very nice
3807680	3813980	words are, that are better than bias, or fairness, I suppose. You know, what does that, what does that
3813980	3823440	look like to design an algorithm in that fashion? I mean, I'll start, and then hopefully others can
3823440	3828840	build. But like, I mean, I think what it would look like is to do, to really think about where,
3828840	3834160	where what your work is doing is being applied, and on whom, right? And like, who you're making
3834160	3839580	it for, like, who's your boss, and then like, who are ultimately like the end users, or who
3839580	3843440	is kind of impacted, who are the decisions being made about because of the system,
3843440	3852320	and to do basically like a power analysis of that system. And then think about what is,
3852320	3858740	that application area, who has been systematically disenfranchised, and then work with the groups
3858740	3863600	who are working towards a better future in that domain to co-create, like what the metrics should
3863600	3868500	be for the future. You know, I mean, I think that we're not asking, I don't, I don't think it's like
3868500	3872620	appropriate that like, like a machine learning engineer would like invent the metrics out of
3872620	3876420	their head or something. You know what I mean? Like in a domain space, they may not know very
3876420	3881720	much about, but I think it is feasible that like a company could work really well with a group of
3881720	3886860	folks, say, working on prison abolition or something like that, and could craft like here
3886860	3891680	are the metrics for what the world that we would like to see. And this is like the kind of outcomes
3891680	3896900	that we want to see. And so that's, that would be my kind of generalized recommendation. But that
3896900	3903120	said, and this is like one of the major tensions of this space, is that often people are building
3903120	3907820	tools that are more generic, right? Like you're looking for generalizability, you're looking for
3907820	3913060	abstraction and like that could be applied in a wide variety of different application domains.
3913060	3918420	And that I think is a tension. I actually think that is like a major tension for AI doing good
3918420	3923920	in the world and being culturally grounded and informed. I don't know how to resolve that. And
3923920	3929000	I would love to hear what like the computer scientists and researchers have to say about
3929000	3935600	if they see that tension as well or not. I fully agree. And I think to me that
3935600	3940940	foundations of that goes back to the education, right? So computer science and engineering has
3940940	3946640	had the military background, you know, it started as a way to get into the military and that's
3946640	3951920	there today. Like, you know, we are told to keep our emotions aside. In fact, that was one of a lot
3951920	3957120	of gaslighting on Twitter is about, oh, why are we emotionally charged about machine learning? Just
3957120	3962940	focus on the math, you know? And so that's something that's pervasive in the educational
3962940	3968040	system and then beyond in the companies, right? Like, you don't worry about this. You do, you know,
3968040	3974520	you derive math. You just, you know, program. So you're asked to become a robot without any emotions.
3974520	3981120	And I think that's just really dangerous, right? Because it's not every engineer who's the problem.
3981120	3987020	I mean, there's absolutely a decision maker who is probably, right, kind of, you know, may or may
3987020	3992860	not understand the implications, but it's kind of directing all this and leading to the pervasive
3992860	3998740	bias and injustice. I think we have to start with education. That's where I would pin my hopes.
3998740	4004800	For instance, at Caltech, we have a course on ethics for AI. I think every university is now
4004800	4010180	doing that. Ultimately more humanistic education. You know, we are kind of, you know, I went to
4010180	4015700	technical school. Now I'm in a technical school, right? The role of humanities there needs to be
4015700	4019640	even further emphasized. You know, it's much better than when I went to school,
4019640	4024200	but it's still a long way to go to integrate humanities and engineering together.
4024600	4030040	I completely agree with Anima. I'm also teaching bias in AI. I think it needs to be part of the
4030040	4036040	curriculum, not for computer scientists only, but also engineers, humanities, social scientists,
4036040	4041180	probably as well, because data scientists or essentially anyone that is interested in
4041180	4048480	computing is using these tools. So they need to understand that all of these tools exist within a
4048480	4053780	context. They don't just exist in an abstract, generalizable way, as Catherine mentioned.
4053780	4059220	It's correct that these tools or algorithms were developed so that they can be applied
4059220	4064640	to hypothetical data sets so that the theory or foundations are presented. But then when
4064640	4071920	we take social data and apply these tools for machine learning tests in the social domain,
4071920	4076940	then we are dealing with all kinds of human complications. So people need to always keep
4076940	4082200	in mind that these systems, yes, they might look objective and neutral, but they are not in the
4082200	4087920	social context because they exist with all the problems or side effects that human decision
4087920	4094220	making brings into the game. But moreover, AI also brings its own kinds of problems, such as
4094220	4100560	new types of biases we are not even aware of yet. And these keep getting amplified in society. So
4100560	4108140	before we start introducing more of those, we need to understand what we are bringing to the world
4108140	4115160	and what we should be careful about, because all of these systems are deployed right now without
4115160	4122220	understanding what side effects they might have or what the implications are on society,
4122220	4127980	economy, democracy, and so on. And now we would need to deal with all of these consequences
4127980	4137960	on top of all the other things. It seems like a major takeaway you guys all advocate for is to
4137960	4144120	kind of broaden the scope of who gets included in the conversation about AI design, about AI data
4144120	4149800	collection, you know, bringing in activists in the relevant spaces, bringing in other stakeholders.
4150880	4156440	And, you know, one thing that I would like to add to that is that the notion of stakeholder
4156440	4164940	is really politically fraught. In industry context, stakeholders are generally conceived of as
4165480	4172780	like, you know, so I suppose I'm at a ML company developing an algorithm that I'm going to sell to
4172780	4177620	another company to use for financial profits. You know, the stakeholders in this process are
4177620	4184780	generally conceived as being me and my team, my superiors at my company, the people I'm selling
4184780	4190380	the algorithm to, and the stockholders of their company. And then I think that misses a really
4190380	4195140	crucial point, which is the stakeholders are the people whose data is going to be analyzed.
4195600	4201400	And their voice is very, very frequently just completely left out of any discussion about how
4201400	4205780	things are designed, why things are designed, and where they come from. And I mean, to an extent,
4206240	4211420	partially by design, because a lot of companies exist out of the fact, because of the fact that
4211420	4216520	they can make money analyzing user data, like that's their sole reason, that's their reason
4216520	4223280	for existing. There are lots of these things. But broadening, I think broadening the notion of a
4223280	4227800	stakeholder, in particular, viewing anyone whose data is going to be processed by an AI as a
4227800	4236660	stakeholder is really important. Yeah, I just I totally agree with that. We have a chapter in
4236660	4241300	our book, which is about the principle embrace pluralism, which is basically this idea that,
4241300	4246920	you know, we create more complete knowledge, like the path to objectivity is not through
4247460	4252540	individual genius, but it's through bringing multiple different perspectives to the table.
4252820	4256060	But I agree with you, I think like the it's important to think about like, because we can
4256060	4261000	say, oh, it was participatory, like that company with all the money and the other company with all
4261000	4267980	the money. So I do think and this is because participation, I mean, there's this whole
4268720	4273960	world, you know, participatory design and participatory methods. And so like, there's
4273960	4278280	even folks have talked about participation washing, where, you know, you're like, oh,
4278280	4281980	yeah, we did a, you know, this also happens in urban planning, like, oh, yeah, we did a public
4281980	4289600	meeting and it was at like 10pm and there was no childcare. So like, what's the quality and
4289600	4294700	the character of the participation? Like who shows up? Who are these stakeholders? And who
4294700	4299680	are the stakeholders that are not convenient to have? And how do you make folks listen to them?
4299680	4304120	I mean, I think that's a big, that's a big question when ultimately, like the goal is to
4304120	4312390	optimize profit. That to me is a conundrum. I think how do you make people listen, I think
4312390	4319010	is a really interesting question. You know, a lot of people, I imagine a lot of people feel like
4319010	4325350	they would like to do something good, you know, they would like to be more just and do their
4325350	4331550	design the right way. But you know, how do you convince your boss or your boss's boss who
4331550	4336790	controls the money and the funding, or the money and the decision making power, and who's maybe
4336790	4341610	more interested in what's profitable? You know, how do you do any of you guys have advice about
4341610	4348410	how to approach that conversation? Or how to how to advocate for justice in AI design,
4348410	4357330	in context where that input is not necessarily the primary concern of the people who pull the strings?
4358530	4365050	I'm interested in that as a research question, if we can figure out a way to simulate AI systems to
4365050	4371610	show their long term gains and losses for all the parties that are involved,
4371610	4380950	I think we would be able to understand how this affects economy in very from various perspectives.
4380950	4384930	But again, this is a hypothetical research question. I'm sure there's already some great
4384930	4393670	research on this yet. But maybe long term goals might be able to convince people that are profit
4393670	4402230	driven as well. I mean, especially if there's concern, I really like the idea a lot, especially
4402230	4409310	if there's concern, I mean, as there's this growing sort of industry of sort of impact assessment,
4409310	4415230	or algorithmic bias assessment, and things like this, this is like, bringing folks in and doing
4415350	4421190	a more participatory design process is a preventative for what could happen later,
4421190	4426390	and so I think maybe that's the, like, if the case needs to be economic rather than moral,
4426390	4429990	I mean, you can make the moral case, but probably what that listens to is the economic case, right?
4430010	4435450	But if the case needs to be economic, I think that, you know, it would be lovely to have some
4435450	4440490	quantifiable research that you could cite to say that like, yeah, there's this like,
4440490	4446230	very strong economic case that this will reduce the number of lawsuits later,
4446230	4450750	that you'll be fully compliant with all federal regulations, but you know what I mean? Like this,
4450750	4457750	these kinds of things, I think that that might be the path for convincing the employers,
4457750	4460930	and like, ultimately, like, you've invested in prevention,
4460930	4464410	so that you don't have to clean up a big mess after the fact.
4467970	4472830	That's why academia and the independent researchers have such an important role,
4472830	4477630	right? Because they can talk openly about these issues, like, you know, what happened with the
4477630	4482810	studies, right? Even though internally, there was a lot of conversation at Amazon,
4482810	4489250	when I was there, you know, having that external light into it, and then the wider public part
4489250	4494170	spreading in it, I think is critical. Because, you know, like being in these companies, it's
4494170	4499450	just the whole system is so short term driven, right? The Wall Street expects quarterly results,
4499450	4505030	that's not delivered, the company won't even exist, right? So the whole system is not built
4505030	4510070	for the long term implications. And I think that's what creates these challenges. And so
4510070	4516130	somehow, maybe then having more participation from the public, right, can show that also,
4516130	4521690	you know, bad AI can lead to bad PR and loss of profits, even in the short term,
4521690	4526090	I think that activism and awareness is critical to change this.
4528630	4530210	Ram, you want to jump in?
4530210	4539730	Yeah, I was just framing about what Anima said. But the way I think about this is, at least again,
4539730	4546230	through lived experience, I can say that what has changed for me is that if you think of like
4546230	4553090	inclusion, as after the fact, that's kind of like really weird. One of the things that I'm very
4553090	4559130	lucky is that inclusion is part of my core priorities that I set. Like, it's just not for me,
4559130	4564010	it's for everybody in the company. Like, you've got to set that, what is your inclusion priority?
4564010	4567710	For me, it might be like, you know, I'm going to take care of Pride Month, like, you know,
4567710	4572130	talks or, you know, whatever things, right? So obviously, it doesn't solve all problems.
4572130	4578890	So first of all, I fundamentally believe that if your organization is questioning
4578890	4584370	why you're doing something ethical, that itself is a big red flag for me.
4584990	4591330	Right, like, and for the business case, I do agree, like, you know, there's this, like, you know,
4591330	4596190	how Catherine said, there could be some financial motivation. I saw like an HPR article about like,
4596190	4602810	how diversity leads to more surplus, because you're selling your product to more people.
4603030	4609350	That could be like an other way. The PR angle, like, I mean, everybody is afraid of like bad
4609350	4614350	marketing. I'd never actually thought about that. So thanks for bringing that up. But for me,
4614350	4619750	intrinsically, you know, I have never been told you can't publish this. I mean, but again,
4619750	4624650	it's a very different setting, like MSR. And like, you know, product setting is very different.
4625150	4633670	But yeah, I would be very uncomfortable if I have to make a case to my manager as to why I'm doing
4633670	4639490	something ethical in the first place. But like Anima pointed out, like, business, I've only
4639490	4645010	worked in one organization for like, my working life. I do not know how others operate. I know
4645010	4650430	some of the CEOs out there question why they should use pronouns. So or like, why they should
4650430	4656150	like, you know, why they do not like fundamentally care about democracy and just want to dismantle
4656150	4660790	it. So I don't know, I do not know how things work there from my lived experience. I don't know if
4660790	4667960	this might actually be helpful for others. All right, so it seems like the kind of two
4667960	4672600	major points you guys have raised is that you can view it as a risk mitigation factor.
4673600	4677600	Because it will stop things from it can stop things from going wrong in the future. And it's
4677600	4683240	in some sense insurance on that. And it's also a PR thing, which is partially a risk mitigation
4683240	4689360	factor. You really don't want the New York Times running an article saying so and so's company
4689360	4694720	develops algorithms systematically discriminates against women in hiring. Oh, I actually saw
4694980	4700840	a headline. Yeah. Oh, sorry. Yeah. I actually saw an email. I only saw it through your tweet
4700840	4715160	about gentrify or genderify. Yeah, another thing after all the right outcry got withdrawn,
4715160	4721160	right? The fact that it even got launched is just appalling the kind of lack of awareness
4721760	4727240	these companies. Yeah, it was shocking. You know, there was another company.
4729100	4735060	I don't know who alerted me to it. But essentially, like their solution for more representation
4735060	4743900	on stock photos was to kind of like artificially generate like photos using GANs. So you will not
4743900	4749980	actually pay money to hire a black model. Oh, let's just use StyleGAN and like to kind of like,
4749980	4754960	like, how do you get through? Like, who's funding this and goes back to an email like
4754960	4761240	initial comment about VCs getting excited. So that I feel is a bigger like thing. I would be
4761240	4766580	very less less worried about managers and more worried about like plutocratic like Silicon
4766580	4777100	Valley folks to then like actually managers. So to kind of contextualize this for people
4777100	4784100	in the audience who aren't aware, Genderify is a product that existed for all of about four hours.
4786900	4792120	What their business model was, was that you can send them user data,
4792120	4796960	you can send them photos, you can send them people's names, and they will
4800440	4804040	they say they will determine the gender of the people that you
4804940	4810460	in that you sell to or the people that are in your data set with an aim of doing better
4810460	4816220	predictive marketing and stuff based on the... I'm having trouble coming up with a word other
4816220	4825380	than assumed gender of the of the people in your data set. And this was this was funded by some VCs
4825380	4832180	and it went online and everybody including lots of people in the ML space who don't really care
4832180	4837000	very much, who don't talk very much publicly about about ML bias lost their shit because it
4837000	4846100	was a completely absurd idea. And just to give an example, right, the word professor was only 1.9%
4846100	4852580	females. So that's the assigned percentage for me in terms of gender. So I think that that was
4852580	4857300	the problematic thing. Like how could you just type any name, like any word, not just name,
4857300	4864140	any word in the vocabulary and assign a gender to that? I mean, and only binary genders, right?
4864140	4868840	It's just appalling the whole concept. Yeah, my friend was saying the only time that's allowed
4868840	4874180	it's in French, any other language, like, you know, because, you know, nouns have gender.
4874260	4881280	Any other time table doesn't have like a male or female. Yes. Anyway. Yeah, but I'll know that like
4881280	4886600	there are like this product is insane. But there are other products like there's all these gender
4886600	4893760	detection libraries out there, both for faces. So like facial detection, which the researcher
4893760	4899280	Oz Keys has written this amazing paper called Misrecognition Machines, which is about how
4899280	4906300	harmful face like gender detection of faces is for trans folks. But then there's also all sorts
4906300	4914400	of libraries that infer gender from name using various kinds of like name databases. And like,
4914400	4919080	maybe I'll be like, sort of a naysayer here, because like, in general, I feel like these
4919080	4925500	applications are very well, again, it depends on the application domain. So like, if you're doing
4925500	4932880	this for predictive marketing, or like Oz talks about how they're doing it for bathroom access,
4932880	4937160	which is like, disgusting, you know, like, are you the right gender to get into the bathroom or
4937160	4942840	whatever? Those are super harmful and awful. And like those kind of applications shouldn't exist.
4942840	4948240	At the same time, these gender detection libraries are helpful for doing like,
4949320	4953580	bibliometric studies, where you're looking across like a huge corpus of say, like,
4953580	4957420	I don't know, like several million authors and trying to see like, what's the
4957960	4963160	over representation of men in that data set or something like that, you know, and so like,
4963160	4967300	that that's why like, again, I think it's hard to just say categorically,
4967300	4972840	hmm, something is bad or good. Although I do have certain things that I do think that way about,
4972840	4980360	but like, that could enable really interesting research that monitors back on the patriarchy,
4980360	4989100	for example. And so like, again, it's like, who's using it? For what? Yeah, like, that's what
4989100	4994140	matters. And just to give an example, we have used the gender detector, but that was to learn
4994140	4999920	about the MeToo conversation. So we, you know, collect lots of MeToo data from Twitter,
4999920	5007260	men talking about it, how are women talking about it? How are Democrats, Republicans? So yeah,
5007260	5012780	so that kind of, so you know, and this is in collaboration with Professor in Sociology,
5012780	5019020	Mike Alvarez. So you know, like, the context certainly matters. We can't just ban all
5019020	5023800	applications of gender detection, but really contextualize and look at whether it's for
5024140	5030240	harm or for good. Yeah. And in medical applications, we especially need very accurate
5030240	5036880	biased representations, because we know that for certain disease, the diagnosis and prognosis
5036880	5044020	depends on ethnic factors, gender, and socioeconomic background as well. Of course,
5044020	5049820	there are biases representing injustices in these medical data sets. But at the same time,
5049820	5057040	there are biases that we need to learn from for more accurate medical treatment. And I guess
5057040	5063660	that's why, for example, in the medical domain, machine decision making is becoming much more
5063660	5070260	accurate than human decision making. But again, yet we need to deal with all the other disadvantages
5070260	5076600	that come with this. So again, I guess everyone keeps mentioning this, it really depends on the
5076600	5084540	context where we are using these things. And name-based gender detection is a kind of an
5084540	5090900	outrageous example. But then there is a supercharged version of this in country X,
5090900	5096300	where machine learning is used to predict the criminality of a person from their face,
5096820	5103640	or actually give people social scores so that they can be sent to workers camps for their entire
5103640	5112380	lives, because the machine said so. So there's a spectrum of these effects that we can discuss.
5112380	5118500	And then what foundations should we be building on, given these examples that already exist in
5118500	5126120	the world? Well, thanks for sharing that. I was just, I completely dismissed it,
5126120	5132660	Catherine. And thank you, Anima and Eileen for, you know, correcting this. Thank you.
5133180	5137140	I didn't mean to correct anything, by the way. Oh no, it was like, you know,
5137140	5142280	I learned something. So thanks for telling me about this. Yes. I appreciate it.
5142280	5148780	What did you learn, Ram? Well, I just completely dismissed like gender detection. I was like,
5148780	5155140	oh, this is like, has literally no use. If I want to ask somebody's gender, I will just ask them if
5155140	5162360	it's part of my business. I'll say it's none of my business. I'm not going to ask. But when,
5162360	5167120	I was like, okay, that kind of makes sense. And when Eileen spoke about the medical contacts,
5167120	5172840	that makes sense. So yeah, this was, like I said, I already learned something from this panel. And
5172840	5177320	within like two more minutes for the panel to get over, I learned something. I learned a lot of
5177320	5186920	things. Yeah, the example that Eileen brought up of criminality prediction is unfortunately
5187560	5193980	running rampant in the AI literature right now. I feel like there is a half dozen papers that
5193980	5197940	have been published and then promptly retracted after a massive public outcry,
5197940	5204280	because they're basically trying to like reinvent phrenology, but doing it with machine learning,
5204280	5211940	and therefore it's good. There's a whole lot that can be said about this that could and actually
5211940	5218560	almost was its own panel at DEF CON. But there's a really good paper on this called
5219980	5225880	Criminality from Facial Illusion, which I'll send out on Discord and on Twitch to people interested
5225880	5231300	in reading about this problem that does a really good job of picking apart a lot of the issues
5231300	5237700	with both morally and also technologically, because a lot of these papers are fundamentally
5237700	5246880	technologically unsound with these kinds of works. So we're approaching an hour and a half.
5247540	5252020	This is the last event of the day. I'm enjoying talking to you guys a lot. I don't know if you
5252020	5258680	have somewhere you need to be, but I understand that. But if you guys are interested in continuing
5258680	5265480	to talk, I am more than happy to do so. I have to go make dinner for my family.
5269740	5274440	And it's actually sunny for a very short period of time in Seattle.
5276360	5284280	Trying to sneak in like a run. Stella, before we all disperse, I first want to thank you
5284280	5290860	so much for organizing this. You know, you have put so much thought and intention into this. Like,
5290860	5295220	I remember like asking you like, Hey, I don't want to be the male in this. Like,
5295220	5300760	have you thought about inclusive panel? I don't want to be a male. You wrote like,
5300760	5307340	perhaps the most comprehensive male, you know, about this. So thank you so much for, you know,
5307340	5313260	moving with intention, Stella. It clearly shows when you organize this panel. Thank you.
5314680	5315860	Thank you. Second.
5315860	5321420	Thank you. This is amazing. I learned so much and it's great to connect with all of you.
5321420	5322540	And thanks, Stella.
5322540	5327520	I enjoyed discussing with all of you. I think this was productive. And now I have more research
5327520	5335420	ideas that I can go on. So quickly before you guys run, I want to take advantage of the fact
5335420	5339220	that I have several experts here to get your thoughts on something I've been working on
5339220	5345640	recently, which we've, we've almost touched on this several times. And it's that a lot of times
5345640	5351140	machine learning algorithms have a underlying ontology. So Catherine mentioned, you know,
5351140	5356380	that some gender algorithms exclude transgender people, you know, but even when you're talking
5356380	5361440	about like all sorts of other applications you know, I work a lot with U.S. government data
5361440	5365120	scientists. And so they're working with data sets that come from the U.S. census. And the U.S.
5365120	5371800	census is a very specific notion of what races, and there isn't an option to use anything else
5371800	5375600	because you're, you know, you're the U.S. government and you're using U.S. census data.
5376260	5381140	But kind of how you, how you define your categories, you know, what, how you choose who
5381140	5388880	gets labeled as white, who gets labeled as Asian, really makes a big impact on what your algorithm
5388880	5393300	does and how it behaves. And whether you include certain categories, you know, if you include
5393300	5399320	transgender people or not. I mean, if transgender, if non-binary transgender people, people who don't
5399320	5407120	identify as, as, as a man or as a woman is not a possible label, then you have a guaranteed 100%
5407120	5414500	incorrect assignment rate for them. So, you know, do you have any, any thoughts or advice about how
5414500	5419280	to kind of tackle, you know, how to, how to approach this problem? Obviously, originally,
5419280	5424740	we can go and talk to the impacted communities and try to figure out what the right buckets are.
5424740	5430720	But oftentimes we start from a position where our data has been pre-collected for us by someone else.
5430720	5435820	And so if you have data that has this kind of problem in the way the questions were asked
5435820	5442980	originally, you know, is there something you can do with it still? I think that's a great question,
5442980	5447440	right? And a difficult one because, you know, the systems are already built. Like you were saying,
5447440	5454180	the categories are there. This is the label data. So I think towards getting to solutions for this,
5454180	5460180	I think an important step is being able to trust the system in terms of confidence levels,
5460180	5466200	right? Like, can the system say truly how confident it is? And the standard neural networks
5466200	5472640	don't have this property. So it can make high confidence errors. You know, it can be like 95%
5472640	5479520	confident and be wrong, right? And that especially happens on these kind of hard examples, which
5479520	5484760	hasn't seen in training data because it also gets the calibration wrong. In one of our papers,
5484760	5491380	we call it angular visual hardness. We proposed a new angular measure for detecting these hard
5491380	5496820	examples that was more robust than the standard accuracy scores and the confidence scores
5497420	5502400	that the systems are built on. And we also have some mathematical intuitions for that.
5502400	5509340	But the outcome of this is we can now look at, you know, examples which are hard for the
5509340	5514620	current classifier, right? So in this gender detection example, it would be most likely the transgender
5514620	5519640	people who would fall as hard examples because it's not getting this accurate, right? No matter
5519640	5526900	what binary gender the person is assigned to and if they don't kind of, you know, display that
5527380	5533780	gender, right? So there is that issue that can be detected through these mathematical means.
5533840	5538640	And then I think at least there is now the problem that has come up, right? That not every
5538640	5545080	example is equally treated and there are hard examples for this classifier. We can go back and
5545080	5553040	then design a system to correct that. And on a related note, I will give a vague answer to this
5553040	5561680	before I actually ask an even more daunting or deeper question. Maybe an email, for example,
5561680	5567460	idea of using more unsupervised approaches might be some kind of a solution for future generations
5567460	5573160	of AI algorithms. Maybe you wouldn't need to categorize yourself as a person, but instead some
5573160	5580720	kind of more unsupervised continuous representation of social entities might be a way to approach this
5580720	5586480	problem. This is not going to happen anytime soon, so I don't really have an answer for today's
5586480	5592520	problems because we know that gender is reflected to other features, all other features as proxies
5592520	5598740	as well. Because when we think about gender, that's the largest social group in society. So
5598740	5606160	it's one of the biggest proxies in the data sets. But my more concerning question is semantics,
5606160	5613040	gender, and race is defined in categories. So it's a proxy, but when you think about it,
5613040	5621180	essentially race doesn't really exist or gender might not even exist, but we have it in our
5621180	5627820	language, in our linguistic system, and that's the way we are constructed to deal with the world. So
5627820	5634260	how are we going to change these problems that are social constructs in AI systems that we are
5634260	5639980	building on these social constructs? So it's a longer-term problem that might require longer-term
5639980	5649400	solutions, which might not be very helpful for your question. Yeah, that really gets us something
5649400	5655440	unfortunate, and a lot of people in the Discord are asking applied questions that I haven't raised
5655440	5659740	because I know that the answer is, we don't know how to do that. We would love to know how to do
5659740	5668280	that too. So one example question is, if you can't bring in experts or social scientists or
5668280	5674080	stakeholders, is there an algorithmic way to kind of pick up on the bias? And the answer is, well,
5674080	5679100	there's 50 and we don't know which ones work better in different contexts. I mean, but there
5679100	5685360	is an algorithmic way to educate yourself, and that's like search engines, right? That's what?
5685560	5689400	That's like, that could be search engines, just like, you know, go type, like, you know,
5689400	5694680	if you want to learn about bias, go read Aileen's papers. If you want to learn about, like, you know,
5694680	5701600	bias, go watch, like, Timnit Gupru's, like, workshops. So I think a lot of this hands-on
5701600	5707360	things, people, smart people have already put out content, and it's just like a matter for us to
5707360	5714040	spend, like, cycles to educate ourselves on. Yeah, and as a, like, very basic kind of check,
5714040	5721080	like, you can look at the output of your system, if it has to do with people, and just look across
5721660	5728320	intersectional groups. So, like, you do, like, assuming that you have gender data and race data,
5728320	5733160	maybe you have income data, or maybe you have age, or maybe you have, like, some other kinds of,
5733160	5741040	you know, dimensions that describe people, is just looking at disparate outcomes within each thing,
5741040	5745400	and then combining those in various different ways to see if there are disparate outcomes
5745400	5751360	amongst the categories. Because, like, the theory of intersectionality from black feminism
5751360	5757900	says that, like, these categories compound in, like, a non-linear way, right? So being,
5757900	5761980	experiencing sexism as a white woman is fundamentally very different from experiencing
5761980	5768400	sexism and racism as a black woman. So, I mean, I think that's a very basic kind of self-audit
5768400	5773280	that you can do when you're working with data that's about people, that, like, you don't need
5773860	5777460	all that much, like, special training, necessarily. But, I mean, I think other people are working on
5777460	5780160	this, right? I mean, I feel like you guys would know this better than me, but, like, folks have
5780160	5786220	been working on things, like, where you would substitute out, like, you do, like, a sort of
5786740	5790680	counterfactual, and you look at, well, what if this data point, like, they have this outcome,
5790680	5794560	but what if they had, their race had been white, would they have had the same outcome with my
5794560	5799280	system? So I think people are working on those kinds of methods. I think just a really basic
5799280	5806820	one is just do a kind of an intersectional audit. Well, Stella, I, for one, look forward
5806820	5812920	to reading your work when it comes out. Thank you so much for joining. Thank you guys so much
5812920	5817940	for joining us. We've gone horribly over, but I've had a wonderful time talking to all of you,
5817940	5824200	and I know that the people listening really enjoyed it. If anyone wants to continue this
5824200	5830640	conversation who's listening, if you head over to the DEF CON Discord, there's a voice channel
5830640	5835500	where I'm happy to continue to talk to anyone who wants to ask questions or get references to
5835500	5840780	papers that people brought up and such. Thank you again to all of our wonderful panelists.
5840780	5848640	This has been amazing. So thank you very much. Thank you. Thank you. Wonderful to talk with
5848640	5855920	all of you. All right. Bye. I was also going to say we've also, we're going to have a journal
5855920	5867620	club focused on ethics and privacy with AI starting on the 18th. It's organized by Suhaka
5868920	5874220	on Thing, and you can check out our website for a link to a description of the whole thing.