[00:00.600 --> 00:06.700]  All right, welcome. This is the AI Village panel on AI ethics, where we're going to discuss the
[00:06.700 --> 00:11.800]  ethical implications of AI technologies and especially their deployment both in general
[00:12.340 --> 00:18.580]  and also specifically in data science and computer security. We have four lovely panelists
[00:18.580 --> 00:24.780]  from a variety of backgrounds here to discuss the topic with us. Catherine is a HackerMOMA
[00:24.780 --> 00:31.140]  scholar and artist who focuses on feminist technology, data literacy, and civic engagement.
[00:31.480 --> 00:37.020]  She wrote a 2020 book called Data Feminism, co-authored with Lauren Klein, where she
[00:37.020 --> 00:42.220]  charts a course for more ethical and empowering data science practices. Catherine is an assistant
[00:42.780 --> 00:48.300]  professor of urban science and planning at the Department of Urban Studies and Planning at MIT,
[00:48.300 --> 00:54.100]  where she's also the director of their Data Plus Feminism Lab. Ram is a data cowboy within the
[00:54.100 --> 00:59.700]  trustworthy ML team at Microsoft, working on the intersection of machine learning and security.
[00:59.700 --> 01:04.700]  His team's goal is to empower engineers to develop and deploy ML systems securely.
[01:04.740 --> 01:09.220]  Ram is also an affiliate at the Berkman Klein Center at Harvard University, where he explores
[01:09.220 --> 01:15.640]  the societal implications of adversarial AI. Ai-Lin is an assistant professor of computer
[01:15.640 --> 01:22.180]  science at George Washington University and at the Institute for Data, Democracy, and Politics.
[01:22.180 --> 01:27.780]  She is interested in AI ethics, bias in AI, machine learning, and the implications of machine
[01:27.780 --> 01:33.000]  learning on privacy, security, fairness, and society. Much of her research is on methods to
[01:33.000 --> 01:39.340]  interpret AI and gain insight into the bias in both artificial and natural intelligence.
[01:39.680 --> 01:46.180]  And she has worked to demonstrate how human-like biases can crop up in natural language analysis
[01:46.700 --> 01:53.920]  due to the statistical regularities of language. And finally, Anima is the Bren Professor at Caltech
[01:53.920 --> 01:59.640]  and the Director of ML Research at NVIDIA. She was previously a principal scientist at Amazon
[01:59.640 --> 02:06.820]  Web Services. She's received several honors, such as the Alfred P. Salone Fellowship,
[02:06.820 --> 02:14.540]  NSF Career Award, Young Investigator Awards from DoD, and several more. She is part of the
[02:14.540 --> 02:18.940]  World Economic Forum's Expert Network and is passionate about designing principled AI
[02:18.940 --> 02:25.900]  algorithms and applying them to interdisciplinary outcomes. She has been outspoken on AI bias and
[02:25.900 --> 02:31.640]  the need for democratization and focuses her research on unsupervised AI optimization and
[02:31.640 --> 02:36.940]  tensor methods. They are all wonderful people and I can personally vouch for the fact that
[02:36.940 --> 02:42.540]  super cool research and I'm more than thrilled to talk with them all.
[02:42.600 --> 02:49.700]  So to kind of get us started, there's been a lot of conversation, at least recently, about AI bias
[02:49.700 --> 02:55.280]  and misuse of AI in the news in the United States recently. A lot of tech companies have talked
[02:55.280 --> 03:00.760]  about how they're going to stop using facial recognition algorithms or stop providing them
[03:00.760 --> 03:10.740]  to police to use. And there's also been a lot of conversation both in the academic literature and
[03:10.740 --> 03:16.200]  also in the national news media about biased AI, the AIs that learn to be racist because they look
[03:16.200 --> 03:21.460]  at racist data or learn to be sexist because they look at sexist data. But AI ethics is kind of a
[03:21.460 --> 03:26.940]  much broader topic than that. So to kick us off, can each of you say a little bit about what the
[03:26.940 --> 03:32.800]  scope of the field is to you and how it relates to the work that you do? Ram, why don't you give us a start?
[03:33.040 --> 03:39.940]  Oh, thank you. First of all, I'd like to say that I'm not an AI ethics expert. I'm so glad that Anima,
[03:39.940 --> 03:47.080]  Catherine, Eileen are here along with Stella. So you should be getting like solid like stellar
[03:47.080 --> 03:52.940]  advice from them. So whatever I'm going to talk is through my lived experience and interacting
[03:52.940 --> 03:58.940]  with folks like Kendra Albert, Jonathan Penny from Berkman and along with Mary Gray and a whole
[03:58.940 --> 04:05.900]  host of other folks. So that's my, you know, that's my preface for this. So for me, like
[04:06.460 --> 04:13.960]  I've been looking at Berkman at the, you know, what is adversarial ML? How does that impact the
[04:13.960 --> 04:21.800]  experience of citizenship? So one of the things that I, you know, for me, like how I, you know,
[04:21.800 --> 04:27.140]  how we've been interpreting this topic is moving away from just like an, you know, like the
[04:27.140 --> 04:33.220]  constraint of like ethics and thinking about power structures, especially in the context of adversarial
[04:33.220 --> 04:39.240]  machine learning. So turns out that, you know, machine learning is political in a Langdon Winner
[04:39.240 --> 04:45.600]  sense and security has a lot of like implications, like, you know, if you think of the moral character
[04:45.600 --> 04:52.620]  of cryptography by Rod Wave. And adversarial machine learning, which is what I work on as a
[04:52.620 --> 05:00.840]  researcher, sits in the intersection of these two topics. So thinking through who is securing like
[05:00.840 --> 05:06.540]  the ML systems? What is securing? Who are we securing it against? Who is the attacker?
[05:06.540 --> 05:11.460]  Are we, is the attacker just like anybody who compromises the confidentiality, integrity,
[05:11.460 --> 05:16.980]  and availability? So those are kind of the broader questions that I'm looking at at Berkman.
[05:18.040 --> 05:19.780]  All right, Ellen.
[05:22.140 --> 05:28.340]  Oh, Stella, thank you for the introduction. And I'm very excited to be part of this amazing panel
[05:28.340 --> 05:41.440]  so that I will get to hear all of your ideas and vision on this topic. My research focuses on,
[05:41.440 --> 05:46.660]  working on the implications of machine learning on privacy and security.
[05:46.660 --> 05:52.200]  Since machine learning or AI technologies are double-edged swords, when they have security
[05:52.200 --> 05:58.080]  enhancing properties, they usually end up infringing upon our privacy. And starting from
[05:58.080 --> 06:05.020]  there, I started slowly seeing the link between privacy and fairness and how when our data is
[06:05.020 --> 06:11.800]  exploited, abused, or collected by third parties, it can lead to fairness concerns at the group
[06:11.800 --> 06:22.180]  level. And nowadays I'm focusing on understanding how AI systems learn bias or essentially how
[06:22.180 --> 06:29.420]  systems can learn some form of intelligence or learning ability and how that relates to bias in
[06:29.420 --> 06:35.100]  natural and artificial intelligence. And when we looked at natural language systems,
[06:35.100 --> 06:40.640]  for example, word embeddings or today's neural language models that can generate text that is
[06:40.640 --> 06:48.280]  almost human level, we see that these systems are learning human-like biases from society
[06:48.280 --> 06:54.860]  very accurately, especially when compared to validated social psychological methods such as
[06:54.860 --> 07:01.120]  implicit association tests. And then these systems are deployed at the world scale impacting
[07:01.120 --> 07:07.420]  billions of people every single day with their consequential decisions or biased outputs.
[07:07.420 --> 07:14.040]  And this ends up turning into this AI bias lifecycle, which is also a feedback cycle
[07:14.040 --> 07:20.360]  of AI ethics as well. And I'm interested in understanding all of the components in this AI
[07:20.360 --> 07:30.320]  ethics lifecycle, the role humans play in this lifecycle, how we can discover biases or harmful
[07:30.320 --> 07:36.900]  side effects of these AI systems, and what we can do to mitigate these harmful effects and prepare
[07:36.900 --> 07:43.380]  for upcoming problems that might have already been caused by AI systems. And during today's panel,
[07:43.380 --> 07:48.660]  we will go into the details of all of these single components, starting from the most technical
[07:48.660 --> 07:54.940]  atomic details of machine learning, going up to probably some philosophical questions at the
[07:54.940 --> 08:06.640]  high level abstraction. Thank you Aylin. Anima? Yeah, I completely agree with Aylin and Ram.
[08:06.640 --> 08:13.320]  You know, it's to me exciting times to be an AI researcher, but also, right, I'm filled with
[08:13.320 --> 08:21.740]  dread that where the world is moving and how fast we are, you know, moving in directions that I find
[08:21.740 --> 08:27.920]  problematic. And to me, the source of it all comes to the foundations under which this deep learning
[08:27.920 --> 08:33.920]  revolution is built on, right? So I'd like to emphasize when I'm giving talks on AI about
[08:33.920 --> 08:39.120]  this trinity, you know, the data, algorithms, and computer infrastructure coming together.
[08:39.120 --> 08:47.280]  And that's what allowed us to have this fast progress in deep learning. But most of that
[08:47.280 --> 08:52.560]  progress has been with supervised learning, right? The availability of large-scale label data starting
[08:52.560 --> 08:58.560]  from the ImageNet data set was the start of this revolution. And in a way, that is quite
[08:58.560 --> 09:04.000]  straightforward. Once you know that you want to optimize for accuracy and you have the label data,
[09:04.000 --> 09:09.840]  you can kind of quickly build these systems. But the problem is sometimes we want not just the
[09:09.840 --> 09:15.000]  accuracy, but like robustness, right? Like what Ram was saying, what are the different changes
[09:15.000 --> 09:21.340]  beyond the training data? And when our training data is biased, how does this affect on minority
[09:21.340 --> 09:27.060]  populations that are not represented in training data? And part of it is not just the data imbalance,
[09:27.060 --> 09:32.260]  but also the algorithmic bias. You know, if you are on Twitter like me, and if you follow the
[09:32.260 --> 09:38.480]  Twitter storm on the presence of algorithmic bias on deep learning, you know what I'm talking about.
[09:39.460 --> 09:45.660]  And just to the gist of it, the idea is I think some proponents of deep learning do not want
[09:45.660 --> 09:51.360]  anything negative being said around deep learning. But I do think these very large-scale models,
[09:51.360 --> 09:57.220]  highly nonlinear models, are the source of the problem. Because the more nonlinear it gets,
[09:57.220 --> 10:02.720]  the more brittle they are, the more they amplify the bias, right? There is no free lunch. You know,
[10:02.720 --> 10:08.740]  you cannot have it all in one mechanism unless we are thoughtful in terms of the objectives we build
[10:08.740 --> 10:15.580]  in, the kind of data we use, and also the kind of testing we do. And so what I work on is unsupervised
[10:15.580 --> 10:22.660]  learning. And I think to me, this is the core for us to move away from these highly biased models.
[10:22.660 --> 10:28.140]  You know, to take a simple example, you know, even before the deep learning revolution,
[10:28.140 --> 10:35.100]  I was working on topic models, right? Where we wanted to categorize large corpus of documents
[10:35.700 --> 10:41.420]  and there's no label data. And so now with these unsupervised learning, we are suggesting these
[10:41.420 --> 10:46.560]  are possible topics, right? And these are the words representing the topics. And then it's up
[10:46.560 --> 10:51.460]  to the human to interpret and judge what are relevant topics, you know, because there are
[10:51.460 --> 10:57.240]  weird correlations in text. So there can be lots of topics that are not human interpretable. And so
[10:57.240 --> 11:03.780]  human in the loop with unsupervised learning, I think is the key to get out of this place where
[11:03.780 --> 11:10.340]  we are stuck with biased models. And, you know, the current revolution now is unsupervised. Now
[11:10.340 --> 11:15.860]  we're seeing a lot of progress in self-supervised learning and robustness. Hopefully that'll lead
[11:15.860 --> 11:23.760]  us the way forward. Thank you. All right. Thank you. And finally, Catherine,
[11:23.760 --> 11:28.840]  where are you coming from with this? Yeah. Thanks so much. Thanks, Stella,
[11:28.840 --> 11:36.000]  for the invitation. It's a super pleasure to be here with you all. Yeah. So I guess
[11:36.580 --> 11:43.300]  where I'm coming from is not as much of a focus on AI, although some of the things that we say,
[11:43.300 --> 11:48.700]  many of the things we say in our book, Data Feminism, apply, but we sort of enlarge the
[11:48.700 --> 11:56.940]  scope more broadly to be data writ large. And thinking about also, you know, one of the things
[11:56.940 --> 12:02.920]  that's happened as this conversation has grown around fairness, accountability, transparency,
[12:02.920 --> 12:10.940]  bias, ethics, and so on. You know, there's been a lot of pushback on just like AI and data are
[12:10.940 --> 12:18.440]  great. They're working great for everybody. And our contribution with Data Feminism is to say,
[12:18.440 --> 12:24.960]  hey, actually, feminism as a kind of body of knowledge, as a kind of intellectual heritage,
[12:24.960 --> 12:30.320]  which has theorized, philosophized, and also done a lot of writing and sort of action,
[12:30.320 --> 12:37.560]  like social movements, action around the topic of inequality has a lot to offer us in terms of
[12:37.560 --> 12:43.780]  conceptual models for how we can think about this current state of affairs. And so that was sort of
[12:43.780 --> 12:51.280]  the premise of Data Feminism is to say, hey, let's look over here at feminism. What do we,
[12:51.280 --> 12:56.760]  what can we bring from feminism to understand how we might work with data science in a more
[12:56.760 --> 13:03.420]  ethical way, and specifically taking structural inequality, which is basically like power
[13:03.420 --> 13:10.700]  differences between social groups as kind of our main locus of attention. And so when you look at
[13:10.700 --> 13:16.200]  that, and you look at things like, you know, one of the things I guess that was irritating, I guess
[13:16.200 --> 13:22.680]  I'll say as we were writing the book, which I'm sure folks here are also irritated by, is that often,
[13:22.680 --> 13:28.140]  you know, when we have a biased, whatever, algorithm or something like this, like a resume
[13:28.140 --> 13:33.800]  screener, let's say, that is favoring men over women, you know, the media reports on it with this
[13:33.800 --> 13:39.080]  thing of like surprise, like, oh my gosh, like, how could this be possible? Like, how could there be
[13:39.080 --> 13:46.540]  bias in a machine? And we're like, oh my gosh, Data Feminism has so much to say about this.
[13:46.540 --> 13:52.740]  And so ultimately, like what the book does is it kind of backs us up. It's in a way less in the
[13:52.740 --> 13:59.180]  details. It's not providing you with a way of like systematically fixing your data set or anything
[13:59.180 --> 14:05.280]  like that. But it backs us up, like way further back in the processing pipeline to say, well,
[14:05.280 --> 14:09.720]  like, here's why these things surface. They surface because of hundreds of years of inequality.
[14:09.840 --> 14:16.020]  They surface because of really flawed models of people like women not being considered fully human
[14:16.020 --> 14:22.100]  or people of color not being considered fully human. And to really take, you know, if we aspire
[14:22.100 --> 14:28.580]  to build something that is ethical, we have to look and we have to undertake that with a different
[14:28.580 --> 14:33.760]  kind of time horizon. And that has to be built into everything that we do. So we're trying to
[14:33.760 --> 14:38.880]  kind of both say like, hey, let's look back at the root causes. What causes structural inequality?
[14:38.880 --> 14:44.580]  Like, why do we end up with biased, you know, crime data sets in the first place before we start
[14:44.580 --> 14:51.700]  trying to like fix them or something? And then how do we use that information to decide which
[14:51.700 --> 14:57.780]  problems do we work on? Who do we work with? Who do we want to benefit with our work? And who do
[14:57.780 --> 15:03.940]  we want to just like refuse to work with entirely? Because there are some really bad actors out there.
[15:04.180 --> 15:12.320]  And so that's... and then we talk a lot about democratizing data, data literacy, and AI as well.
[15:12.320 --> 15:18.060]  Because that, I think, is really part of the key of how we're gonna get through this. So we're
[15:18.060 --> 15:25.940]  trying to balance being not only doom and gloom with like, how do we move forward? And we propose
[15:26.160 --> 15:32.440]  a set of seven principles for how we move forward with a kind of feminist approach to data science.
[15:33.360 --> 15:39.820]  All right, thank you. So I think a common thread in what all of y'all said is that
[15:39.820 --> 15:46.960]  there are problems both... problems in machine learning exist both in the pipeline process,
[15:46.960 --> 15:52.180]  where that is coming from, how it's processed, but also... but that that isn't all of the...
[15:52.180 --> 15:58.260]  all of the problems. A lot of the time in the media and in conversations, people have this idea
[15:58.260 --> 16:04.300]  that data is like some kind of objective truth, and that when you come up with biased results or
[16:04.300 --> 16:10.140]  incorrect results, it's because the analyst messed up somewhere. So, you know, do you think that's
[16:10.140 --> 16:18.420]  true? Spoiler alert, I'm pretty sure you're all going to say no. But, you know, what would you say
[16:18.420 --> 16:23.780]  to someone who thinks that way? And like, what is... what are the problems with thinking about
[16:23.780 --> 16:32.860]  data that way? Whoever wants to kind of get started. I can quickly start with a positive
[16:32.860 --> 16:41.380]  note. Data certainly reflects the state of the society, but it does it in a defective way. So
[16:41.700 --> 16:48.080]  we are not getting a perfect representation of society, especially with regards to minorities,
[16:48.080 --> 16:54.180]  because they are underrepresented. But when we have such data sets, at the same time,
[16:54.180 --> 17:00.180]  one advantage is that we can also study the historical injustices these minorities,
[17:00.180 --> 17:05.180]  underrepresented or disadvantaged groups have been going through, because it's reflected in
[17:05.180 --> 17:13.880]  the data set. As I already mentioned, since these data sets are a reflection of society,
[17:14.240 --> 17:21.980]  and then when we start building models on these systems, then we have data that is reflecting
[17:21.980 --> 17:27.940]  these biases. But at the same time, it also reflects some facts about the world, such as
[17:27.940 --> 17:34.600]  statistical biases, inductive biases, or some kind of facts that can also be useful.
[17:34.600 --> 17:41.420]  But that does not justify the fact that there are all these harmful information that gets
[17:41.420 --> 17:49.560]  very accurately embedded in these systems. So this is my double-edged view on this
[17:50.440 --> 17:57.780]  data topic for now. I can follow up with that. I completely agree with Arlene. I think
[17:58.360 --> 18:05.440]  the problem is how the data on the internet is reflecting the society, but only in very
[18:05.440 --> 18:11.400]  balanced ways. And this gets amplified when you want large data sets. The latest
[18:12.060 --> 18:20.800]  one in this aspect is the GPT-3. It's trained on links from Reddit. I'm almost never on Reddit.
[18:20.800 --> 18:25.920]  I've only been a victim of trolling and gun threats on Reddit. I stay away. I don't know
[18:25.920 --> 18:30.420]  any woman in machine learning that I'm aware of who is active on Reddit.
[18:30.420 --> 18:36.300]  So you can see that even among all the imbalance in the internet,
[18:36.300 --> 18:41.160]  the Reddit is even further amplified. So the decision to go for Reddit links and claim
[18:41.160 --> 18:47.320]  that's what we want to build the largest language model on, we can spend tens of millions of dollars
[18:47.840 --> 18:54.800]  on this data set and nothing else, is to me extremely disappointing and disheartening that
[18:55.920 --> 19:04.780]  the VCs, the decision makers, they're always going for the worst of the toxic forms of
[19:04.780 --> 19:09.840]  data that's available on the internet. And I really can't justify this. I don't know how they
[19:09.840 --> 19:15.000]  sleep every night. I don't know how they can say with a straight face that this is the path
[19:15.000 --> 19:21.640]  towards AGI. I mean, this is nonsense, right? So I get frustrated all the time. I know Aileen
[19:21.640 --> 19:27.860]  was trying to paint a more positive picture. I'm just extremely disappointed that I'm sorry for
[19:27.860 --> 19:34.360]  being open about that. Yeah, well, maybe I'll pop in on the negative note.
[19:36.420 --> 19:42.940]  I mean, what happens that I think we're pointing to here is that data are very resource intensive,
[19:42.940 --> 19:47.360]  right? And so it's only certain actors, it's something we say in the book, it's only certain
[19:47.360 --> 19:56.820]  actors in the world that have the resources to collect, store, maintain, utilize, analyze, deploy
[19:56.820 --> 20:03.660]  in some product or whatever, these kinds of data. And so at root, you only have certain actors in
[20:03.660 --> 20:09.900]  the world that are able to actually do that work. And because data are so expensive, we see all
[20:09.900 --> 20:14.020]  these kinds of shortcuts, like exactly like this, where it's like, oh, like, where can we get a big
[20:14.020 --> 20:20.400]  data set? Okay, Reddit, or like, oh, you know, where can we get images to train image, you know,
[20:20.400 --> 20:25.100]  face recognition algorithm? Oh, okay, like celebrity faces or whatever. And just like not
[20:25.100 --> 20:32.280]  connecting the dots of again, like, in what ways like who is not represented in those data sets.
[20:32.280 --> 20:38.260]  And there's like all these multiple points of kind of where inequality kind of enters that
[20:38.260 --> 20:43.240]  pipeline and like not catching it at any of those points, because then the people that are working
[20:43.240 --> 20:51.320]  with the data are often they are the white elite, non-immigrant men, you know, and they
[20:51.320 --> 20:56.080]  are representative of that demographic that are celebrities, right? So like, you know, that you
[20:56.080 --> 21:00.800]  have these kind of multiple stages at which those things don't get caught. But I mean, at the root
[21:00.800 --> 21:08.080]  of it, I often think of the constraint in a lot of ways is, it's financial, like folks are looking
[21:08.080 --> 21:21.200]  for a shortcut. I mean, I'm so glad, you know, Catherine, you brought the issue of like celebrity
[21:21.200 --> 21:28.420]  data set. So in the news event, you know, you've heard about Fox, which is kind of like a cloaking
[21:28.420 --> 21:34.200]  tool, you know, for privacy, it uses like very fundamental techniques and adversarial machine
[21:34.200 --> 21:39.380]  learning so that, you know, people can kind of, you know, like save their privacy from like terrible
[21:39.380 --> 21:47.740]  tools like Clearview AI. And Kashmiri Hill, kind of like the New York Times reporter, tried it out.
[21:47.860 --> 21:54.000]  It was so weird, you know, she got she got some really weird photos. And when she asked the PI
[21:54.000 --> 21:59.500]  of that project, like, hey, why are these photos looking really awry? I thought the whole point is
[21:59.500 --> 22:06.260]  to keep the face same in the cloaked picture. And he said something to the effect, and I'm
[22:06.260 --> 22:12.800]  perhaps going to misquote him, he said, Oh, yeah, women get mustaches and like, you know, men get
[22:12.800 --> 22:17.400]  eyelashes. That's how you kind of like cloak. You know, that's kind of like an outcome. I'm sure
[22:17.400 --> 22:24.240]  there's a whole like, semester of SDS gender studies to unpack there, because they use like
[22:24.240 --> 22:30.840]  celebrity faces. And like, how many transgenders are represented in like celebrity data sets?
[22:31.240 --> 22:37.160]  My first thought was, have you seen the winner of Eurovision 2019? Like, shame on you. Have you
[22:37.160 --> 22:44.700]  seen like RuPaul's Drag Race? Shame on you. But, but yeah, it is. And I'm so sorry, Eileen, that,
[22:44.700 --> 22:51.180]  you know, we're not like raining on your parade. But even though, even though somebody starts with
[22:51.180 --> 22:56.180]  really good intent, and I'm sure like the Fox team did it with very good intent of like, I am going
[22:56.180 --> 23:02.920]  to, you know, forward privacy, I'm going to like, you know, empower people. But like you said, like
[23:02.920 --> 23:07.300]  when they don't connect the dots, or like how Anima said, when they take like shortcuts,
[23:09.040 --> 23:14.060]  yeah, it doesn't lead to less than like harmful outcomes.
[23:14.600 --> 23:20.000]  So just as a quick reminder for the people listening, Fox is the paper that we discussed
[23:20.000 --> 23:26.100]  yesterday at Journal Club. So if you were there for that, awesome. If not, you can read about the
[23:26.100 --> 23:30.600]  paper. You can listen to me and a couple others talk about the paper I was talking about on our
[23:30.600 --> 23:34.920]  YouTube channel, where it should be up by now. Sorry, Eileen, did I cut you off?
[23:35.260 --> 23:38.600]  Oh, I saved my pessimistic comments to the end.
[23:41.420 --> 23:47.400]  But now we have our feedback loop completed. So I'm really curious about all the data practices
[23:47.400 --> 23:52.440]  that are going on out there, or the data impractices or abuse, because as I said,
[23:52.440 --> 23:57.620]  I'm coming from a privacy background, and how people's data is abused, so that they can be
[23:57.800 --> 24:04.180]  de-anonymized, they can be tracked, and essentially, entities that hold power can use this for
[24:04.180 --> 24:11.700]  surveillance or for large scale manipulation. And unfortunately, as citizens, or as human beings
[24:11.700 --> 24:17.160]  living in the world, we don't consent to our data being collected, but we don't have an option to
[24:17.160 --> 24:26.040]  opt out. So all of our data, under the premise of free applications, free products, or just walking
[24:26.040 --> 24:33.780]  on the streets in country X, is being collected by these entities that hold power to build large
[24:33.780 --> 24:39.520]  databases of any kind of information. And in some cases, this data is not being even used,
[24:39.520 --> 24:43.340]  because they think that we don't have a way to use it right now. But in the future, we will
[24:43.340 --> 24:48.920]  figure it out, figure out a way to make use of this data to make one type of consequential
[24:48.920 --> 24:55.480]  decision again. And this is extremely problematic, because the citizen is not informed about these
[24:55.480 --> 25:03.100]  practices and what the long term effects might be on individuals and society. And on the other hand,
[25:03.100 --> 25:10.400]  when we look at other power holders, such as proprietary software, or all companies that are
[25:10.400 --> 25:17.240]  providing these services for the entire world that are free, supposedly free, are making so much
[25:17.240 --> 25:23.840]  money because they are making use of the data that we are providing them. And after that, they keep
[25:23.840 --> 25:30.620]  building stronger machine learning models so that they can keep amplifying these biases in their
[25:30.620 --> 25:36.900]  direction with their targeted ads, or whatever their incentive is. And here there is a serious
[25:36.900 --> 25:44.420]  problem about the incentives of these systems. And both Catherine and Anima mentioned that,
[25:44.420 --> 25:48.880]  for example, we have these objective functions optimizing for accuracy in these systems
[25:49.380 --> 25:56.660]  that actually are helping these power entities. But the objective function could also be how can
[25:56.660 --> 26:04.660]  we use these data sets for social good, as opposed to optimizing for accuracy. So there are so many
[26:04.660 --> 26:11.600]  different faceted and different levels of abstraction of problems involved in this
[26:13.180 --> 26:18.200]  data ecosystem again. And I guess we can keep discussing these in more details,
[26:18.200 --> 26:24.080]  but today the goal is kind of give a glimpse of what is going on in this area.
[26:24.480 --> 26:29.740]  So Aileen, you picked out... guess wrong? Oh sorry, I just have like a quick follow-up for
[26:29.740 --> 26:37.020]  Aileen and Anima. I know that both of you have kind of revealed the shortcomings of ML systems.
[26:37.020 --> 26:42.440]  Anima, you've done extremely awesome work about how GPT-3 is flawed. And Aileen, your groundbreaking
[26:42.440 --> 26:50.080]  work on word embeddings. And I know that both of you have communicated that research to the
[26:50.080 --> 26:57.320]  community in a very approachable fashion. But I don't know about you, Aileen, but I visibly see
[26:57.320 --> 27:05.340]  the kind of like negative, rancorous feedback that you get. And from very well-intentioned
[27:05.340 --> 27:10.400]  research, I mean I don't know well-intentioned, from like people who are in the ML community,
[27:10.400 --> 27:16.280]  why do you think that happens? And Catherine, like from a feminist perspective, why are people from
[27:16.280 --> 27:25.220]  the own educated community like unloading on these researchers? I would love to get your
[27:25.220 --> 27:30.240]  perspectives on this. Sorry, Stella, I didn't mean to like... No, that's fine. That's very close
[27:30.240 --> 27:35.820]  to where I was going to go anyway. I mean, just maybe I can get started on this, right? I mean,
[27:35.820 --> 27:40.620]  to me, I think it's at the root of human psychology. Like no one wants to be the bad guy.
[27:40.720 --> 27:45.620]  No one wants to think of themselves as the bad guys. So they believe, right, they are going
[27:45.620 --> 27:50.980]  towards AGI through this GPT-3 model, for instance, right? I mean, I think they truly believe.
[27:50.980 --> 27:59.640]  So let's assume that. And I think the thing is they really don't want to also hear anything
[27:59.640 --> 28:05.720]  that may harm that, right? So they believe anything that may harm that path is that are
[28:05.720 --> 28:10.320]  their enemies. People who are voicing opinions against it are their enemies. And so that's why
[28:10.320 --> 28:17.840]  we see this vicious pushback on social media platforms like Twitter and, you know, and then
[28:17.840 --> 28:22.380]  they have huge follower counts. So that builds up, right? And all they have to do is
[28:22.960 --> 28:29.720]  do it, just kind of give their followers a sense to go after people who are bringing up these
[28:29.720 --> 28:34.200]  difficult issues. And honestly, because there's also a lot of insecurity, they don't know the
[28:34.200 --> 28:40.640]  answers to this. They don't know a way out. And they worry that their entire existence is in
[28:40.640 --> 28:46.500]  question, which I think that's why, you know, to me, I feel like we are on the right track. So we
[28:46.500 --> 28:52.880]  have to keep pushing this on and increasing awareness, have more allies. I think we will
[28:52.880 --> 28:58.720]  get more support. And it's already been amazing over the last few years, right? I mean, it's
[28:58.720 --> 29:04.320]  happened so quickly, even just the mention of AI bias was not there. It was completely dismissed
[29:04.320 --> 29:10.180]  in machine learning conferences. People working on it were considered not technical, you know,
[29:10.180 --> 29:15.320]  were not respected. There was a lot of gaslighting, you know, because it happens to be women, it
[29:15.320 --> 29:22.320]  happens to be minorities. So there was even more of all that racism and sexism coming into play when
[29:22.320 --> 29:28.540]  such research was reported. But now, you know, especially with the Black Lives Matter movement,
[29:28.540 --> 29:34.060]  that's when these companies finally said, we won't be selling face recognition to law enforcement,
[29:34.060 --> 29:38.140]  right? We've been asking for this, you know, we've signed letters, we've gone to the press,
[29:38.140 --> 29:43.700]  we've been doing this for a while. And it's been led by black women, which is amazing.
[29:43.700 --> 29:50.040]  But, you know, it took a broader global movement to make this happen. And I think we are building
[29:50.040 --> 29:55.600]  this momentum up, because it's not just about AI, it's not just about data ethics. It's a much
[29:55.600 --> 30:01.260]  broader question for humanity. And I think we're, you know, I have to, I remain hopeful.
[30:01.960 --> 30:07.360]  So since Anima brought this up, I wanted to take a moment and ask you guys, do any of you,
[30:07.360 --> 30:14.600]  as yourselves personally, or the companies or institutions that you represent, have any
[30:14.600 --> 30:21.700]  policies about what types of organizations or what types of tasks you will not do ML research
[30:21.700 --> 30:26.700]  or data research for? You know, be it police applications, military applications are two of
[30:26.700 --> 30:33.300]  the ones I know are really popular. I mean, speaking personally, disclaimer, sorry, before
[30:33.300 --> 30:38.480]  I say this, my company has a large number of lawyers who'd like me to tell you that my reviews
[30:38.480 --> 30:43.820]  on the stream represent myself and in no way represent my company at all. I work for a US
[30:43.820 --> 30:50.260]  government contractor. And a lot of the people who work at my company do ML research for the
[30:50.260 --> 30:55.900]  Department of Defense and the US military. And that's a very contentious topic, both within my
[30:55.900 --> 31:02.640]  company and I know within the ML community at large. So, you know, I have been personally trying
[31:02.640 --> 31:07.640]  to decide how to navigate that line myself. But I was wondering if any of you guys had
[31:08.400 --> 31:16.880]  particular stances you've developed? I mean, I'll just say, for my own work,
[31:16.880 --> 31:21.340]  and I'm not speaking about MIT, because obviously MIT is all wound up with all sorts of different
[31:21.340 --> 31:29.800]  things. But at least as a professor, you have autonomy to do what you want. And so at least
[31:29.800 --> 31:37.060]  from my perspective, I've decided to only work with folks who are doing some kind of counter
[31:37.060 --> 31:45.660]  hegemonic work. So activists, journalists, social movements, community based organizations.
[31:45.660 --> 31:51.520]  That's also kind of the, like, that's also in my research interest is to think about, like,
[31:51.520 --> 31:57.820]  how do we think about building data literacy in those kinds of sectors as well. So it's like a
[31:57.820 --> 32:03.240]  really nice fit. And that happens to be where my interests are. But I'm sympathetic to the stance
[32:03.240 --> 32:07.400]  that like, yeah, when you work in a large organization, as many of these places are,
[32:07.400 --> 32:10.640]  you have a lot less autonomy and thinking about, well, like, how do you navigate that as an
[32:10.640 --> 32:15.940]  individual? How do you navigate that as a team? And how do you navigate that at the level of the
[32:15.940 --> 32:22.560]  institution? But I will say, like, I am really energized by all of the organizing and activism
[32:22.560 --> 32:27.620]  that's happened within large companies right now as well. And so I think that is a huge,
[32:28.400 --> 32:37.900]  potentially, is a huge force for change as well. Yeah, I agree. And I want to add what Stella
[32:37.900 --> 32:45.780]  said earlier, right? I am not representing NVIDIA or Celtic for that matter. But I see a lot of
[32:45.780 --> 32:51.900]  support building up from the grassroots in these organizations. And NVIDIA being the platform for
[32:51.900 --> 32:57.460]  AI computing, you know, we are not directly selling these kind of AI applications, right? So it's
[32:57.460 --> 33:03.100]  really our customers and partners. And so there is less of a control in sense of everything the
[33:03.100 --> 33:07.920]  others do, but there's always periodic reviews. And, you know, there have been discussions
[33:07.920 --> 33:15.260]  internally in terms of how to make sure that our platforms are being used for good causes rather
[33:15.260 --> 33:22.480]  than bad ones. But I agree, there is a lot more work to be done. But I'm very energized by activism
[33:23.020 --> 33:28.260]  internally at NVIDIA, and also at Celtic, especially the students, everything, you know,
[33:28.260 --> 33:33.160]  from the recent Black Lives Matter, how it was really the students who have been
[33:33.660 --> 33:37.440]  energizing this movement. And so that gives me a lot of hope.
[33:38.240 --> 33:42.020]  Yeah, and I'm happy. Oh, sorry. Go ahead, Aileen. Please.
[33:42.020 --> 33:49.040]  I was just going to say that I'm not representing anyone except my humble opinion that might not be
[33:49.040 --> 33:55.680]  very well formed. But I try to build my own ethical framework when approaching these issues,
[33:55.680 --> 34:01.620]  since we don't have the AI ethics framework that is widely accepted by anyone. I basically
[34:01.620 --> 34:07.340]  try not to work on any confidential project. I don't try, I haven't worked on any confidential
[34:07.340 --> 34:14.080]  or secret projects. And other than that, I try to get involved in pro bono activities to help
[34:14.080 --> 34:20.780]  companies, entities, research their systems to understand what kind of negative side effects
[34:20.780 --> 34:29.120]  they might have, especially with regards to bias, can we come up with new forms of ways to measure
[34:29.120 --> 34:36.600]  bias, develop methods. And that has been my approach to this issue right now. But if we had
[34:36.760 --> 34:46.140]  a well accepted code of ethics among researchers, I think everyone would benefit from such a framework.
[34:49.720 --> 34:57.640]  I mean, I totally agree. I think one of the things that's kind of like helped me personally,
[34:57.640 --> 35:02.940]  again, you know, boilerplate, boilerplate, I'm only representing myself. When Anima said like
[35:02.940 --> 35:10.720]  reviews that kind of like really resonated with me. There is like, within Microsoft,
[35:10.720 --> 35:18.000]  there's this like committee called the AI Ethics and Research Organization. It's called ETHER.
[35:18.000 --> 35:23.520]  I'm perhaps butchering the acronym. But one of the things that they do, and it is like,
[35:23.520 --> 35:29.800]  you know, folks from MSR, you know, and it's just not the ML folks, but MSR is such a vibrant
[35:29.800 --> 35:34.460]  community. So, you know, you've got like sociologists, you know, folks from like,
[35:34.460 --> 35:42.080]  who think about bias from MSR New York, in Eric Horowitz's organization, kind of like,
[35:42.080 --> 35:48.320]  you know, making sure for critical like ML applications, you know, you need to get ETHER's
[35:48.320 --> 35:55.820]  sign off. So that has been a, you know, a guiding light. The second thing for me is like, you know,
[35:55.820 --> 36:02.540]  it's, I completely agree with Eileen, if there's some like, like a framework, and I, you know,
[36:02.540 --> 36:07.240]  that way you can meet it out to engineering engineers. The thing that's kind of worked for
[36:07.240 --> 36:12.860]  me is that the standards of business conduct at Microsoft is surprisingly good, which is a very,
[36:12.860 --> 36:18.180]  which is a very good thing to say, because they make it, they shoot it like Netflix style videos.
[36:18.240 --> 36:23.040]  So it's not the boring training that you have to kind of go through. I mean, I mean, I've worked
[36:23.040 --> 36:28.740]  in Microsoft, like, in 2013, it was just like the most like boring thing, but you still have to do
[36:28.740 --> 36:33.120]  it. It's like a read through it. But now it's like an actual like Netflix series, they've got the
[36:33.120 --> 36:41.520]  same actors coming in different years. But I mean, long story short, there was a module about how
[36:42.180 --> 36:49.620]  this guy tries to train his model on this, like private customer data, and how he gets into such
[36:50.800 --> 36:56.340]  so much of, you know, like, like a big, like big soup, he can't get out of it, you know,
[36:56.340 --> 37:00.440]  he's trying to demo it and he has to like talk to his manager and managers like, oh my gosh,
[37:00.440 --> 37:06.160]  you're gonna like, you know, lose your job. So they really rain it down on you in a more relatable
[37:06.160 --> 37:13.980]  fashion. So all of this to say that, you know, having this, you know, this committee to kind of
[37:13.980 --> 37:21.680]  like guide for critical like applications, and kind of having this like relatable, you know,
[37:21.680 --> 37:30.420]  ethics discussion has kind of like helped at least me and my team personally a lot,
[37:30.420 --> 37:34.320]  quickly ask Ram, so can they release this to the public? I think
[37:36.080 --> 37:41.060]  immensely from this as well. Yeah, I really hope that they do because it's,
[37:41.640 --> 37:47.220]  it's really captivating, like you're not bored. You know, when you're watching that.
[37:48.020 --> 37:52.380]  So I have a question from the discord that is something you guys have all touched on a little
[37:52.380 --> 37:56.940]  bit, but I think it's worth like directly addressing, which is that so we've been talking
[37:56.940 --> 38:02.700]  about how we were, we all represent large companies, or major universities, you know,
[38:02.700 --> 38:10.940]  institutions that have power and money and resources. And, you know, how, what is it?
[38:11.120 --> 38:18.020]  So what the question asked on discord was, was, you know, if, what's the role that
[38:18.700 --> 38:24.380]  these large institutions is powerful and franchise institutions play? And, you know,
[38:24.380 --> 38:28.780]  if the ability to collect large data sets or train these very large expensive models
[38:28.780 --> 38:34.440]  were somehow democratized, would that improve the situation? Ram, I think we should throw this over
[38:34.440 --> 38:41.780]  to you, because I know you've recently written a paper on this topic. Yeah, that's a really
[38:42.300 --> 38:50.480]  hard question for me. You know, just because, you know, again, from, from a, from a, from a
[38:50.480 --> 38:56.940]  winner sense, like, is machine learning inherently authoritarian, you know, because like, or is it
[38:56.940 --> 39:02.880]  inherently democratic? And, you know, there, there are arguments to be made on both sides,
[39:02.880 --> 39:09.900]  like, you know, like Anima started off with data compute, and the algorithms, that knowledge is
[39:09.900 --> 39:20.460]  kind of centralized in a lot of, you know, in a lot of aspects, right? You, you know,
[39:20.460 --> 39:27.780]  like, $10 million or so with their TPU training pods, which, you know, like a mom and pop shop,
[39:27.780 --> 39:33.420]  like an individual researcher is not going to have. And data, as we all know, like Catherine
[39:33.420 --> 39:41.180]  kind of like outlines in her book beautifully, like how it is really, it's situated with power,
[39:41.180 --> 39:46.400]  almost. So, you know, in a, in a, in a, in a land and winner sense, it really does,
[39:46.400 --> 39:51.280]  you know, one may almost say that, you know, machine learning is authoritarian. On the other
[39:51.280 --> 39:58.120]  hand, actually, one of the other papers that we read was Anima's Born Again Neural Nets paper,
[39:58.120 --> 40:05.200]  where she shows that, you know, a student can actually in certain like conditions outperform
[40:05.200 --> 40:11.020]  the teacher. And, and if you think about the revolution that transfer learning is bringing
[40:11.020 --> 40:16.740]  along, you know, you, a company can put out this like pre-trained model, and then you can just
[40:16.740 --> 40:24.640]  retrain the last layer for your like custom, like applications. So, and, and now with like,
[40:24.640 --> 40:30.540]  you know, systems like fast.ai and like, you know, they're bringing like machine learning,
[40:30.540 --> 40:34.820]  like two programmers, which again is not making it democratic. So it's almost like,
[40:34.820 --> 40:40.440]  you know, you kind of trying to install solar panels in your, in your house, as opposed to
[40:40.440 --> 40:47.140]  depending on like a nuclear power station. So I completely struggle with this question. I keep
[40:47.140 --> 40:54.000]  vacillating back and forth. And unfortunately I do not have like, I'm not, I'm not, I'm not like
[40:54.000 --> 40:59.600]  settled on an answer. And perhaps like other panelists will guide us, will guide us through
[40:59.600 --> 41:07.640]  this cave. Thank you Ram for bringing up the paper. I think to me, I, you know, I again pin my
[41:07.640 --> 41:13.560]  hope back on the algorithms, right? So I think we are just at the infancy in terms of algorithmic
[41:13.560 --> 41:19.640]  development for deep learning. We've been using the very basic algorithms. The question is,
[41:19.640 --> 41:25.380]  you know, where to go next? You know, I mentioned unsupervised learning earlier as a broad topic.
[41:25.380 --> 41:30.480]  I think within that, what we want is more controllability, right? Like for instance,
[41:30.480 --> 41:37.440]  generating text. If you could control what needs to be generated, then you can try, you know,
[41:37.440 --> 41:42.280]  head towards removing these biases, at least in what's generated, even if it's inherent in the
[41:42.280 --> 41:46.820]  model. And in fact, that's what we need in practical applications. You know, we don't want
[41:46.820 --> 41:52.380]  it to write weird poetry. We want it to maybe, right, give us knowledge facts or ultimately
[41:52.380 --> 41:58.420]  program, you know, do program synthesis and so on. And so that's something we've been working on.
[41:58.420 --> 42:04.380]  Same for text, same for images, right? Ultimately the GAN should be able to have controllable
[42:04.380 --> 42:10.940]  generation and it can control various attributes, right? And that way then it's really at the hands
[42:10.940 --> 42:17.500]  of the human on how to use these models. And currently we don't have good frameworks for that.
[42:17.500 --> 42:21.800]  And I think unsupervised learning is the key to enable that because you have to learn what are
[42:21.800 --> 42:27.500]  the hidden variations in data and then how to control and change that and be able to generate
[42:27.500 --> 42:34.740]  new examples. Yeah, so I think that's the direction to go forward. I completely agree
[42:34.740 --> 42:39.700]  with Anima, by the way. She has been mentioning unsupervised learning. I also think that the
[42:39.700 --> 42:45.800]  future is probably in unsupervised, self-supervised learning that is a bit maybe more intelligent,
[42:45.800 --> 42:50.820]  depends on how you define intelligent, because again, this is semantics and a vague term, but
[42:50.820 --> 42:56.460]  it seems like once we get there so that the system is able to figure out this universal
[42:56.460 --> 43:03.620]  learning pattern somehow, and then starts figuring out different kinds of tests by itself,
[43:03.620 --> 43:09.120]  we might be able to somehow incorporate some kind of awareness to the system. And one option would
[43:09.120 --> 43:14.220]  be a human in the loop, for example. And again, Anima mentioned this a few minutes ago, saying
[43:14.220 --> 43:21.480]  that human psychology and awareness plays a very important role here. Like, we are not able to
[43:21.480 --> 43:27.880]  agree on many of these questions, because we are dealing with ethical questions. Everyone has
[43:27.880 --> 43:35.320]  their own understanding, their own answer, and their own desired fairness notion. And we can't
[43:35.320 --> 43:40.920]  satisfy all of these at the same time. However, if we are able to bring some kind of awareness
[43:41.500 --> 43:51.460]  in these systems, then we might be able to change actions and decisions. Because,
[43:52.200 --> 43:59.980]  the evolution of how people are reacting to biases depends on how aware we are of our biases.
[43:59.980 --> 44:05.860]  It's not like people are trying to discriminate other groups. It's mostly caused by favoring the
[44:05.860 --> 44:11.580]  in-group. Like, there are no harms that are explicitly trying to be imposed upon other
[44:11.580 --> 44:17.880]  groups. But this unintentional bias ends up causing all of these harms. So, in an unsupervised way,
[44:17.880 --> 44:25.240]  we can figure out some kind of an efficient learning pattern, where we can also incorporate
[44:25.240 --> 44:30.700]  ethical mechanisms, then we will start getting there. But of course, all of these will require
[44:31.200 --> 44:36.480]  a lot of informed discussions from many different perspectives, and many different disciplines as
[44:36.480 --> 44:43.400]  well, because this is affecting all disciplines, all the sciences, and every social fabric.
[44:45.700 --> 44:50.940]  And yeah, maybe I'll just, I mean, I think the question of, you know, is machine learning
[44:50.940 --> 44:56.260]  authoritarian or not is a super interesting one. But ultimately, the way that we would answer it
[44:56.260 --> 45:03.060]  from a feminist perspective is, in whose hands is it? So, you know, so feminism, basically,
[45:03.060 --> 45:10.160]  what feminism brings is who questions. So, like, who is using it, for whom, to support whom,
[45:10.160 --> 45:16.740]  to oppress whom, and so on. And so, I mean, I think that, for me, is the,
[45:16.740 --> 45:20.640]  those are the questions to be asking when we're thinking about, like, democratizing AI. Like,
[45:20.640 --> 45:27.720]  I think it's lovely to think about making it easier to use models, democratizing access to
[45:27.720 --> 45:32.080]  data and computational power. Those are all great things as, like, a kind of infrastructure
[45:32.080 --> 45:39.460]  of democratization. But on their own, they won't create a democratic kind of usage of AI,
[45:39.460 --> 45:45.620]  because we also need to do the bridge to the groups that could really actually benefit from AI.
[45:45.620 --> 45:50.180]  And for me, like the, again, like I'm saying, like, I'm interested in sort of counter-hegemonic
[45:50.180 --> 45:54.940]  groups and thinking about, like, what are groups that actually hold power accountable? They're
[45:54.940 --> 46:01.180]  groups like lawyers and legal scholars, journalists, community organizations, social movements. Like,
[46:01.180 --> 46:08.200]  these are the folks who need to be kind of trained and brought to the table, if we're
[46:08.200 --> 46:12.480]  actually talking about a comprehensive effort to democratize AI. Because those are the folks that
[46:12.480 --> 46:17.280]  you want to understand, like, what are actually the creative, creative potentials. And there are
[46:17.280 --> 46:21.060]  many, like, some of the journalists that have started to discover the power of machine learning
[46:21.060 --> 46:28.500]  have been doing amazing things, like detecting, using image recognition to detect amber mining
[46:28.500 --> 46:32.820]  from satellite imagery in the Ukraine, for example. Like, you can do this, like, kind of amazing
[46:32.820 --> 46:37.720]  accountability work, but not every person is going to have that idea, because they haven't been
[46:37.720 --> 46:46.260]  exposed to working in this way. So, yeah. So, if we're talking about, you know, AI algorithms
[46:46.260 --> 46:51.100]  being developed by institutions of power, and the fact that, like, who is using algorithms matters a
[46:51.100 --> 46:57.080]  lot, is there a role in, like, adversarial AI? Adversarial AI is usually cast as a bad thing,
[46:57.080 --> 47:02.400]  because machine learning engineers do good, and those pesky evildoers are trying to ruin it. But
[47:02.400 --> 47:06.460]  if machine learning algorithms are being used for bad, you know, does that mean that adversarial AI
[47:06.460 --> 47:14.060]  techniques can be used for good? Oh, I love that. Totally. I totally think so, yes, because, like,
[47:14.060 --> 47:21.220]  there are so many reasons why groups or individuals would need to be protecting themselves, like,
[47:21.220 --> 47:26.580]  from, you know, we were just talking about Reddit, for example, you know, like, there's so many reasons
[47:26.580 --> 47:35.420]  why, you know, like, an act of, kind of, an adversarial act is not necessarily bad. Like,
[47:35.420 --> 47:44.860]  for example, I have a friend who, as a way of, kind of, fooling the just rampant data capture
[47:44.860 --> 47:51.240]  that's done by promotional ads and things in the browser, and so, like, the browser is always sending
[47:51.240 --> 47:59.400]  the IP address and whatnot, and what it does is, instead of, like, sending what it clicks on,
[47:59.400 --> 48:06.360]  the browser clicks on, it actually just sends every link on the page. So it's, like,
[48:06.360 --> 48:12.060]  hacking it through obfuscation, you know, and so it's, like, I think that's a brilliant technique
[48:12.060 --> 48:18.480]  for then preserving privacy. So you're obfuscating what the user is actually doing, and screwing
[48:18.480 --> 48:23.720]  with the ad systems or whatever. And so I think, especially in situations where power is not equal,
[48:23.720 --> 48:30.560]  which is every situation right now, like, we also need, like, good, like, white hat adversarial ML,
[48:30.560 --> 48:40.400]  basically. I love this, Catherine. I got so excited for the ad blocker research. I mean,
[48:41.000 --> 48:45.140]  right now, in the field of adversarial machine learning, the use of adversarial machine learning
[48:45.140 --> 48:51.160]  to enhance the experience of citizenship is still nascent, but, you know, there's fantastic work
[48:51.160 --> 48:56.800]  that's happening. I'm so excited for this. Bogdan Kulinich, I'm sorry if I'm pronouncing his last
[48:56.800 --> 49:02.060]  name incorrectly, developed this paper called POTS, which is Protective Optimization Techniques.
[49:02.060 --> 49:07.500]  We're into, kind of, like, empower. So what they do is, it's really cool, it turns out that,
[49:07.500 --> 49:14.820]  obviously, credit card scoring is biased against minority population, and they show, like, how
[49:14.820 --> 49:20.520]  you can poison the system tactically, like, do it, literally launch, like, a poisoning attack
[49:20.520 --> 49:28.160]  to, kind of, like, you know, make it better for the target population. There's really another
[49:28.160 --> 49:36.680]  cool work from some researchers who found that Facebook's ads platform is micro-targeting,
[49:36.680 --> 49:45.600]  and they, again, use adversarial ML to show, like, you know, faults in that. So that opens up, like,
[49:45.600 --> 49:51.650]  exactly, like, how Catherine says, white hat adversarial examples. Unfortunately,
[49:52.440 --> 49:59.560]  you know, it's not the dominant thinking. In fact, DARPA, when it released its
[50:00.360 --> 50:05.140]  guard challenge, which is, you know, securing against, like, AI challenge,
[50:05.140 --> 50:11.160]  they said, we need to protect ML system from attackers. And they do not define who attackers
[50:11.160 --> 50:18.920]  are. So that's... And I completely agree with Ram and Catherine. And I think there's also,
[50:18.920 --> 50:23.900]  right, because we don't know who are the attackers, I feel like there's also people
[50:23.900 --> 50:29.240]  are going in all kinds of weird directions with adversarial AI research, right? Because they don't
[50:29.240 --> 50:33.420]  know what kind of attack to, you know, and they're like, okay, maybe I'll do this norm bound or some
[50:33.420 --> 50:39.400]  other. And then it's like, okay, why does this matter? We don't know, right? So I think perceptual
[50:39.400 --> 50:44.960]  attacks for images are much more meaningful. I mean, you know, if those that kind of preserve the
[50:44.960 --> 50:52.240]  for humans, the perception, but, you know, change the statistics of data, right. And to me, I think
[50:52.240 --> 51:00.200]  the next step is to really have the regulatory agencies, you know, take on these tests. I mean,
[51:00.200 --> 51:04.520]  we have a long way to go. Because if you look at the NIST, which is the standards data sets for
[51:04.520 --> 51:10.940]  face recognition, that itself is heavily biased against minorities. But ultimately, the hope is
[51:10.940 --> 51:16.360]  regulatory agencies in each domain will be using these as tests, you know, especially for autonomous
[51:16.360 --> 51:22.580]  driving, this is a must, right? I mean, given the long tail, the only way to do is also have very
[51:22.580 --> 51:28.940]  heavy testing would need a lot of compute, but only then certify that this is roadworthy and
[51:28.940 --> 51:31.760]  otherwise be very careful about that.
[51:32.580 --> 51:38.300]  And Anima, to my understanding, NIST has been tasked to develop these AI standards. So they
[51:38.300 --> 51:43.820]  are especially working on those given the example with the face recognition or computer vision data
[51:43.820 --> 51:50.180]  sets they had as well. So that was a great point that you identified. And again, there are all of
[51:50.180 --> 51:56.960]  these privacy enhancing or fairness enhancing adversarial applications of machine learning,
[51:56.960 --> 52:02.700]  but they might not be very common right now. But the most simplistic version we can think of is
[52:02.700 --> 52:09.380]  synthetically generating data to oversample minority populations so that we can improve
[52:09.380 --> 52:15.700]  their representation. Because at least in real world statistics, it's well known that oversampling
[52:15.700 --> 52:23.700]  minority groups is a helpful approach. But at the same time, again, this calls for the need of some
[52:23.700 --> 52:30.580]  standardization in this process. And again, this ties back into the question of is only data bias
[52:30.580 --> 52:36.100]  or algorithms also biased? Why are we creating these algorithms? What are the goals of these
[52:36.100 --> 52:44.980]  algorithms? Who are the bias power holders implementing these algorithms? And again,
[52:44.980 --> 52:50.480]  so it really matters. And this might bring us to the Kant question here. Should we have this
[52:50.480 --> 52:56.900]  ontological approach? Or should we go with utilitarianism? And so on. So there are many
[52:56.900 --> 53:01.820]  interesting questions in this direction as well that we are starting to just scratch the surface
[53:01.820 --> 53:07.880]  nowadays, because it's becoming a very hot topic. So why are we even building these systems? Why are
[53:07.880 --> 53:14.380]  they going to empower at the end? Absolutely, I just want to quickly add to Alan's point on the
[53:14.380 --> 53:20.460]  oversampling for minorities. I think, yeah, if you don't have more data, I think that's a quick fix
[53:20.460 --> 53:25.520]  right? That's the starting point. But that itself can also create problems because GANs have mode
[53:25.520 --> 53:31.980]  collapse. So you just have very less diversity among, say, the minority class, right? For instance,
[53:31.980 --> 53:37.880]  in fact, it was shown when the images that are generated based on the celeb data set,
[53:37.880 --> 53:43.420]  the celebrity data set, black women tended to have all the makeup on, right? So it was and
[53:43.420 --> 53:49.260]  with these face recognition systems, when black women didn't have makeup, it was even worse.
[53:49.260 --> 53:55.240]  You know, so then there is kind of further bias even within the class because of the lack of
[53:55.240 --> 54:00.480]  diversity and variation. And that's why I think we need to disentangle all these factors and do
[54:00.480 --> 54:05.960]  unsupervised learning and learn, is there also enough diversity within the class in terms of
[54:05.960 --> 54:13.740]  these variations? Yeah, go ahead Stella, sorry. So that sort of feeds into something that someone
[54:13.740 --> 54:19.360]  in the Slack asked about as well. And it's been in the academic discussion of AI bias a lot
[54:19.360 --> 54:28.520]  recently, which is how models can exacerbate or create biases out of imperfections. Even things
[54:28.520 --> 54:33.820]  that aren't normally thought of as imperfections, just the choice of your loss function implicitly
[54:33.820 --> 54:40.520]  decides how you're going to weigh different samples. And this is crude, but if 90% of the
[54:40.520 --> 54:45.720]  people in a data set are white and 10% of the people in the data set are black, then an algorithm
[54:45.720 --> 54:51.120]  that gets a little gain in performance at the cost of a little loss on a white person, at the cost of
[54:51.260 --> 54:57.360]  a little loss on a black person, is going to perform better on average, if it just mostly
[54:57.360 --> 55:05.900]  pays attention to white people. So, you know, lost the rest of my train of thought.
[55:07.220 --> 55:12.340]  How, so it's not, it's not just that when we talk about bias in algorithms, we're not just
[55:12.340 --> 55:17.320]  talking about the data coming in, but kind of algorithms can create their own sorts of little
[55:17.320 --> 55:24.340]  biases. Like, what does it look like to combat that? And kind of how does that happen? You know,
[55:24.340 --> 55:27.680]  from the perspective of a data scientist who's looking at something, it's like, you know, how do
[55:27.680 --> 55:33.860]  you know that's happening? And how do you combat that? Just a quick anecdote, because I don't have
[55:33.860 --> 55:40.940]  an answer to your question. A quick anecdote was one of the resume screening algorithms,
[55:40.940 --> 55:46.500]  it wasn't Amazon, but it was like a different company, when they reverse engineered it to see,
[55:46.500 --> 55:51.180]  like, why was it preferring men over women? One of the factors that they found, one of the strongest
[55:51.180 --> 55:56.320]  predictors was like, the name. So if somebody was named Jared, in particular, it was a very
[55:56.320 --> 56:02.400]  strong predictor of success, as well as if they were a lacrosse player or not on their resume.
[56:03.020 --> 56:08.860]  And so it's like, just a sort of very interesting thing, where, in a way, what happens there,
[56:08.860 --> 56:12.380]  like, they're not even they're considering, well, I don't know, I don't really know the model. But
[56:12.380 --> 56:17.900]  like, I don't think they were directly considering race or gender in that case. But if you look at
[56:17.900 --> 56:27.140]  those things, like Jared is a very white, like kind of Anglo name. Lacrosse is a very elite,
[56:27.140 --> 56:30.400]  white, sort of private school sport in the United States, despite the fact that it was
[56:30.400 --> 56:36.140]  invented by Native Americans, actually. And so like, those things sort of, they seep in,
[56:36.140 --> 56:41.520]  and then they become these factors, and they become predictive factors. But like, behind those
[56:41.520 --> 56:46.980]  is always like, is race and gender and all of those things are like, kind of still underlying.
[56:48.640 --> 56:54.920]  Yeah, these systems learn the associations, and you were both asking what kind of metrics we
[56:54.920 --> 56:59.680]  should, for example, develop to understand how to improve these systems. So I was wondering,
[56:59.680 --> 57:04.700]  maybe this exists in literature, but do we have, for example, diversity dispersion measurement
[57:04.700 --> 57:10.700]  metrics to understand how diverse a group is, so that we can try to improve diversity within one
[57:10.700 --> 57:16.940]  social group? And then the other thing is, Stella, you mentioned objective functions,
[57:16.940 --> 57:22.640]  or machine learning algorithms that are trying to optimize based on the data representation,
[57:22.640 --> 57:28.400]  where people are categorized into groups, racial groups, or gender. And it's not even clear how
[57:28.400 --> 57:34.220]  ethical this approach is. Should people be represented in categories? Is this the correct proxy
[57:34.220 --> 57:42.440]  to a person's identity or being? And how should we deal with system design when we are developing
[57:42.440 --> 57:48.360]  new machine learning algorithms? Because each algorithm depends on these data types, the way
[57:48.360 --> 57:53.980]  they are represented, numeric values, ordinal, or whatever. And should we come up with more creative
[57:53.980 --> 58:01.960]  ways to design new algorithms for machine learning to encounter, to counter balance,
[58:01.960 --> 58:09.280]  these problems? And Stella, I mean, I want to rewrite this question back to Catherine, because
[58:09.840 --> 58:16.400]  when I was reading a chapter from her book, Catherine, you spoke about concepts which
[58:16.400 --> 58:23.040]  maintain power, like fairness, bias, and ethics. And you give this amazing framework about concepts
[58:23.040 --> 58:29.880]  of challenge power. I didn't even think about that. Is that something you want to talk about?
[58:29.880 --> 58:34.780]  I feel it's a really cool framework for engineers to think beyond the fact star or fact framework
[58:34.780 --> 58:45.780]  that we're used to. Sure. Yeah. I think that's in chapter one or two. I can't remember. But
[58:46.400 --> 58:50.360]  yeah, I mean, so one of the things I think that Lauren and I were frustrated by
[58:50.360 --> 58:56.260]  is that we just felt like fairness, and this is not, I guess I should say a little bit of a
[58:56.260 --> 59:00.200]  disclaimer, because I actually think the work that's going on in the fact star community is
[59:00.200 --> 59:04.580]  super exciting. And so like, this is not to say that, like, no one should do that work, because
[59:04.580 --> 59:11.120]  I am really happy that work is being done. But that's just some of the organizing framing concepts
[59:11.120 --> 59:15.800]  for what we're talking about aren't, they don't go far enough. And so it's like, instead of fairness,
[59:15.800 --> 59:21.820]  we propose justice, or no, instead of ethics, we propose justice, like, we're really thinking
[59:21.820 --> 59:29.660]  about what does it mean to make a just system, which means repairing past harms, like harm has
[59:29.660 --> 59:35.000]  been done in the world, we come to the world in the bodies, in which harm has been inflicted at
[59:35.000 --> 59:40.280]  the group level, for some of us for centuries, you know, and so like, how do we build systems that
[59:40.280 --> 59:48.000]  not only kind of treat us all equally as if we're all the same, but actually take that harm into
[59:48.000 --> 59:54.200]  account. And so we propose other kinds of like, justice, concepts like...
[59:55.380 --> 59:57.420]  My favorite one is transgression.
[59:58.060 --> 01:00:01.920]  That's what I was gonna say. Yeah, like, kind of like, why settle for accountability,
[01:00:01.920 --> 01:00:06.500]  because often accountability is after the fact, you know, like, why can't we build
[01:00:06.500 --> 01:00:12.120]  systems that liberate us? Why can't we build systems that heal us? Like, I feel like we,
[01:00:12.820 --> 01:00:17.900]  we have the power to imagine just much more expansively. So like, it doesn't always have
[01:00:17.900 --> 01:00:22.820]  to be after the fact that we're trying to, like, you know, repair things or whatever, we can really,
[01:00:22.820 --> 01:00:28.500]  we can really aspire to do much more. So co-liberation means, it's this idea that
[01:00:28.500 --> 01:00:34.880]  none of us are free until all of us are free. So we work with the most marginalized groups
[01:00:34.880 --> 01:00:40.880]  and center, like, this is a case where we would like, build a data set around whoever is most
[01:00:40.880 --> 01:00:45.100]  marginalized in whatever the application area is, and we would work with them directly, and they
[01:00:45.100 --> 01:00:49.920]  would be part of the process. And whatever is built, the system that is built would like,
[01:00:49.920 --> 01:00:55.800]  actively serve their needs in a really culturally grounded way, you know. And so I think that's
[01:00:56.440 --> 01:00:59.600]  totally feasible. It's just not the way that things are done right now.
[01:01:00.440 --> 01:01:06.360]  I fully agree. I think these are great points, right? Ultimately, we have to kind of rework the
[01:01:06.360 --> 01:01:12.320]  whole framework. But I think to begin with, even, right, like, the way these systems are now just
[01:01:12.320 --> 01:01:18.520]  kind of put out with almost no testing, right? And, and then only kind of somebody like, kind
[01:01:18.520 --> 01:01:23.080]  of, right, brings up the problem, usually a minority researcher, and then the company is
[01:01:23.080 --> 01:01:29.040]  doing everything to push back and use all the resources to discredit the researcher, I think,
[01:01:29.040 --> 01:01:34.380]  at least we should change that conversation. And I think that can happen, especially, you know,
[01:01:34.380 --> 01:01:41.200]  with the viewers of this panel, we need more hackers into these AI systems, go hack every AI
[01:01:41.200 --> 01:01:49.800]  system out there, reveal all the vulnerabilities, take it down, rebuild a new one.
[01:01:49.800 --> 01:01:54.440]  Yeah, you can have like a brigade of like algorithmic bias assessors.
[01:01:54.440 --> 01:01:58.300]  I mean, good trolls, good trolls.
[01:01:59.840 --> 01:02:04.520]  To kind of move this towards, you know, advice or takeaways from the audience, because
[01:02:04.520 --> 01:02:09.120]  this has been a question popping up in the chat a lot, you know, so you guys, you know,
[01:02:09.120 --> 01:02:13.620]  you guys are talking about how there's definitely steps that people can take to make progress in the
[01:02:13.620 --> 01:02:17.740]  way that they design their algorithms to be more inclusive and stuff like that. And one thing that
[01:02:17.740 --> 01:02:23.320]  kind of stood out to me is being implicitly present in a lot of what you guys said, but I don't think
[01:02:23.320 --> 01:02:27.380]  anyone said this explicitly, is that kind of the default view of AI is it's supposed to be
[01:02:27.380 --> 01:02:32.560]  descriptive, you take a description of the world as it was, and you put that through an algorithm,
[01:02:32.560 --> 01:02:37.820]  and the role of the algorithm is to make the future look like the past. And so in a very real sense,
[01:02:37.820 --> 01:02:43.360]  AI is intrinsically regressive. But at the same time, nobody, there's nobody saying you have to
[01:02:43.360 --> 01:02:48.900]  do that, you know, you can decide that you want to choose to value certain things more than others,
[01:02:48.900 --> 01:02:53.660]  and you want to kind of bring about certain impacts, and you can proscriptively put those
[01:02:53.660 --> 01:03:01.620]  into your algorithms. Personally, I have tried to advocate for this, and largely speaking,
[01:03:01.620 --> 01:03:05.780]  failed, because people are scandalized by the idea that you might tell the data how it's supposed to
[01:03:05.780 --> 01:03:13.880]  look. But you know, so if you're a, if you're a data scientist, or a machine learning research
[01:03:14.460 --> 01:03:21.400]  engineer, listening to this chat, you know, can, can, how can they proscriptively design their,
[01:03:21.400 --> 01:03:27.680]  their algorithms for equity, for justice, for liberation, for whatever all these very nice
[01:03:27.680 --> 01:03:33.980]  words are, that are better than bias, or fairness, I suppose. You know, what does that, what does that
[01:03:33.980 --> 01:03:43.440]  look like to design an algorithm in that fashion? I mean, I'll start, and then hopefully others can
[01:03:43.440 --> 01:03:48.840]  build. But like, I mean, I think what it would look like is to do, to really think about where,
[01:03:48.840 --> 01:03:54.160]  where what your work is doing is being applied, and on whom, right? And like, who you're making
[01:03:54.160 --> 01:03:59.580]  it for, like, who's your boss, and then like, who are ultimately like the end users, or who
[01:03:59.580 --> 01:04:03.440]  is kind of impacted, who are the decisions being made about because of the system,
[01:04:03.440 --> 01:04:12.320]  and to do basically like a power analysis of that system. And then think about what is,
[01:04:12.320 --> 01:04:18.740]  that application area, who has been systematically disenfranchised, and then work with the groups
[01:04:18.740 --> 01:04:23.600]  who are working towards a better future in that domain to co-create, like what the metrics should
[01:04:23.600 --> 01:04:28.500]  be for the future. You know, I mean, I think that we're not asking, I don't, I don't think it's like
[01:04:28.500 --> 01:04:32.620]  appropriate that like, like a machine learning engineer would like invent the metrics out of
[01:04:32.620 --> 01:04:36.420]  their head or something. You know what I mean? Like in a domain space, they may not know very
[01:04:36.420 --> 01:04:41.720]  much about, but I think it is feasible that like a company could work really well with a group of
[01:04:41.720 --> 01:04:46.860]  folks, say, working on prison abolition or something like that, and could craft like here
[01:04:46.860 --> 01:04:51.680]  are the metrics for what the world that we would like to see. And this is like the kind of outcomes
[01:04:51.680 --> 01:04:56.900]  that we want to see. And so that's, that would be my kind of generalized recommendation. But that
[01:04:56.900 --> 01:05:03.120]  said, and this is like one of the major tensions of this space, is that often people are building
[01:05:03.120 --> 01:05:07.820]  tools that are more generic, right? Like you're looking for generalizability, you're looking for
[01:05:07.820 --> 01:05:13.060]  abstraction and like that could be applied in a wide variety of different application domains.
[01:05:13.060 --> 01:05:18.420]  And that I think is a tension. I actually think that is like a major tension for AI doing good
[01:05:18.420 --> 01:05:23.920]  in the world and being culturally grounded and informed. I don't know how to resolve that. And
[01:05:23.920 --> 01:05:29.000]  I would love to hear what like the computer scientists and researchers have to say about
[01:05:29.000 --> 01:05:35.600]  if they see that tension as well or not. I fully agree. And I think to me that
[01:05:35.600 --> 01:05:40.940]  foundations of that goes back to the education, right? So computer science and engineering has
[01:05:40.940 --> 01:05:46.640]  had the military background, you know, it started as a way to get into the military and that's
[01:05:46.640 --> 01:05:51.920]  there today. Like, you know, we are told to keep our emotions aside. In fact, that was one of a lot
[01:05:51.920 --> 01:05:57.120]  of gaslighting on Twitter is about, oh, why are we emotionally charged about machine learning? Just
[01:05:57.120 --> 01:06:02.940]  focus on the math, you know? And so that's something that's pervasive in the educational
[01:06:02.940 --> 01:06:08.040]  system and then beyond in the companies, right? Like, you don't worry about this. You do, you know,
[01:06:08.040 --> 01:06:14.520]  you derive math. You just, you know, program. So you're asked to become a robot without any emotions.
[01:06:14.520 --> 01:06:21.120]  And I think that's just really dangerous, right? Because it's not every engineer who's the problem.
[01:06:21.120 --> 01:06:27.020]  I mean, there's absolutely a decision maker who is probably, right, kind of, you know, may or may
[01:06:27.020 --> 01:06:32.860]  not understand the implications, but it's kind of directing all this and leading to the pervasive
[01:06:32.860 --> 01:06:38.740]  bias and injustice. I think we have to start with education. That's where I would pin my hopes.
[01:06:38.740 --> 01:06:44.800]  For instance, at Caltech, we have a course on ethics for AI. I think every university is now
[01:06:44.800 --> 01:06:50.180]  doing that. Ultimately more humanistic education. You know, we are kind of, you know, I went to
[01:06:50.180 --> 01:06:55.700]  technical school. Now I'm in a technical school, right? The role of humanities there needs to be
[01:06:55.700 --> 01:06:59.640]  even further emphasized. You know, it's much better than when I went to school,
[01:06:59.640 --> 01:07:04.200]  but it's still a long way to go to integrate humanities and engineering together.
[01:07:04.600 --> 01:07:10.040]  I completely agree with Anima. I'm also teaching bias in AI. I think it needs to be part of the
[01:07:10.040 --> 01:07:16.040]  curriculum, not for computer scientists only, but also engineers, humanities, social scientists,
[01:07:16.040 --> 01:07:21.180]  probably as well, because data scientists or essentially anyone that is interested in
[01:07:21.180 --> 01:07:28.480]  computing is using these tools. So they need to understand that all of these tools exist within a
[01:07:28.480 --> 01:07:33.780]  context. They don't just exist in an abstract, generalizable way, as Catherine mentioned.
[01:07:33.780 --> 01:07:39.220]  It's correct that these tools or algorithms were developed so that they can be applied
[01:07:39.220 --> 01:07:44.640]  to hypothetical data sets so that the theory or foundations are presented. But then when
[01:07:44.640 --> 01:07:51.920]  we take social data and apply these tools for machine learning tests in the social domain,
[01:07:51.920 --> 01:07:56.940]  then we are dealing with all kinds of human complications. So people need to always keep
[01:07:56.940 --> 01:08:02.200]  in mind that these systems, yes, they might look objective and neutral, but they are not in the
[01:08:02.200 --> 01:08:07.920]  social context because they exist with all the problems or side effects that human decision
[01:08:07.920 --> 01:08:14.220]  making brings into the game. But moreover, AI also brings its own kinds of problems, such as
[01:08:14.220 --> 01:08:20.560]  new types of biases we are not even aware of yet. And these keep getting amplified in society. So
[01:08:20.560 --> 01:08:28.140]  before we start introducing more of those, we need to understand what we are bringing to the world
[01:08:28.140 --> 01:08:35.160]  and what we should be careful about, because all of these systems are deployed right now without
[01:08:35.160 --> 01:08:42.220]  understanding what side effects they might have or what the implications are on society,
[01:08:42.220 --> 01:08:47.980]  economy, democracy, and so on. And now we would need to deal with all of these consequences
[01:08:47.980 --> 01:08:57.960]  on top of all the other things. It seems like a major takeaway you guys all advocate for is to
[01:08:57.960 --> 01:09:04.120]  kind of broaden the scope of who gets included in the conversation about AI design, about AI data
[01:09:04.120 --> 01:09:09.800]  collection, you know, bringing in activists in the relevant spaces, bringing in other stakeholders.
[01:09:10.880 --> 01:09:16.440]  And, you know, one thing that I would like to add to that is that the notion of stakeholder
[01:09:16.440 --> 01:09:24.940]  is really politically fraught. In industry context, stakeholders are generally conceived of as
[01:09:25.480 --> 01:09:32.780]  like, you know, so I suppose I'm at a ML company developing an algorithm that I'm going to sell to
[01:09:32.780 --> 01:09:37.620]  another company to use for financial profits. You know, the stakeholders in this process are
[01:09:37.620 --> 01:09:44.780]  generally conceived as being me and my team, my superiors at my company, the people I'm selling
[01:09:44.780 --> 01:09:50.380]  the algorithm to, and the stockholders of their company. And then I think that misses a really
[01:09:50.380 --> 01:09:55.140]  crucial point, which is the stakeholders are the people whose data is going to be analyzed.
[01:09:55.600 --> 01:10:01.400]  And their voice is very, very frequently just completely left out of any discussion about how
[01:10:01.400 --> 01:10:05.780]  things are designed, why things are designed, and where they come from. And I mean, to an extent,
[01:10:06.240 --> 01:10:11.420]  partially by design, because a lot of companies exist out of the fact, because of the fact that
[01:10:11.420 --> 01:10:16.520]  they can make money analyzing user data, like that's their sole reason, that's their reason
[01:10:16.520 --> 01:10:23.280]  for existing. There are lots of these things. But broadening, I think broadening the notion of a
[01:10:23.280 --> 01:10:27.800]  stakeholder, in particular, viewing anyone whose data is going to be processed by an AI as a
[01:10:27.800 --> 01:10:36.660]  stakeholder is really important. Yeah, I just I totally agree with that. We have a chapter in
[01:10:36.660 --> 01:10:41.300]  our book, which is about the principle embrace pluralism, which is basically this idea that,
[01:10:41.300 --> 01:10:46.920]  you know, we create more complete knowledge, like the path to objectivity is not through
[01:10:47.460 --> 01:10:52.540]  individual genius, but it's through bringing multiple different perspectives to the table.
[01:10:52.820 --> 01:10:56.060]  But I agree with you, I think like the it's important to think about like, because we can
[01:10:56.060 --> 01:11:01.000]  say, oh, it was participatory, like that company with all the money and the other company with all
[01:11:01.000 --> 01:11:07.980]  the money. So I do think and this is because participation, I mean, there's this whole
[01:11:08.720 --> 01:11:13.960]  world, you know, participatory design and participatory methods. And so like, there's
[01:11:13.960 --> 01:11:18.280]  even folks have talked about participation washing, where, you know, you're like, oh,
[01:11:18.280 --> 01:11:21.980]  yeah, we did a, you know, this also happens in urban planning, like, oh, yeah, we did a public
[01:11:21.980 --> 01:11:29.600]  meeting and it was at like 10pm and there was no childcare. So like, what's the quality and
[01:11:29.600 --> 01:11:34.700]  the character of the participation? Like who shows up? Who are these stakeholders? And who
[01:11:34.700 --> 01:11:39.680]  are the stakeholders that are not convenient to have? And how do you make folks listen to them?
[01:11:39.680 --> 01:11:44.120]  I mean, I think that's a big, that's a big question when ultimately, like the goal is to
[01:11:44.120 --> 01:11:52.390]  optimize profit. That to me is a conundrum. I think how do you make people listen, I think
[01:11:52.390 --> 01:11:59.010]  is a really interesting question. You know, a lot of people, I imagine a lot of people feel like
[01:11:59.010 --> 01:12:05.350]  they would like to do something good, you know, they would like to be more just and do their
[01:12:05.350 --> 01:12:11.550]  design the right way. But you know, how do you convince your boss or your boss's boss who
[01:12:11.550 --> 01:12:16.790]  controls the money and the funding, or the money and the decision making power, and who's maybe
[01:12:16.790 --> 01:12:21.610]  more interested in what's profitable? You know, how do you do any of you guys have advice about
[01:12:21.610 --> 01:12:28.410]  how to approach that conversation? Or how to how to advocate for justice in AI design,
[01:12:28.410 --> 01:12:37.330]  in context where that input is not necessarily the primary concern of the people who pull the strings?
[01:12:38.530 --> 01:12:45.050]  I'm interested in that as a research question, if we can figure out a way to simulate AI systems to
[01:12:45.050 --> 01:12:51.610]  show their long term gains and losses for all the parties that are involved,
[01:12:51.610 --> 01:13:00.950]  I think we would be able to understand how this affects economy in very from various perspectives.
[01:13:00.950 --> 01:13:04.930]  But again, this is a hypothetical research question. I'm sure there's already some great
[01:13:04.930 --> 01:13:13.670]  research on this yet. But maybe long term goals might be able to convince people that are profit
[01:13:13.670 --> 01:13:22.230]  driven as well. I mean, especially if there's concern, I really like the idea a lot, especially
[01:13:22.230 --> 01:13:29.310]  if there's concern, I mean, as there's this growing sort of industry of sort of impact assessment,
[01:13:29.310 --> 01:13:35.230]  or algorithmic bias assessment, and things like this, this is like, bringing folks in and doing
[01:13:35.350 --> 01:13:41.190]  a more participatory design process is a preventative for what could happen later,
[01:13:41.190 --> 01:13:46.390]  and so I think maybe that's the, like, if the case needs to be economic rather than moral,
[01:13:46.390 --> 01:13:49.990]  I mean, you can make the moral case, but probably what that listens to is the economic case, right?
[01:13:50.010 --> 01:13:55.450]  But if the case needs to be economic, I think that, you know, it would be lovely to have some
[01:13:55.450 --> 01:14:00.490]  quantifiable research that you could cite to say that like, yeah, there's this like,
[01:14:00.490 --> 01:14:06.230]  very strong economic case that this will reduce the number of lawsuits later,
[01:14:06.230 --> 01:14:10.750]  that you'll be fully compliant with all federal regulations, but you know what I mean? Like this,
[01:14:10.750 --> 01:14:17.750]  these kinds of things, I think that that might be the path for convincing the employers,
[01:14:17.750 --> 01:14:20.930]  and like, ultimately, like, you've invested in prevention,
[01:14:20.930 --> 01:14:24.410]  so that you don't have to clean up a big mess after the fact.
[01:14:27.970 --> 01:14:32.830]  That's why academia and the independent researchers have such an important role,
[01:14:32.830 --> 01:14:37.630]  right? Because they can talk openly about these issues, like, you know, what happened with the
[01:14:37.630 --> 01:14:42.810]  studies, right? Even though internally, there was a lot of conversation at Amazon,
[01:14:42.810 --> 01:14:49.250]  when I was there, you know, having that external light into it, and then the wider public part
[01:14:49.250 --> 01:14:54.170]  spreading in it, I think is critical. Because, you know, like being in these companies, it's
[01:14:54.170 --> 01:14:59.450]  just the whole system is so short term driven, right? The Wall Street expects quarterly results,
[01:14:59.450 --> 01:15:05.030]  that's not delivered, the company won't even exist, right? So the whole system is not built
[01:15:05.030 --> 01:15:10.070]  for the long term implications. And I think that's what creates these challenges. And so
[01:15:10.070 --> 01:15:16.130]  somehow, maybe then having more participation from the public, right, can show that also,
[01:15:16.130 --> 01:15:21.690]  you know, bad AI can lead to bad PR and loss of profits, even in the short term,
[01:15:21.690 --> 01:15:26.090]  I think that activism and awareness is critical to change this.
[01:15:28.630 --> 01:15:30.210]  Ram, you want to jump in?
[01:15:30.210 --> 01:15:39.730]  Yeah, I was just framing about what Anima said. But the way I think about this is, at least again,
[01:15:39.730 --> 01:15:46.230]  through lived experience, I can say that what has changed for me is that if you think of like
[01:15:46.230 --> 01:15:53.090]  inclusion, as after the fact, that's kind of like really weird. One of the things that I'm very
[01:15:53.090 --> 01:15:59.130]  lucky is that inclusion is part of my core priorities that I set. Like, it's just not for me,
[01:15:59.130 --> 01:16:04.010]  it's for everybody in the company. Like, you've got to set that, what is your inclusion priority?
[01:16:04.010 --> 01:16:07.710]  For me, it might be like, you know, I'm going to take care of Pride Month, like, you know,
[01:16:07.710 --> 01:16:12.130]  talks or, you know, whatever things, right? So obviously, it doesn't solve all problems.
[01:16:12.130 --> 01:16:18.890]  So first of all, I fundamentally believe that if your organization is questioning
[01:16:18.890 --> 01:16:24.370]  why you're doing something ethical, that itself is a big red flag for me.
[01:16:24.990 --> 01:16:31.330]  Right, like, and for the business case, I do agree, like, you know, there's this, like, you know,
[01:16:31.330 --> 01:16:36.190]  how Catherine said, there could be some financial motivation. I saw like an HPR article about like,
[01:16:36.190 --> 01:16:42.810]  how diversity leads to more surplus, because you're selling your product to more people.
[01:16:43.030 --> 01:16:49.350]  That could be like an other way. The PR angle, like, I mean, everybody is afraid of like bad
[01:16:49.350 --> 01:16:54.350]  marketing. I'd never actually thought about that. So thanks for bringing that up. But for me,
[01:16:54.350 --> 01:16:59.750]  intrinsically, you know, I have never been told you can't publish this. I mean, but again,
[01:16:59.750 --> 01:17:04.650]  it's a very different setting, like MSR. And like, you know, product setting is very different.
[01:17:05.150 --> 01:17:13.670]  But yeah, I would be very uncomfortable if I have to make a case to my manager as to why I'm doing
[01:17:13.670 --> 01:17:19.490]  something ethical in the first place. But like Anima pointed out, like, business, I've only
[01:17:19.490 --> 01:17:25.010]  worked in one organization for like, my working life. I do not know how others operate. I know
[01:17:25.010 --> 01:17:30.430]  some of the CEOs out there question why they should use pronouns. So or like, why they should
[01:17:30.430 --> 01:17:36.150]  like, you know, why they do not like fundamentally care about democracy and just want to dismantle
[01:17:36.150 --> 01:17:40.790]  it. So I don't know, I do not know how things work there from my lived experience. I don't know if
[01:17:40.790 --> 01:17:47.960]  this might actually be helpful for others. All right, so it seems like the kind of two
[01:17:47.960 --> 01:17:52.600]  major points you guys have raised is that you can view it as a risk mitigation factor.
[01:17:53.600 --> 01:17:57.600]  Because it will stop things from it can stop things from going wrong in the future. And it's
[01:17:57.600 --> 01:18:03.240]  in some sense insurance on that. And it's also a PR thing, which is partially a risk mitigation
[01:18:03.240 --> 01:18:09.360]  factor. You really don't want the New York Times running an article saying so and so's company
[01:18:09.360 --> 01:18:14.720]  develops algorithms systematically discriminates against women in hiring. Oh, I actually saw
[01:18:14.980 --> 01:18:20.840]  a headline. Yeah. Oh, sorry. Yeah. I actually saw an email. I only saw it through your tweet
[01:18:20.840 --> 01:18:35.160]  about gentrify or genderify. Yeah, another thing after all the right outcry got withdrawn,
[01:18:35.160 --> 01:18:41.160]  right? The fact that it even got launched is just appalling the kind of lack of awareness
[01:18:41.760 --> 01:18:47.240]  these companies. Yeah, it was shocking. You know, there was another company.
[01:18:49.100 --> 01:18:55.060]  I don't know who alerted me to it. But essentially, like their solution for more representation
[01:18:55.060 --> 01:19:03.900]  on stock photos was to kind of like artificially generate like photos using GANs. So you will not
[01:19:03.900 --> 01:19:09.980]  actually pay money to hire a black model. Oh, let's just use StyleGAN and like to kind of like,
[01:19:09.980 --> 01:19:14.960]  like, how do you get through? Like, who's funding this and goes back to an email like
[01:19:14.960 --> 01:19:21.240]  initial comment about VCs getting excited. So that I feel is a bigger like thing. I would be
[01:19:21.240 --> 01:19:26.580]  very less less worried about managers and more worried about like plutocratic like Silicon
[01:19:26.580 --> 01:19:37.100]  Valley folks to then like actually managers. So to kind of contextualize this for people
[01:19:37.100 --> 01:19:44.100]  in the audience who aren't aware, Genderify is a product that existed for all of about four hours.
[01:19:46.900 --> 01:19:52.120]  What their business model was, was that you can send them user data,
[01:19:52.120 --> 01:19:56.960]  you can send them photos, you can send them people's names, and they will
[01:20:00.440 --> 01:20:04.040]  they say they will determine the gender of the people that you
[01:20:04.940 --> 01:20:10.460]  in that you sell to or the people that are in your data set with an aim of doing better
[01:20:10.460 --> 01:20:16.220]  predictive marketing and stuff based on the... I'm having trouble coming up with a word other
[01:20:16.220 --> 01:20:25.380]  than assumed gender of the of the people in your data set. And this was this was funded by some VCs
[01:20:25.380 --> 01:20:32.180]  and it went online and everybody including lots of people in the ML space who don't really care
[01:20:32.180 --> 01:20:37.000]  very much, who don't talk very much publicly about about ML bias lost their shit because it
[01:20:37.000 --> 01:20:46.100]  was a completely absurd idea. And just to give an example, right, the word professor was only 1.9%
[01:20:46.100 --> 01:20:52.580]  females. So that's the assigned percentage for me in terms of gender. So I think that that was
[01:20:52.580 --> 01:20:57.300]  the problematic thing. Like how could you just type any name, like any word, not just name,
[01:20:57.300 --> 01:21:04.140]  any word in the vocabulary and assign a gender to that? I mean, and only binary genders, right?
[01:21:04.140 --> 01:21:08.840]  It's just appalling the whole concept. Yeah, my friend was saying the only time that's allowed
[01:21:08.840 --> 01:21:14.180]  it's in French, any other language, like, you know, because, you know, nouns have gender.
[01:21:14.260 --> 01:21:21.280]  Any other time table doesn't have like a male or female. Yes. Anyway. Yeah, but I'll know that like
[01:21:21.280 --> 01:21:26.600]  there are like this product is insane. But there are other products like there's all these gender
[01:21:26.600 --> 01:21:33.760]  detection libraries out there, both for faces. So like facial detection, which the researcher
[01:21:33.760 --> 01:21:39.280]  Oz Keys has written this amazing paper called Misrecognition Machines, which is about how
[01:21:39.280 --> 01:21:46.300]  harmful face like gender detection of faces is for trans folks. But then there's also all sorts
[01:21:46.300 --> 01:21:54.400]  of libraries that infer gender from name using various kinds of like name databases. And like,
[01:21:54.400 --> 01:21:59.080]  maybe I'll be like, sort of a naysayer here, because like, in general, I feel like these
[01:21:59.080 --> 01:22:05.500]  applications are very well, again, it depends on the application domain. So like, if you're doing
[01:22:05.500 --> 01:22:12.880]  this for predictive marketing, or like Oz talks about how they're doing it for bathroom access,
[01:22:12.880 --> 01:22:17.160]  which is like, disgusting, you know, like, are you the right gender to get into the bathroom or
[01:22:17.160 --> 01:22:22.840]  whatever? Those are super harmful and awful. And like those kind of applications shouldn't exist.
[01:22:22.840 --> 01:22:28.240]  At the same time, these gender detection libraries are helpful for doing like,
[01:22:29.320 --> 01:22:33.580]  bibliometric studies, where you're looking across like a huge corpus of say, like,
[01:22:33.580 --> 01:22:37.420]  I don't know, like several million authors and trying to see like, what's the
[01:22:37.960 --> 01:22:43.160]  over representation of men in that data set or something like that, you know, and so like,
[01:22:43.160 --> 01:22:47.300]  that that's why like, again, I think it's hard to just say categorically,
[01:22:47.300 --> 01:22:52.840]  hmm, something is bad or good. Although I do have certain things that I do think that way about,
[01:22:52.840 --> 01:23:00.360]  but like, that could enable really interesting research that monitors back on the patriarchy,
[01:23:00.360 --> 01:23:09.100]  for example. And so like, again, it's like, who's using it? For what? Yeah, like, that's what
[01:23:09.100 --> 01:23:14.140]  matters. And just to give an example, we have used the gender detector, but that was to learn
[01:23:14.140 --> 01:23:19.920]  about the MeToo conversation. So we, you know, collect lots of MeToo data from Twitter,
[01:23:19.920 --> 01:23:27.260]  men talking about it, how are women talking about it? How are Democrats, Republicans? So yeah,
[01:23:27.260 --> 01:23:32.780]  so that kind of, so you know, and this is in collaboration with Professor in Sociology,
[01:23:32.780 --> 01:23:39.020]  Mike Alvarez. So you know, like, the context certainly matters. We can't just ban all
[01:23:39.020 --> 01:23:43.800]  applications of gender detection, but really contextualize and look at whether it's for
[01:23:44.140 --> 01:23:50.240]  harm or for good. Yeah. And in medical applications, we especially need very accurate
[01:23:50.240 --> 01:23:56.880]  biased representations, because we know that for certain disease, the diagnosis and prognosis
[01:23:56.880 --> 01:24:04.020]  depends on ethnic factors, gender, and socioeconomic background as well. Of course,
[01:24:04.020 --> 01:24:09.820]  there are biases representing injustices in these medical data sets. But at the same time,
[01:24:09.820 --> 01:24:17.040]  there are biases that we need to learn from for more accurate medical treatment. And I guess
[01:24:17.040 --> 01:24:23.660]  that's why, for example, in the medical domain, machine decision making is becoming much more
[01:24:23.660 --> 01:24:30.260]  accurate than human decision making. But again, yet we need to deal with all the other disadvantages
[01:24:30.260 --> 01:24:36.600]  that come with this. So again, I guess everyone keeps mentioning this, it really depends on the
[01:24:36.600 --> 01:24:44.540]  context where we are using these things. And name-based gender detection is a kind of an
[01:24:44.540 --> 01:24:50.900]  outrageous example. But then there is a supercharged version of this in country X,
[01:24:50.900 --> 01:24:56.300]  where machine learning is used to predict the criminality of a person from their face,
[01:24:56.820 --> 01:25:03.640]  or actually give people social scores so that they can be sent to workers camps for their entire
[01:25:03.640 --> 01:25:12.380]  lives, because the machine said so. So there's a spectrum of these effects that we can discuss.
[01:25:12.380 --> 01:25:18.500]  And then what foundations should we be building on, given these examples that already exist in
[01:25:18.500 --> 01:25:26.120]  the world? Well, thanks for sharing that. I was just, I completely dismissed it,
[01:25:26.120 --> 01:25:32.660]  Catherine. And thank you, Anima and Eileen for, you know, correcting this. Thank you.
[01:25:33.180 --> 01:25:37.140]  I didn't mean to correct anything, by the way. Oh no, it was like, you know,
[01:25:37.140 --> 01:25:42.280]  I learned something. So thanks for telling me about this. Yes. I appreciate it.
[01:25:42.280 --> 01:25:48.780]  What did you learn, Ram? Well, I just completely dismissed like gender detection. I was like,
[01:25:48.780 --> 01:25:55.140]  oh, this is like, has literally no use. If I want to ask somebody's gender, I will just ask them if
[01:25:55.140 --> 01:26:02.360]  it's part of my business. I'll say it's none of my business. I'm not going to ask. But when,
[01:26:02.360 --> 01:26:07.120]  I was like, okay, that kind of makes sense. And when Eileen spoke about the medical contacts,
[01:26:07.120 --> 01:26:12.840]  that makes sense. So yeah, this was, like I said, I already learned something from this panel. And
[01:26:12.840 --> 01:26:17.320]  within like two more minutes for the panel to get over, I learned something. I learned a lot of
[01:26:17.320 --> 01:26:26.920]  things. Yeah, the example that Eileen brought up of criminality prediction is unfortunately
[01:26:27.560 --> 01:26:33.980]  running rampant in the AI literature right now. I feel like there is a half dozen papers that
[01:26:33.980 --> 01:26:37.940]  have been published and then promptly retracted after a massive public outcry,
[01:26:37.940 --> 01:26:44.280]  because they're basically trying to like reinvent phrenology, but doing it with machine learning,
[01:26:44.280 --> 01:26:51.940]  and therefore it's good. There's a whole lot that can be said about this that could and actually
[01:26:51.940 --> 01:26:58.560]  almost was its own panel at DEF CON. But there's a really good paper on this called
[01:26:59.980 --> 01:27:05.880]  Criminality from Facial Illusion, which I'll send out on Discord and on Twitch to people interested
[01:27:05.880 --> 01:27:11.300]  in reading about this problem that does a really good job of picking apart a lot of the issues
[01:27:11.300 --> 01:27:17.700]  with both morally and also technologically, because a lot of these papers are fundamentally
[01:27:17.700 --> 01:27:26.880]  technologically unsound with these kinds of works. So we're approaching an hour and a half.
[01:27:27.540 --> 01:27:32.020]  This is the last event of the day. I'm enjoying talking to you guys a lot. I don't know if you
[01:27:32.020 --> 01:27:38.680]  have somewhere you need to be, but I understand that. But if you guys are interested in continuing
[01:27:38.680 --> 01:27:45.480]  to talk, I am more than happy to do so. I have to go make dinner for my family.
[01:27:49.740 --> 01:27:54.440]  And it's actually sunny for a very short period of time in Seattle.
[01:27:56.360 --> 01:28:04.280]  Trying to sneak in like a run. Stella, before we all disperse, I first want to thank you
[01:28:04.280 --> 01:28:10.860]  so much for organizing this. You know, you have put so much thought and intention into this. Like,
[01:28:10.860 --> 01:28:15.220]  I remember like asking you like, Hey, I don't want to be the male in this. Like,
[01:28:15.220 --> 01:28:20.760]  have you thought about inclusive panel? I don't want to be a male. You wrote like,
[01:28:20.760 --> 01:28:27.340]  perhaps the most comprehensive male, you know, about this. So thank you so much for, you know,
[01:28:27.340 --> 01:28:33.260]  moving with intention, Stella. It clearly shows when you organize this panel. Thank you.
[01:28:34.680 --> 01:28:35.860]  Thank you. Second.
[01:28:35.860 --> 01:28:41.420]  Thank you. This is amazing. I learned so much and it's great to connect with all of you.
[01:28:41.420 --> 01:28:42.540]  And thanks, Stella.
[01:28:42.540 --> 01:28:47.520]  I enjoyed discussing with all of you. I think this was productive. And now I have more research
[01:28:47.520 --> 01:28:55.420]  ideas that I can go on. So quickly before you guys run, I want to take advantage of the fact
[01:28:55.420 --> 01:28:59.220]  that I have several experts here to get your thoughts on something I've been working on
[01:28:59.220 --> 01:29:05.640]  recently, which we've, we've almost touched on this several times. And it's that a lot of times
[01:29:05.640 --> 01:29:11.140]  machine learning algorithms have a underlying ontology. So Catherine mentioned, you know,
[01:29:11.140 --> 01:29:16.380]  that some gender algorithms exclude transgender people, you know, but even when you're talking
[01:29:16.380 --> 01:29:21.440]  about like all sorts of other applications you know, I work a lot with U.S. government data
[01:29:21.440 --> 01:29:25.120]  scientists. And so they're working with data sets that come from the U.S. census. And the U.S.
[01:29:25.120 --> 01:29:31.800]  census is a very specific notion of what races, and there isn't an option to use anything else
[01:29:31.800 --> 01:29:35.600]  because you're, you know, you're the U.S. government and you're using U.S. census data.
[01:29:36.260 --> 01:29:41.140]  But kind of how you, how you define your categories, you know, what, how you choose who
[01:29:41.140 --> 01:29:48.880]  gets labeled as white, who gets labeled as Asian, really makes a big impact on what your algorithm
[01:29:48.880 --> 01:29:53.300]  does and how it behaves. And whether you include certain categories, you know, if you include
[01:29:53.300 --> 01:29:59.320]  transgender people or not. I mean, if transgender, if non-binary transgender people, people who don't
[01:29:59.320 --> 01:30:07.120]  identify as, as, as a man or as a woman is not a possible label, then you have a guaranteed 100%
[01:30:07.120 --> 01:30:14.500]  incorrect assignment rate for them. So, you know, do you have any, any thoughts or advice about how
[01:30:14.500 --> 01:30:19.280]  to kind of tackle, you know, how to, how to approach this problem? Obviously, originally,
[01:30:19.280 --> 01:30:24.740]  we can go and talk to the impacted communities and try to figure out what the right buckets are.
[01:30:24.740 --> 01:30:30.720]  But oftentimes we start from a position where our data has been pre-collected for us by someone else.
[01:30:30.720 --> 01:30:35.820]  And so if you have data that has this kind of problem in the way the questions were asked
[01:30:35.820 --> 01:30:42.980]  originally, you know, is there something you can do with it still? I think that's a great question,
[01:30:42.980 --> 01:30:47.440]  right? And a difficult one because, you know, the systems are already built. Like you were saying,
[01:30:47.440 --> 01:30:54.180]  the categories are there. This is the label data. So I think towards getting to solutions for this,
[01:30:54.180 --> 01:31:00.180]  I think an important step is being able to trust the system in terms of confidence levels,
[01:31:00.180 --> 01:31:06.200]  right? Like, can the system say truly how confident it is? And the standard neural networks
[01:31:06.200 --> 01:31:12.640]  don't have this property. So it can make high confidence errors. You know, it can be like 95%
[01:31:12.640 --> 01:31:19.520]  confident and be wrong, right? And that especially happens on these kind of hard examples, which
[01:31:19.520 --> 01:31:24.760]  hasn't seen in training data because it also gets the calibration wrong. In one of our papers,
[01:31:24.760 --> 01:31:31.380]  we call it angular visual hardness. We proposed a new angular measure for detecting these hard
[01:31:31.380 --> 01:31:36.820]  examples that was more robust than the standard accuracy scores and the confidence scores
[01:31:37.420 --> 01:31:42.400]  that the systems are built on. And we also have some mathematical intuitions for that.
[01:31:42.400 --> 01:31:49.340]  But the outcome of this is we can now look at, you know, examples which are hard for the
[01:31:49.340 --> 01:31:54.620]  current classifier, right? So in this gender detection example, it would be most likely the transgender
[01:31:54.620 --> 01:31:59.640]  people who would fall as hard examples because it's not getting this accurate, right? No matter
[01:31:59.640 --> 01:32:06.900]  what binary gender the person is assigned to and if they don't kind of, you know, display that
[01:32:07.380 --> 01:32:13.780]  gender, right? So there is that issue that can be detected through these mathematical means.
[01:32:13.840 --> 01:32:18.640]  And then I think at least there is now the problem that has come up, right? That not every
[01:32:18.640 --> 01:32:25.080]  example is equally treated and there are hard examples for this classifier. We can go back and
[01:32:25.080 --> 01:32:33.040]  then design a system to correct that. And on a related note, I will give a vague answer to this
[01:32:33.040 --> 01:32:41.680]  before I actually ask an even more daunting or deeper question. Maybe an email, for example,
[01:32:41.680 --> 01:32:47.460]  idea of using more unsupervised approaches might be some kind of a solution for future generations
[01:32:47.460 --> 01:32:53.160]  of AI algorithms. Maybe you wouldn't need to categorize yourself as a person, but instead some
[01:32:53.160 --> 01:33:00.720]  kind of more unsupervised continuous representation of social entities might be a way to approach this
[01:33:00.720 --> 01:33:06.480]  problem. This is not going to happen anytime soon, so I don't really have an answer for today's
[01:33:06.480 --> 01:33:12.520]  problems because we know that gender is reflected to other features, all other features as proxies
[01:33:12.520 --> 01:33:18.740]  as well. Because when we think about gender, that's the largest social group in society. So
[01:33:18.740 --> 01:33:26.160]  it's one of the biggest proxies in the data sets. But my more concerning question is semantics,
[01:33:26.160 --> 01:33:33.040]  gender, and race is defined in categories. So it's a proxy, but when you think about it,
[01:33:33.040 --> 01:33:41.180]  essentially race doesn't really exist or gender might not even exist, but we have it in our
[01:33:41.180 --> 01:33:47.820]  language, in our linguistic system, and that's the way we are constructed to deal with the world. So
[01:33:47.820 --> 01:33:54.260]  how are we going to change these problems that are social constructs in AI systems that we are
[01:33:54.260 --> 01:33:59.980]  building on these social constructs? So it's a longer-term problem that might require longer-term
[01:33:59.980 --> 01:34:09.400]  solutions, which might not be very helpful for your question. Yeah, that really gets us something
[01:34:09.400 --> 01:34:15.440]  unfortunate, and a lot of people in the Discord are asking applied questions that I haven't raised
[01:34:15.440 --> 01:34:19.740]  because I know that the answer is, we don't know how to do that. We would love to know how to do
[01:34:19.740 --> 01:34:28.280]  that too. So one example question is, if you can't bring in experts or social scientists or
[01:34:28.280 --> 01:34:34.080]  stakeholders, is there an algorithmic way to kind of pick up on the bias? And the answer is, well,
[01:34:34.080 --> 01:34:39.100]  there's 50 and we don't know which ones work better in different contexts. I mean, but there
[01:34:39.100 --> 01:34:45.360]  is an algorithmic way to educate yourself, and that's like search engines, right? That's what?
[01:34:45.560 --> 01:34:49.400]  That's like, that could be search engines, just like, you know, go type, like, you know,
[01:34:49.400 --> 01:34:54.680]  if you want to learn about bias, go read Aileen's papers. If you want to learn about, like, you know,
[01:34:54.680 --> 01:35:01.600]  bias, go watch, like, Timnit Gupru's, like, workshops. So I think a lot of this hands-on
[01:35:01.600 --> 01:35:07.360]  things, people, smart people have already put out content, and it's just like a matter for us to
[01:35:07.360 --> 01:35:14.040]  spend, like, cycles to educate ourselves on. Yeah, and as a, like, very basic kind of check,
[01:35:14.040 --> 01:35:21.080]  like, you can look at the output of your system, if it has to do with people, and just look across
[01:35:21.660 --> 01:35:28.320]  intersectional groups. So, like, you do, like, assuming that you have gender data and race data,
[01:35:28.320 --> 01:35:33.160]  maybe you have income data, or maybe you have age, or maybe you have, like, some other kinds of,
[01:35:33.160 --> 01:35:41.040]  you know, dimensions that describe people, is just looking at disparate outcomes within each thing,
[01:35:41.040 --> 01:35:45.400]  and then combining those in various different ways to see if there are disparate outcomes
[01:35:45.400 --> 01:35:51.360]  amongst the categories. Because, like, the theory of intersectionality from black feminism
[01:35:51.360 --> 01:35:57.900]  says that, like, these categories compound in, like, a non-linear way, right? So being,
[01:35:57.900 --> 01:36:01.980]  experiencing sexism as a white woman is fundamentally very different from experiencing
[01:36:01.980 --> 01:36:08.400]  sexism and racism as a black woman. So, I mean, I think that's a very basic kind of self-audit
[01:36:08.400 --> 01:36:13.280]  that you can do when you're working with data that's about people, that, like, you don't need
[01:36:13.860 --> 01:36:17.460]  all that much, like, special training, necessarily. But, I mean, I think other people are working on
[01:36:17.460 --> 01:36:20.160]  this, right? I mean, I feel like you guys would know this better than me, but, like, folks have
[01:36:20.160 --> 01:36:26.220]  been working on things, like, where you would substitute out, like, you do, like, a sort of
[01:36:26.740 --> 01:36:30.680]  counterfactual, and you look at, well, what if this data point, like, they have this outcome,
[01:36:30.680 --> 01:36:34.560]  but what if they had, their race had been white, would they have had the same outcome with my
[01:36:34.560 --> 01:36:39.280]  system? So I think people are working on those kinds of methods. I think just a really basic
[01:36:39.280 --> 01:36:46.820]  one is just do a kind of an intersectional audit. Well, Stella, I, for one, look forward
[01:36:46.820 --> 01:36:52.920]  to reading your work when it comes out. Thank you so much for joining. Thank you guys so much
[01:36:52.920 --> 01:36:57.940]  for joining us. We've gone horribly over, but I've had a wonderful time talking to all of you,
[01:36:57.940 --> 01:37:04.200]  and I know that the people listening really enjoyed it. If anyone wants to continue this
[01:37:04.200 --> 01:37:10.640]  conversation who's listening, if you head over to the DEF CON Discord, there's a voice channel
[01:37:10.640 --> 01:37:15.500]  where I'm happy to continue to talk to anyone who wants to ask questions or get references to
[01:37:15.500 --> 01:37:20.780]  papers that people brought up and such. Thank you again to all of our wonderful panelists.
[01:37:20.780 --> 01:37:28.640]  This has been amazing. So thank you very much. Thank you. Thank you. Wonderful to talk with
[01:37:28.640 --> 01:37:35.920]  all of you. All right. Bye. I was also going to say we've also, we're going to have a journal
[01:37:35.920 --> 01:37:47.620]  club focused on ethics and privacy with AI starting on the 18th. It's organized by Suhaka
[01:37:48.920 --> 01:37:54.220]  on Thing, and you can check out our website for a link to a description of the whole thing.