WEBVTT 00:00.600 --> 00:01.820 All right, welcome. 00:01.820 --> 00:16.840 This is the AI Village panel on AI ethics, where we're going to discuss the ethical implications of AI technologies and especially their deployment both in general and also specifically in data science and computer security. 00:16.840 --> 00:22.820 We have four lovely panelists from a variety of backgrounds here to discuss the topic with us. 00:22.820 --> 00:31.140 Catherine is a HackerMOMA scholar and artist who focuses on feminist technology, data literacy, and civic engagement. 00:31.480 --> 00:40.900 She wrote a 2020 book called Data Feminism, co-authored with Lauren Klein, where she charts a course for more ethical and empowering data science practices. 00:41.160 --> 00:51.900 Catherine is an assistant professor of urban science and planning at the Department of Urban Studies and Planning at MIT, where she's also the director of their Data Plus Feminism Lab. 00:52.120 --> 00:59.700 Ram is a data cowboy within the trustworthy ML team at Microsoft, working on the intersection of machine learning and security. 00:59.700 --> 01:04.700 His team's goal is to empower engineers to develop and deploy ML systems securely. 01:04.740 --> 01:12.320 Ram is also an affiliate at the Berkman Klein Center at Harvard University, where he explores the societal implications of adversarial AI. 01:12.900 --> 01:22.180 Ai-Lin is an assistant professor of computer science at George Washington University and at the Institute for Data, Democracy, and Politics. 01:22.180 --> 01:30.940 She is interested in AI ethics, bias in AI, machine learning, and the implications of machine learning on privacy, security, fairness, and society. 01:30.940 --> 01:39.340 Much of her research is on methods to interpret AI and gain insight into the bias in both artificial and natural intelligence. 01:39.680 --> 01:49.860 And she has worked to demonstrate how human-like biases can crop up in natural language analysis due to the statistical regularities of language. 01:50.080 --> 01:56.860 And finally, Anima is the Bren Professor at Caltech and the Director of ML Research at NVIDIA. 01:56.860 --> 02:01.220 She was previously a principal scientist at Amazon Web Services. 02:03.040 --> 02:05.860 She's received several honors, such as the Alfred P. 02:05.860 --> 02:12.820 Salone Fellowship, NSF Career Award, Young Investigator Awards from DoD, and several more. 02:12.820 --> 02:23.880 She is part of the World Economic Forum's Expert Network and is passionate about designing principled AI algorithms and applying them to interdisciplinary outcomes. 02:23.880 --> 02:33.160 She has been outspoken on AI bias and the need for democratization and focuses her research on unsupervised AI optimization and tensor methods. 02:33.620 --> 02:42.540 They are all wonderful people and I can personally vouch for the fact that super cool research and I'm more than thrilled to talk with them all. 02:42.600 --> 02:53.900 So to kind of get us started, there's been a lot of conversation, at least recently, about AI bias and misuse of AI in the news in the United States recently. 02:53.900 --> 03:03.040 A lot of tech companies have talked about how they're going to stop using facial recognition algorithms or stop providing them to police to use. 03:03.040 --> 03:19.420 And there's also been a lot of conversation both in the academic literature and also in the national news media about biased AI, the AIs that learn to be racist because they look at racist data or learn to be sexist because they look at sexist data. 03:19.780 --> 03:22.600 But AI ethics is kind of a much broader topic than that. 03:22.600 --> 03:30.140 So to kick us off, can each of you say a little bit about what the scope of the field is to you and how it relates to the work that you do? 03:30.660 --> 03:32.800 Ram, why don't you give us a start? 03:33.040 --> 03:34.440 Oh, thank you. 03:34.440 --> 03:38.100 First of all, I'd like to say that I'm not an AI ethics expert. 03:38.100 --> 03:43.280 I'm so glad that Anima, Catherine, Eileen are here along with Stella. 03:43.460 --> 03:48.180 So you should be getting like solid like stellar advice from them. 03:48.180 --> 04:00.120 So whatever I'm going to talk is through my lived experience and interacting with folks like Kendra Albert, Jonathan Penny from Berkman and along with Mary Gray and a whole host of other folks. 04:00.120 --> 04:04.400 So that's my, you know, that's my preface for this. 04:04.520 --> 04:12.740 So for me, like I've been looking at Berkman at the, you know, what is adversarial ML? 04:12.740 --> 04:15.780 How does that impact the experience of citizenship? 04:16.040 --> 04:34.220 So one of the things that I, you know, for me, like how I, you know, how we've been interpreting this topic is moving away from just like an, you know, like the constraint of like ethics and thinking about power structures, especially in the context of adversarial machine learning. 04:34.220 --> 04:47.560 So turns out that, you know, machine learning is political in a Langdon Winner sense and security has a lot of like implications, like, you know, if you think of the moral character of cryptography by Rod Wave. 04:48.160 --> 04:55.600 And adversarial machine learning, which is what I work on as a researcher , sits in the intersection of these two topics. 04:55.600 --> 05:01.880 So thinking through who is securing like the ML systems? 05:01.880 --> 05:03.960 What is securing? 05:03.960 --> 05:05.200 Who are we securing it against? 05:05.200 --> 05:06.540 Who is the attacker? 05:06.540 --> 05:12.360 Are we, is the attacker just like anybody who compromises the confidentiality, integrity, and availability? 05:12.720 --> 05:16.980 So those are kind of the broader questions that I'm looking at at Berkman. 05:18.040 --> 05:19.780 All right, Ellen. 05:22.140 --> 05:24.800 Oh, Stella, thank you for the introduction. 05:24.800 --> 05:33.240 And I'm very excited to be part of this amazing panel so that I will get to hear all of your ideas and vision on this topic. 05:33.240 --> 05:46.660 My research focuses on, working on the implications of machine learning on privacy and security. 05:46.660 --> 05:57.240 Since machine learning or AI technologies are double-edged swords, when they have security enhancing properties, they usually end up infringing upon our privacy. 05:57.240 --> 06:13.040 And starting from there, I started slowly seeing the link between privacy and fairness and how when our data is exploited, abused, or collected by third parties, it can lead to fairness concerns at the group level. 06:13.100 --> 06:32.360 And nowadays I'm focusing on understanding how AI systems learn bias or essentially how systems can learn some form of intelligence or learning ability and how that relates to bias in natural and artificial intelligence. 06:32.360 --> 06:57.300 And when we looked at natural language systems, for example, word embeddings or today's neural language models that can generate text that is almost human level, we see that these systems are learning human-like biases from society very accurately, especially when compared to validated social psychological methods such as implicit association tests. 06:57.380 --> 07:07.420 And then these systems are deployed at the world scale impacting billions of people every single day with their consequential decisions or biased outputs. 07:07.420 --> 07:16.480 And this ends up turning into this AI bias lifecycle, which is also a feedback cycle of AI ethics as well. 07:16.480 --> 07:42.200 And I'm interested in understanding all of the components in this AI ethics lifecycle, the role humans play in this lifecycle, how we can discover biases or harmful side effects of these AI systems, and what we can do to mitigate these harmful effects and prepare for upcoming problems that might have already been caused by AI systems. 07:42.200 --> 07:56.820 And during today's panel, we will go into the details of all of these single components, starting from the most technical atomic details of machine learning, going up to probably some philosophical questions at the high level abstraction. 07:58.500 --> 08:00.340 Thank you Aylin. 08:00.520 --> 08:01.700 Anima? 08:01.900 --> 08:06.640 Yeah, I completely agree with Aylin and Ram. 08:06.640 --> 08:22.940 You know, it's to me exciting times to be an AI researcher, but also, right, I'm filled with dread that where the world is moving and how fast we are, you know, moving in directions that I find problematic. 08:22.940 --> 08:30.020 And to me, the source of it all comes to the foundations under which this deep learning revolution is built on, right? 08:30.020 --> 08:39.120 So I'd like to emphasize when I'm giving talks on AI about this trinity, you know, the data, algorithms, and computer infrastructure coming together. 08:39.120 --> 08:46.040 And that's what allowed us to have this fast progress in deep learning. 08:46.160 --> 08:50.040 But most of that progress has been with supervised learning, right? 08:50.040 --> 08:56.440 The availability of large-scale label data starting from the ImageNet data set was the start of this revolution. 08:56.660 --> 08:59.400 And in a way, that is quite straightforward. 08:59.400 --> 09:07.280 Once you know that you want to optimize for accuracy and you have the label data, you can kind of quickly build these systems. 09:07.280 --> 09:11.960 But the problem is sometimes we want not just the accuracy, but like robustness, right? 09:11.960 --> 09:16.720 Like what Ram was saying, what are the different changes beyond the training data? 09:16.720 --> 09:24.460 And when our training data is biased, how does this affect on minority populations that are not represented in training data? 09:24.460 --> 09:29.360 And part of it is not just the data imbalance, but also the algorithmic bias. 09:29.500 --> 09:38.480 You know, if you are on Twitter like me, and if you follow the Twitter storm on the presence of algorithmic bias on deep learning, you know what I'm talking about. 09:39.460 --> 09:48.840 And just to the gist of it, the idea is I think some proponents of deep learning do not want anything negative being said around deep learning. 09:48.840 --> 09:54.840 But I do think these very large-scale models, highly nonlinear models, are the source of the problem. 09:54.840 --> 10:01.220 Because the more nonlinear it gets, the more brittle they are, the more they amplify the bias, right? 10:01.220 --> 10:02.620 There is no free lunch. 10:02.620 --> 10:13.360 You know, you cannot have it all in one mechanism unless we are thoughtful in terms of the objectives we build in, the kind of data we use, and also the kind of testing we do. 10:13.520 --> 10:16.320 And so what I work on is unsupervised learning. 10:16.320 --> 10:22.660 And I think to me, this is the core for us to move away from these highly biased models. 10:22.660 --> 10:30.500 You know, to take a simple example, you know, even before the deep learning revolution, I was working on topic models, right? 10:30.500 --> 10:37.800 Where we wanted to categorize large corpus of documents and there's no label data. 10:37.820 --> 10:43.140 And so now with these unsupervised learning, we are suggesting these are possible topics, right? 10:43.140 --> 10:45.940 And these are the words representing the topics. 10:45.940 --> 10:53.720 And then it's up to the human to interpret and judge what are relevant topics, you know, because there are weird correlations in text. 10:53.720 --> 10:56.940 So there can be lots of topics that are not human interpretable. 10:56.940 --> 11:06.280 And so human in the loop with unsupervised learning, I think is the key to get out of this place where we are stuck with biased models. 11:06.400 --> 11:10.200 And, you know, the current revolution now is unsupervised. 11:10.200 --> 11:14.840 Now we're seeing a lot of progress in self-supervised learning and robustness. 11:14.840 --> 11:17.200 Hopefully that'll lead us the way forward. 11:17.360 --> 11:18.360 Thank you. 11:19.280 --> 11:20.040 All right. 11:20.340 --> 11:21.220 Thank you. 11:21.220 --> 11:25.900 And finally, Catherine, where are you coming from with this? 11:26.360 --> 11:27.040 Yeah. 11:27.040 --> 11:28.220 Thanks so much. 11:28.220 --> 11:29.900 Thanks, Stella, for the invitation. 11:29.900 --> 11:32.760 It's a super pleasure to be here with you all. 11:34.080 --> 11:34.380 Yeah. 11:34.380 --> 11:51.980 So I guess where I'm coming from is not as much of a focus on AI, although some of the things that we say, many of the things we say in our book, Data Feminism, apply, but we sort of enlarge the scope more broadly to be data writ large. 11:52.680 --> 12:05.220 And thinking about also, you know, one of the things that's happened as this conversation has grown around fairness, accountability, transparency, bias, ethics, and so on. 12:05.840 --> 12:11.400 You know, there's been a lot of pushback on just like AI and data are great. 12:11.400 --> 12:13.300 They're working great for everybody. 12:14.100 --> 12:31.580 And our contribution with Data Feminism is to say, hey, actually, feminism as a kind of body of knowledge, as a kind of intellectual heritage, which has theorized, philosophized, and also done a lot of writing and sort of action, like social movements, 12:31.580 --> 12:42.860 action around the topic of inequality has a lot to offer us in terms of conceptual models for how we can think about this current state of affairs. 12:42.860 --> 12:50.620 And so that was sort of the premise of Data Feminism is to say, hey, let's look over here at feminism. 12:50.620 --> 13:09.420 What do we, what can we bring from feminism to understand how we might work with data science in a more ethical way, and specifically taking structural inequality, which is basically like power differences between social groups as kind of our main locus of attention. 13:09.420 --> 13:24.660 And so when you look at that, and you look at things like, you know, one of the things I guess that was irritating, I guess I'll say as we were writing the book, which I'm sure folks here are also irritated by, is that often, you know, when we have a biased, 13:24.660 --> 13:37.780 whatever, algorithm or something like this, like a resume screener, let's say, that is favoring men over women, you know, the media reports on it with this thing of like surprise, like, oh my gosh, like, how could this be possible? 13:37.840 --> 13:41.240 Like, how could there be bias in a machine? 13:42.020 --> 13:46.540 And we're like, oh my gosh, Data Feminism has so much to say about this. 13:46.540 --> 13:51.380 And so ultimately, like what the book does is it kind of backs us up. 13:51.380 --> 13:53.500 It's in a way less in the details. 13:53.500 --> 13:59.820 It's not providing you with a way of like systematically fixing your data set or anything like that. 13:59.820 --> 14:07.240 But it backs us up, like way further back in the processing pipeline to say, well, like, here's why these things surface. 14:07.240 --> 14:09.720 They surface because of hundreds of years of inequality. 14:09.840 --> 14:18.800 They surface because of really flawed models of people like women not being considered fully human or people of color not being considered fully human. 14:18.800 --> 14:29.980 And to really take, you know, if we aspire to build something that is ethical, we have to look and we have to undertake that with a different kind of time horizon. 14:29.980 --> 14:32.940 And that has to be built into everything that we do. 14:32.940 --> 14:37.180 So we're trying to kind of both say like, hey, let's look back at the root causes. 14:37.180 --> 14:38.880 What causes structural inequality? 14:38.880 --> 14:46.360 Like, why do we end up with biased, you know, crime data sets in the first place before we start trying to like fix them or something? 14:46.760 --> 14:53.720 And then how do we use that information to decide which problems do we work on? 14:53.780 --> 14:55.260 Who do we work with? 14:55.260 --> 14:57.360 Who do we want to benefit with our work? 14:57.360 --> 15:00.660 And who do we want to just like refuse to work with entirely? 15:00.660 --> 15:03.940 Because there are some really bad actors out there. 15:04.180 --> 15:12.320 And so that's... and then we talk a lot about democratizing data, data literacy, and AI as well. 15:12.320 --> 15:17.660 Because that, I think, is really part of the key of how we're gonna get through this. 15:17.660 --> 15:25.140 So we're trying to balance being not only doom and gloom with like, how do we move forward? 15:25.140 --> 15:32.440 And we propose a set of seven principles for how we move forward with a kind of feminist approach to data science. 15:33.360 --> 15:34.780 All right, thank you. 15:34.780 --> 15:52.180 So I think a common thread in what all of y'all said is that there are problems both... problems in machine learning exist both in the pipeline process, where that is coming from, how it's processed, but also... but that that isn't all of the... 15:52.180 --> 15:53.800 all of the problems. 15:53.800 --> 16:07.220 A lot of the time in the media and in conversations, people have this idea that data is like some kind of objective truth, and that when you come up with biased results or incorrect results, it's because the analyst messed up somewhere. 16:07.800 --> 16:10.760 So, you know, do you think that's true? 16:10.760 --> 16:13.720 Spoiler alert, I'm pretty sure you're all going to say no. 16:14.960 --> 16:21.420 But, you know, what would you say to someone who thinks that way? 16:21.420 --> 16:24.980 And like, what is... what are the problems with thinking about data that way? 16:27.260 --> 16:30.140 Whoever wants to kind of get started. 16:30.600 --> 16:33.960 I can quickly start with a positive note. 16:34.060 --> 16:41.100 Data certainly reflects the state of the society, but it does it in a defective way. 16:41.100 --> 16:50.940 So we are not getting a perfect representation of society, especially with regards to minorities, because they are underrepresented. 16:50.940 --> 17:06.400 But when we have such data sets, at the same time, one advantage is that we can also study the historical injustices these minorities, underrepresented or disadvantaged groups have been going through, because it's reflected in the data set. 17:06.400 --> 17:23.620 As I already mentioned, since these data sets are a reflection of society, and then when we start building models on these systems, then we have data that is reflecting these biases. 17:23.620 --> 17:34.600 But at the same time, it also reflects some facts about the world, such as statistical biases, inductive biases, or some kind of facts that can also be useful. 17:34.600 --> 17:45.020 But that does not justify the fact that there are all these harmful information that gets very accurately embedded in these systems. 17:45.020 --> 17:53.000 So this is my double-edged view on this data topic for now. 17:53.000 --> 17:55.760 I can follow up with that. 17:55.760 --> 17:57.380 I completely agree with Arlene. 17:57.380 --> 18:06.620 I think the problem is how the data on the internet is reflecting the society, but only in very balanced ways. 18:06.620 --> 18:10.740 And this gets amplified when you want large data sets. 18:10.740 --> 18:15.620 The latest one in this aspect is the GPT-3. 18:15.960 --> 18:18.500 It's trained on links from Reddit. 18:18.640 --> 18:20.800 I'm almost never on Reddit. 18:20.800 --> 18:24.580 I've only been a victim of trolling and gun threats on Reddit. 18:24.580 --> 18:25.520 I stay away. 18:25.520 --> 18:30.420 I don't know any woman in machine learning that I'm aware of who is active on Reddit. 18:30.420 --> 18:38.320 So you can see that even among all the imbalance in the internet, the Reddit is even further amplified. 18:38.320 --> 18:59.600 So the decision to go for Reddit links and claim that's what we want to build the largest language model on, we can spend tens of millions of dollars on this data set and nothing else, is to me extremely disappointing and disheartening that the VCs, the decision makers, 18:59.600 --> 19:07.260 they're always going for the worst of the toxic forms of data that's available on the internet. 19:07.260 --> 19:08.920 And I really can't justify this. 19:08.920 --> 19:11.500 I don't know how they sleep every night. 19:11.500 --> 19:16.780 I don't know how they can say with a straight face that this is the path towards AGI. 19:16.780 --> 19:18.660 I mean, this is nonsense, right? 19:18.660 --> 19:21.160 So I get frustrated all the time. 19:21.160 --> 19:24.660 I know Aileen was trying to paint a more positive picture. 19:24.760 --> 19:30.220 I'm just extremely disappointed that I'm sorry for being open about that. 19:31.160 --> 19:34.360 Yeah, well, maybe I'll pop in on the negative note. 19:36.420 --> 19:43.240 I mean, what happens that I think we're pointing to here is that data are very resource intensive, right? 19:43.240 --> 20:00.980 And so it's only certain actors, it's something we say in the book, it's only certain actors in the world that have the resources to collect, store, maintain, utilize, analyze, deploy in some product or whatever, these kinds of data. 20:00.980 --> 20:07.140 And so at root, you only have certain actors in the world that are able to actually do that work. 20:07.140 --> 20:14.620 And because data are so expensive, we see all these kinds of shortcuts, like exactly like this, where it's like, oh, like, where can we get a big data set? 20:14.620 --> 20:21.860 Okay, Reddit, or like, oh, you know, where can we get images to train image, you know, face recognition algorithm? 20:21.860 --> 20:24.200 Oh, okay, like celebrity faces or whatever. 20:24.200 --> 20:32.280 And just like not connecting the dots of again, like, in what ways like who is not represented in those data sets. 20:32.280 --> 20:50.280 And there's like all these multiple points of kind of where inequality kind of enters that pipeline and like not catching it at any of those points, because then the people that are working with the data are often they are the white elite, non-immigrant men, 20:50.280 --> 20:54.660 you know, and they are representative of that demographic that are celebrities, right? 20:54.660 --> 21:00.080 So like, you know, that you have these kind of multiple stages at which those things don't get caught. 21:00.080 --> 21:09.460 But I mean, at the root of it, I often think of the constraint in a lot of ways is, it's financial, like folks are looking for a shortcut. 21:13.120 --> 21:22.100 I mean, I'm so glad, you know, Catherine, you brought the issue of like celebrity data set. 21:22.100 --> 21:41.140 So in the news event, you know, you've heard about Fox, which is kind of like a cloaking tool, you know, for privacy, it uses like very fundamental techniques and adversarial machine learning so that, you know, people can kind of, you know, like save their privacy from like terrible tools like Clearview AI. 21:41.660 --> 21:47.740 And Kashmiri Hill, kind of like the New York Times reporter, tried it out. 21:47.860 --> 21:52.640 It was so weird, you know, she got she got some really weird photos. 21:52.680 --> 21:58.540 And when she asked the PI of that project, like, hey, why are these photos looking really awry? 21:58.540 --> 22:02.820 I thought the whole point is to keep the face same in the cloaked picture. 22:03.040 --> 22:13.360 And he said something to the effect, and I'm perhaps going to misquote him, he said, Oh, yeah, women get mustaches and like, you know, men get eyelashes. 22:13.360 --> 22:15.380 That's how you kind of like cloak. 22:15.680 --> 22:17.140 You know, that's kind of like an outcome. 22:17.140 --> 22:25.500 I'm sure there's a whole like, semester of SDS gender studies to unpack there, because they use like celebrity faces. 22:25.720 --> 22:30.840 And like, how many transgenders are represented in like celebrity data sets? 22:31.240 --> 22:35.760 My first thought was, have you seen the winner of Eurovision 2019? 22:35.800 --> 22:36.940 Like, shame on you. 22:36.940 --> 22:38.900 Have you seen like RuPaul's Drag Race? 22:38.900 --> 22:39.860 Shame on you. 22:40.000 --> 22:42.620 But, but yeah, it is. 22:42.620 --> 22:46.900 And I'm so sorry, Eileen, that, you know, we're not like raining on your parade. 22:47.120 --> 23:01.440 But even though, even though somebody starts with really good intent, and I'm sure like the Fox team did it with very good intent of like, I am going to, you know, forward privacy, I'm going to like, you know, empower people. 23:01.440 --> 23:14.060 But like you said, like when they don't connect the dots, or like how Anima said, when they take like shortcuts, yeah, it doesn't lead to less than like harmful outcomes. 23:14.600 --> 23:21.520 So just as a quick reminder for the people listening, Fox is the paper that we discussed yesterday at Journal Club. 23:22.200 --> 23:24.540 So if you were there for that, awesome. 23:24.540 --> 23:26.520 If not, you can read about the paper. 23:26.520 --> 23:33.100 You can listen to me and a couple others talk about the paper I was talking about on our YouTube channel, where it should be up by now. 23:33.160 --> 23:34.920 Sorry, Eileen, did I cut you off? 23:35.260 --> 23:38.600 Oh, I saved my pessimistic comments to the end. 23:41.420 --> 23:44.460 But now we have our feedback loop completed. 23:44.460 --> 23:59.660 So I'm really curious about all the data practices that are going on out there, or the data impractices or abuse, because as I said, I'm coming from a privacy background, and how people's data is abused, so that they can be de-anonymized, they can be tracked, 23:59.660 --> 24:07.160 and essentially, entities that hold power can use this for surveillance or for large scale manipulation. 24:07.240 --> 24:18.080 And unfortunately, as citizens, or as human beings living in the world, we don't consent to our data being collected, but we don't have an option to opt out. 24:18.080 --> 24:36.780 So all of our data, under the premise of free applications, free products, or just walking on the streets in country X, is being collected by these entities that hold power to build large databases of any kind of information. 24:36.840 --> 24:42.200 And in some cases, this data is not being even used, because they think that we don't have a way to use it right now. 24:42.200 --> 24:50.180 But in the future, we will figure it out, figure out a way to make use of this data to make one type of consequential decision again. 24:50.180 --> 25:02.040 And this is extremely problematic, because the citizen is not informed about these practices and what the long term effects might be on individuals and society. 25:02.040 --> 25:21.800 And on the other hand, when we look at other power holders, such as proprietary software, or all companies that are providing these services for the entire world that are free, supposedly free, are making so much money because they are making use of the data that we are providing them. 25:21.800 --> 25:35.540 And after that, they keep building stronger machine learning models so that they can keep amplifying these biases in their direction with their targeted ads, or whatever their incentive is. 25:35.540 --> 25:40.660 And here there is a serious problem about the incentives of these systems. 25:40.660 --> 25:54.080 And both Catherine and Anima mentioned that, for example, we have these objective functions optimizing for accuracy in these systems that actually are helping these power entities. 25:54.080 --> 26:03.040 But the objective function could also be how can we use these data sets for social good, as opposed to optimizing for accuracy. 26:03.040 --> 26:15.040 So there are so many different faceted and different levels of abstraction of problems involved in this data ecosystem again. 26:15.040 --> 26:24.080 And I guess we can keep discussing these in more details, but today the goal is kind of give a glimpse of what is going on in this area. 26:24.480 --> 26:27.320 So Aileen, you picked out... guess wrong? 26:27.380 --> 26:31.240 Oh sorry, I just have like a quick follow-up for Aileen and Anima. 26:31.940 --> 26:37.020 I know that both of you have kind of revealed the shortcomings of ML systems. 26:37.020 --> 26:40.920 Anima, you've done extremely awesome work about how GPT-3 is flawed. 26:40.920 --> 26:44.560 And Aileen, your groundbreaking work on word embeddings. 26:45.540 --> 26:52.500 And I know that both of you have communicated that research to the community in a very approachable fashion. 26:52.980 --> 27:02.960 But I don't know about you, Aileen, but I visibly see the kind of like negative, rancorous feedback that you get. 27:03.240 --> 27:12.240 And from very well-intentioned research, I mean I don't know well-intentioned, from like people who are in the ML community, why do you think that happens? 27:12.240 --> 27:23.960 And Catherine, like from a feminist perspective, why are people from the own educated community like unloading on these researchers? 27:23.960 --> 27:26.660 I would love to get your perspectives on this. 27:26.680 --> 27:29.360 Sorry, Stella, I didn't mean to like... No, that's fine. 27:29.360 --> 27:31.620 That's very close to where I was going to go anyway. 27:31.820 --> 27:35.560 I mean, just maybe I can get started on this, right? 27:35.560 --> 27:38.500 I mean, to me, I think it's at the root of human psychology. 27:38.500 --> 27:40.620 Like no one wants to be the bad guy. 27:40.720 --> 27:43.560 No one wants to think of themselves as the bad guys. 27:43.560 --> 27:49.020 So they believe, right, they are going towards AGI through this GPT-3 model, for instance, right? 27:49.020 --> 27:50.980 I mean, I think they truly believe. 27:50.980 --> 27:55.520 So let's assume that. 27:55.520 --> 28:01.400 And I think the thing is they really don't want to also hear anything that may harm that, right? 28:01.400 --> 28:06.360 So they believe anything that may harm that path is that are their enemies. 28:06.360 --> 28:09.640 People who are voicing opinions against it are their enemies. 28:09.640 --> 28:19.700 And so that's why we see this vicious pushback on social media platforms like Twitter and, you know, and then they have huge follower counts. 28:19.700 --> 28:21.020 So that builds up, right? 28:21.020 --> 28:30.780 And all they have to do is do it, just kind of give their followers a sense to go after people who are bringing up these difficult issues. 28:30.860 --> 28:35.280 And honestly, because there's also a lot of insecurity, they don't know the answers to this. 28:35.280 --> 28:37.620 They don't know a way out. 28:37.620 --> 28:46.160 And they worry that their entire existence is in question, which I think that's why, you know, to me, I feel like we are on the right track. 28:46.160 --> 28:51.220 So we have to keep pushing this on and increasing awareness, have more allies. 28:51.360 --> 28:54.300 I think we will get more support. 28:54.300 --> 28:57.960 And it's already been amazing over the last few years, right? 28:57.960 --> 29:03.020 I mean, it's happened so quickly, even just the mention of AI bias was not there. 29:03.020 --> 29:06.240 It was completely dismissed in machine learning conferences. 29:06.340 --> 29:11.560 People working on it were considered not technical, you know, were not respected. 29:11.860 --> 29:17.580 There was a lot of gaslighting, you know, because it happens to be women, it happens to be minorities. 29:17.620 --> 29:25.140 So there was even more of all that racism and sexism coming into play when such research was reported. 29:25.440 --> 29:34.200 But now, you know, especially with the Black Lives Matter movement, that's when these companies finally said, we won't be selling face recognition to law enforcement, right? 29:34.200 --> 29:39.860 We've been asking for this, you know, we've signed letters, we've gone to the press, we've been doing this for a while. 29:39.860 --> 29:43.700 And it's been led by black women, which is amazing. 29:43.700 --> 29:48.860 But, you know, it took a broader global movement to make this happen. 29:48.860 --> 29:54.960 And I think we are building this momentum up, because it's not just about AI, it's not just about data ethics. 29:54.960 --> 29:58.000 It's a much broader question for humanity. 29:58.100 --> 30:01.260 And I think we're, you know, I have to, I remain hopeful. 30:01.960 --> 30:23.260 So since Anima brought this up, I wanted to take a moment and ask you guys, do any of you, as yourselves personally, or the companies or institutions that you represent, have any policies about what types of organizations or what types of tasks you will not do ML research or data research for? 30:23.260 --> 30:28.660 You know, be it police applications, military applications are two of the ones I know are really popular. 30:29.520 --> 30:41.980 I mean, speaking personally, disclaimer, sorry, before I say this, my company has a large number of lawyers who'd like me to tell you that my reviews on the stream represent myself and in no way represent my company at all. 30:42.260 --> 30:45.120 I work for a US government contractor. 30:45.300 --> 30:52.360 And a lot of the people who work at my company do ML research for the Department of Defense and the US military. 30:52.780 --> 30:59.320 And that's a very contentious topic, both within my company and I know within the ML community at large. 30:59.520 --> 31:06.380 So, you know, I have been personally trying to decide how to navigate that line myself. 31:06.420 --> 31:10.620 But I was wondering if any of you guys had particular stances you've developed? 31:13.880 --> 31:21.860 I mean, I'll just say, for my own work, and I'm not speaking about MIT, because obviously MIT is all wound up with all sorts of different things. 31:21.860 --> 31:29.360 But at least as a professor, you have autonomy to do what you want. 31:29.360 --> 31:38.260 And so at least from my perspective, I've decided to only work with folks who are doing some kind of counter hegemonic work. 31:38.260 --> 31:45.660 So activists, journalists, social movements, community based organizations. 31:45.660 --> 31:57.080 That's also kind of the, like, that's also in my research interest is to think about, like, how do we think about building data literacy in those kinds of sectors as well. 31:57.080 --> 31:58.600 So it's like a really nice fit. 31:58.600 --> 32:00.680 And that happens to be where my interests are. 32:00.720 --> 32:11.480 But I'm sympathetic to the stance that like, yeah, when you work in a large organization, as many of these places are, you have a lot less autonomy and thinking about, well, like, how do you navigate that as an individual? 32:11.480 --> 32:13.420 How do you navigate that as a team? 32:13.420 --> 32:17.100 And how do you navigate that at the level of the institution? 32:17.320 --> 32:25.940 But I will say, like, I am really energized by all of the organizing and activism that's happened within large companies right now as well. 32:25.940 --> 32:32.960 And so I think that is a huge, potentially, is a huge force for change as well. 32:33.800 --> 32:35.640 Yeah, I agree. 32:35.640 --> 32:39.400 And I want to add what Stella said earlier, right? 32:39.400 --> 32:44.160 I am not representing NVIDIA or Celtic for that matter. 32:44.160 --> 32:49.480 But I see a lot of support building up from the grassroots in these organizations. 32:49.500 --> 32:57.180 And NVIDIA being the platform for AI computing, you know, we are not directly selling these kind of AI applications, right? 32:57.180 --> 32:59.760 So it's really our customers and partners. 32:59.900 --> 33:06.080 And so there is less of a control in sense of everything the others do, but there's always periodic reviews. 33:06.080 --> 33:16.820 And, you know, there have been discussions internally in terms of how to make sure that our platforms are being used for good causes rather than bad ones. 33:17.000 --> 33:20.060 But I agree, there is a lot more work to be done. 33:20.060 --> 33:35.340 But I'm very energized by activism internally at NVIDIA, and also at Celtic, especially the students, everything, you know, from the recent Black Lives Matter, how it was really the students who have been energizing this movement. 33:35.340 --> 33:37.440 And so that gives me a lot of hope. 33:38.240 --> 33:39.480 Yeah, and I'm happy. 33:39.480 --> 33:39.920 Oh, sorry. 33:39.920 --> 33:40.940 Go ahead, Aileen. 33:41.220 --> 33:42.020 Please. 33:42.020 --> 33:50.380 I was just going to say that I'm not representing anyone except my humble opinion that might not be very well formed. 33:50.380 --> 34:01.100 But I try to build my own ethical framework when approaching these issues, since we don't have the AI ethics framework that is widely accepted by anyone. 34:01.100 --> 34:05.020 I basically try not to work on any confidential project. 34:05.020 --> 34:09.280 I don't try, I haven't worked on any confidential or secret projects. 34:09.280 --> 34:30.120 And other than that, I try to get involved in pro bono activities to help companies, entities, research their systems to understand what kind of negative side effects they might have, especially with regards to bias, can we come up with new forms of ways to measure bias, 34:30.120 --> 34:31.860 develop methods. 34:31.860 --> 34:35.840 And that has been my approach to this issue right now. 34:35.840 --> 34:46.140 But if we had a well accepted code of ethics among researchers, I think everyone would benefit from such a framework. 34:49.720 --> 34:51.480 I mean, I totally agree. 34:51.480 --> 35:02.020 I think one of the things that's kind of like helped me personally, again, you know, boilerplate, boilerplate, I'm only representing myself. 35:02.020 --> 35:06.620 When Anima said like reviews that kind of like really resonated with me. 35:07.140 --> 35:16.560 There is like, within Microsoft, there's this like committee called the AI Ethics and Research Organization. 35:16.620 --> 35:18.000 It's called ETHER. 35:18.000 --> 35:20.520 I'm perhaps butchering the acronym. 35:20.720 --> 35:30.400 But one of the things that they do, and it is like, you know, folks from MSR, you know, and it's just not the ML folks, but MSR is such a vibrant community. 35:30.400 --> 35:49.480 So, you know, you've got like sociologists, you know, folks from like, who think about bias from MSR New York, in Eric Horowitz's organization, kind of like, you know, making sure for critical like ML applications, you know, you need to get ETHER's sign off. 35:49.540 --> 35:52.840 So that has been a, you know, a guiding light. 35:52.900 --> 36:05.440 The second thing for me is like, you know, it's, I completely agree with Eileen, if there's some like, like a framework, and I, you know, that way you can meet it out to engineering engineers. 36:05.780 --> 36:18.180 The thing that's kind of worked for me is that the standards of business conduct at Microsoft is surprisingly good, which is a very, which is a very good thing to say, because they make it, they shoot it like Netflix style videos. 36:18.240 --> 36:21.480 So it's not the boring training that you have to kind of go through. 36:21.480 --> 36:28.960 I mean, I mean, I've worked in Microsoft, like, in 2013, it was just like the most like boring thing, but you still have to do it. 36:28.960 --> 36:30.360 It's like a read through it. 36:30.360 --> 36:36.220 But now it's like an actual like Netflix series, they've got the same actors coming in different years. 36:36.220 --> 36:59.640 But I mean, long story short, there was a module about how this guy tries to train his model on this, like private customer data, and how he gets into such so much of, you know, like, like a big, like big soup, he can't get out of it, you know, he's trying to demo it and he has to like talk to his manager and managers like, 36:59.640 --> 37:02.300 oh my gosh, you're gonna like, you know, lose your job. 37:02.300 --> 37:07.120 So they really rain it down on you in a more relatable fashion. 37:07.120 --> 37:30.420 So all of this to say that, you know, having this, you know, this committee to kind of like guide for critical like applications, and kind of having this like relatable, you know, ethics discussion has kind of like helped at least me and my team personally a lot, 37:30.420 --> 37:34.140 quickly ask Ram, so can they release this to the public? 37:34.140 --> 37:38.100 I think immensely from this as well. 37:38.100 --> 37:44.600 Yeah, I really hope that they do because it's, it's really captivating, like you're not bored. 37:45.280 --> 37:47.220 You know, when you're watching that. 37:48.020 --> 38:02.260 So I have a question from the discord that is something you guys have all touched on a little bit, but I think it's worth like directly addressing, which is that so we've been talking about how we were, we all represent large companies, or major universities, 38:02.260 --> 38:06.140 you know, institutions that have power and money and resources. 38:06.320 --> 38:10.940 And, you know, how, what is it? 38:11.120 --> 38:23.240 So what the question asked on discord was, was, you know, if, what's the role that these large institutions is powerful and franchise institutions play? 38:23.240 --> 38:32.320 And, you know, if the ability to collect large data sets or train these very large expensive models were somehow democratized, would that improve the situation? 38:32.980 --> 38:37.320 Ram, I think we should throw this over to you, because I know you've recently written a paper on this topic. 38:39.840 --> 38:44.260 Yeah, that's a really hard question for me. 38:44.500 --> 38:58.400 You know, just because, you know, again, from, from a, from a, from a winner sense, like, is machine learning inherently authoritarian, you know, because like, or is it inherently democratic? 38:58.580 --> 39:15.240 And, you know, there, there are arguments to be made on both sides, like, you know, like Anima started off with data compute, and the algorithms, that knowledge is kind of centralized in a lot of, you know, in a lot of aspects, right? 39:15.240 --> 39:30.960 You, you know, like, $10 million or so with their TPU training pods, which, you know, like a mom and pop shop, like an individual researcher is not going to have. 39:31.200 --> 39:42.000 And data, as we all know, like Catherine kind of like outlines in her book beautifully, like how it is really, it's situated with power, almost. 39:42.520 --> 39:50.540 So, you know, in a, in a, in a, in a land and winner sense, it really does, you know, one may almost say that, you know, machine learning is authoritarian. 39:50.540 --> 40:06.300 On the other hand, actually, one of the other papers that we read was Anima's Born Again Neural Nets paper, where she shows that, you know, a student can actually in certain like conditions outperform the teacher. 40:06.860 --> 40:20.920 And, and if you think about the revolution that transfer learning is bringing along, you know, you, a company can put out this like pre-trained model, and then you can just retrain the last layer for your like custom, like applications. 40:21.240 --> 40:33.520 So, and, and now with like, you know, systems like fast.ai and like, you know, they're bringing like machine learning, like two programmers, which again is not making it democratic. 40:33.520 --> 40:43.460 So it's almost like, you know, you kind of trying to install solar panels in your, in your house, as opposed to depending on like a nuclear power station. 40:43.480 --> 40:46.840 So I completely struggle with this question. 40:46.840 --> 40:49.120 I keep vacillating back and forth. 40:49.120 --> 40:56.140 And unfortunately I do not have like, I'm not, I'm not, I'm not like settled on an answer. 40:56.140 --> 41:00.800 And perhaps like other panelists will guide us, will guide us through this cave. 41:02.380 --> 41:05.140 Thank you Ram for bringing up the paper. 41:05.140 --> 41:09.960 I think to me, I, you know, I again pin my hope back on the algorithms, right? 41:09.960 --> 41:15.340 So I think we are just at the infancy in terms of algorithmic development for deep learning. 41:15.340 --> 41:18.220 We've been using the very basic algorithms. 41:18.220 --> 41:21.320 The question is, you know, where to go next? 41:21.320 --> 41:25.380 You know, I mentioned unsupervised learning earlier as a broad topic. 41:25.380 --> 41:29.500 I think within that, what we want is more controllability, right? 41:29.500 --> 41:31.620 Like for instance, generating text. 41:31.620 --> 41:43.100 If you could control what needs to be generated, then you can try, you know, head towards removing these biases, at least in what's generated, even if it's inherent in the model. 41:43.240 --> 41:46.100 And in fact, that's what we need in practical applications. 41:46.100 --> 41:48.660 You know, we don't want it to write weird poetry. 41:48.800 --> 41:56.280 We want it to maybe, right, give us knowledge facts or ultimately program, you know, do program synthesis and so on. 41:56.280 --> 41:58.420 And so that's something we've been working on. 41:58.420 --> 42:00.740 Same for text, same for images, right? 42:00.740 --> 42:07.880 Ultimately the GAN should be able to have controllable generation and it can control various attributes, right? 42:07.880 --> 42:13.760 And that way then it's really at the hands of the human on how to use these models. 42:13.760 --> 42:17.500 And currently we don't have good frameworks for that. 42:17.500 --> 42:28.860 And I think unsupervised learning is the key to enable that because you have to learn what are the hidden variations in data and then how to control and change that and be able to generate new examples. 42:29.360 --> 42:32.180 Yeah, so I think that's the direction to go forward. 42:33.100 --> 42:36.020 I completely agree with Anima, by the way. 42:36.020 --> 42:38.400 She has been mentioning unsupervised learning. 42:38.400 --> 42:58.840 I also think that the future is probably in unsupervised, self-supervised learning that is a bit maybe more intelligent, depends on how you define intelligent, because again, this is semantics and a vague term, but it seems like once we get there so that the system is able to figure out this universal learning pattern somehow, 42:58.840 --> 43:08.400 and then starts figuring out different kinds of tests by itself, we might be able to somehow incorporate some kind of awareness to the system. 43:08.400 --> 43:11.520 And one option would be a human in the loop, for example. 43:11.520 --> 43:19.420 And again, Anima mentioned this a few minutes ago, saying that human psychology and awareness plays a very important role here. 43:19.420 --> 43:27.320 Like, we are not able to agree on many of these questions, because we are dealing with ethical questions. 43:27.320 --> 43:33.800 Everyone has their own understanding, their own answer, and their own desired fairness notion. 43:34.140 --> 43:38.100 And we can't satisfy all of these at the same time. 43:38.100 --> 43:47.560 However, if we are able to bring some kind of awareness in these systems, then we might be able to change actions and decisions. 43:47.560 --> 43:59.980 Because, the evolution of how people are reacting to biases depends on how aware we are of our biases. 43:59.980 --> 44:03.320 It's not like people are trying to discriminate other groups. 44:03.320 --> 44:06.680 It's mostly caused by favoring the in-group. 44:06.680 --> 44:12.000 Like, there are no harms that are explicitly trying to be imposed upon other groups. 44:12.000 --> 44:16.020 But this unintentional bias ends up causing all of these harms. 44:16.020 --> 44:29.060 So, in an unsupervised way, we can figure out some kind of an efficient learning pattern, where we can also incorporate ethical mechanisms, then we will start getting there. 44:29.060 --> 44:43.400 But of course, all of these will require a lot of informed discussions from many different perspectives, and many different disciplines as well, because this is affecting all disciplines, all the sciences, and every social fabric. 44:45.700 --> 44:54.140 And yeah, maybe I'll just, I mean, I think the question of, you know, is machine learning authoritarian or not is a super interesting one. 44:54.140 --> 45:00.580 But ultimately, the way that we would answer it from a feminist perspective is, in whose hands is it? 45:00.580 --> 45:05.960 So, you know, so feminism, basically, what feminism brings is who questions. 45:05.960 --> 45:12.660 So, like, who is using it, for whom, to support whom, to oppress whom, and so on. 45:12.660 --> 45:20.520 And so, I mean, I think that, for me, is the, those are the questions to be asking when we're thinking about, like, democratizing AI. 45:20.520 --> 45:29.440 Like, I think it's lovely to think about making it easier to use models, democratizing access to data and computational power. 45:29.440 --> 45:33.880 Those are all great things as, like, a kind of infrastructure of democratization. 45:33.880 --> 45:45.620 But on their own, they won't create a democratic kind of usage of AI, because we also need to do the bridge to the groups that could really actually benefit from AI. 45:45.620 --> 45:54.840 And for me, like the, again, like I'm saying, like, I'm interested in sort of counter-hegemonic groups and thinking about, like, what are groups that actually hold power accountable? 45:54.840 --> 46:00.820 They're groups like lawyers and legal scholars, journalists, community organizations, social movements. 46:00.820 --> 46:11.660 Like, these are the folks who need to be kind of trained and brought to the table, if we're actually talking about a comprehensive effort to democratize AI. 46:11.660 --> 46:16.740 Because those are the folks that you want to understand, like, what are actually the creative, creative potentials. 46:16.740 --> 46:31.000 And there are many, like, some of the journalists that have started to discover the power of machine learning have been doing amazing things, like detecting, using image recognition to detect amber mining from satellite imagery in the Ukraine, for example. 46:31.000 --> 46:40.620 Like, you can do this, like, kind of amazing accountability work, but not every person is going to have that idea, because they haven't been exposed to working in this way. 46:40.620 --> 46:42.560 So, yeah. 46:43.140 --> 46:54.440 So, if we're talking about, you know, AI algorithms being developed by institutions of power, and the fact that, like, who is using algorithms matters a lot , is there a role in, like, adversarial AI? 46:54.440 --> 47:01.980 Adversarial AI is usually cast as a bad thing, because machine learning engineers do good, and those pesky evildoers are trying to ruin it. 47:01.980 --> 47:08.280 But if machine learning algorithms are being used for bad, you know, does that mean that adversarial AI techniques can be used for good? 47:08.700 --> 47:09.480 Oh, I love that. 47:09.480 --> 47:10.300 Totally. 47:11.120 --> 47:28.880 I totally think so, yes, because, like, there are so many reasons why groups or individuals would need to be protecting themselves, like, from, you know, we were just talking about Reddit, for example, you know, like, there's so many reasons why, you know, 47:28.880 --> 47:35.200 like, an act of, kind of, an adversarial act is not necessarily bad. 47:35.200 --> 47:57.500 Like, for example, I have a friend who, as a way of, kind of, fooling the just rampant data capture that's done by promotional ads and things in the browser, and so, like, the browser is always sending the IP address and whatnot, and what it does is, instead of, 47:57.500 --> 48:03.580 like, sending what it clicks on, the browser clicks on, it actually just sends every link on the page. 48:03.980 --> 48:14.280 So it's, like, hacking it through obfuscation, you know, and so it's, like, I think that's a brilliant technique for then preserving privacy. 48:14.280 --> 48:20.040 So you're obfuscating what the user is actually doing, and screwing with the ad systems or whatever. 48:20.040 --> 48:31.460 And so I think, especially in situations where power is not equal, which is every situation right now, like, we also need, like, good, like, white hat adversarial ML, basically. 48:32.100 --> 48:34.320 I love this, Catherine. 48:34.320 --> 48:38.420 I got so excited for the ad blocker research. 48:39.400 --> 48:51.900 I mean, right now, in the field of adversarial machine learning, the use of adversarial machine learning to enhance the experience of citizenship is still nascent, but, you know, there's fantastic work that's happening. 48:51.900 --> 48:53.260 I'm so excited for this. 48:53.280 --> 49:02.060 Bogdan Kulinich, I'm sorry if I'm pronouncing his last name incorrectly, developed this paper called POTS, which is Protective Optimization Techniques. 49:02.060 --> 49:04.320 We're into, kind of, like, empower. 49:04.380 --> 49:21.560 So what they do is, it's really cool, it turns out that, obviously, credit card scoring is biased against minority population, and they show, like, how you can poison the system tactically, like, do it, literally launch, like, a poisoning attack to, kind of, 49:21.560 --> 49:26.880 like, you know, make it better for the target population. 49:27.000 --> 49:42.840 There's really another cool work from some researchers who found that Facebook's ads platform is micro-targeting, and they, again, use adversarial ML to show, like, you know, faults in that. 49:42.840 --> 49:50.640 So that opens up, like, exactly, like, how Catherine says, white hat adversarial examples. 49:50.660 --> 49:57.060 Unfortunately, you know, it's not the dominant thinking. 49:57.060 --> 50:08.860 In fact, DARPA, when it released its guard challenge, which is, you know, securing against, like, AI challenge, they said, we need to protect ML system from attackers. 50:08.860 --> 50:11.920 And they do not define who attackers are. 50:12.020 --> 50:17.420 So that's... And I completely agree with Ram and Catherine. 50:17.420 --> 50:28.600 And I think there's also, right, because we don't know who are the attackers, I feel like there's also people are going in all kinds of weird directions with adversarial AI research, right? 50:28.600 --> 50:34.120 Because they don't know what kind of attack to, you know, and they're like, okay, maybe I'll do this norm bound or some other. 50:34.380 --> 50:36.620 And then it's like, okay, why does this matter? 50:36.620 --> 50:37.740 We don't know, right? 50:37.740 --> 50:42.040 So I think perceptual attacks for images are much more meaningful. 50:42.040 --> 50:51.120 I mean, you know, if those that kind of preserve the for humans, the perception, but, you know, change the statistics of data, right. 50:51.120 --> 50:59.800 And to me, I think the next step is to really have the regulatory agencies, you know, take on these tests. 51:00.060 --> 51:01.360 I mean, we have a long way to go. 51:01.360 --> 51:09.020 Because if you look at the NIST, which is the standards data sets for face recognition, that itself is heavily biased against minorities. 51:09.280 --> 51:18.240 But ultimately, the hope is regulatory agencies in each domain will be using these as tests, you know, especially for autonomous driving, this is a must, right? 51:18.240 --> 51:31.760 I mean, given the long tail, the only way to do is also have very heavy testing would need a lot of compute, but only then certify that this is roadworthy and otherwise be very careful about that. 51:32.580 --> 51:37.840 And Anima, to my understanding, NIST has been tasked to develop these AI standards. 51:37.840 --> 51:45.180 So they are especially working on those given the example with the face recognition or computer vision data sets they had as well. 51:45.180 --> 51:48.540 So that was a great point that you identified. 51:48.540 --> 51:59.880 And again, there are all of these privacy enhancing or fairness enhancing adversarial applications of machine learning, but they might not be very common right now. 51:59.880 --> 52:10.920 But the most simplistic version we can think of is synthetically generating data to oversample minority populations so that we can improve their representation. 52:10.920 --> 52:19.200 Because at least in real world statistics, it's well known that oversampling minority groups is a helpful approach. 52:19.200 --> 52:26.600 But at the same time, again, this calls for the need of some standardization in this process. 52:26.600 --> 52:32.760 And again, this ties back into the question of is only data bias or algorithms also biased? 52:32.760 --> 52:34.840 Why are we creating these algorithms? 52:34.840 --> 52:37.260 What are the goals of these algorithms? 52:37.420 --> 52:44.300 Who are the bias power holders implementing these algorithms? 52:44.300 --> 52:46.580 And again, so it really matters. 52:46.580 --> 52:49.500 And this might bring us to the Kant question here. 52:49.500 --> 52:52.200 Should we have this ontological approach? 52:52.200 --> 52:55.040 Or should we go with utilitarianism? 52:55.040 --> 52:55.760 And so on. 52:55.760 --> 53:05.120 So there are many interesting questions in this direction as well that we are starting to just scratch the surface nowadays, because it's becoming a very hot topic. 53:05.120 --> 53:07.660 So why are we even building these systems? 53:07.660 --> 53:10.120 Why are they going to empower at the end? 53:11.100 --> 53:16.540 Absolutely, I just want to quickly add to Alan's point on the oversampling for minorities. 53:16.600 --> 53:20.760 I think, yeah, if you don't have more data, I think that's a quick fix right? 53:20.760 --> 53:22.160 That's the starting point. 53:22.160 --> 53:26.180 But that itself can also create problems because GANs have mode collapse. 53:26.180 --> 53:30.840 So you just have very less diversity among, say, the minority class, right? 53:30.840 --> 53:42.840 For instance, in fact, it was shown when the images that are generated based on the celeb data set, the celebrity data set, black women tended to have all the makeup on, right? 53:42.840 --> 53:49.260 So it was and with these face recognition systems, when black women didn't have makeup, it was even worse. 53:49.260 --> 53:56.980 You know, so then there is kind of further bias even within the class because of the lack of diversity and variation. 53:57.300 --> 54:07.140 And that's why I think we need to disentangle all these factors and do unsupervised learning and learn, is there also enough diversity within the class in terms of these variations? 54:08.020 --> 54:11.380 Yeah, go ahead Stella, sorry. 54:11.380 --> 54:15.260 So that sort of feeds into something that someone in the Slack asked about as well. 54:15.260 --> 54:27.440 And it's been in the academic discussion of AI bias a lot recently, which is how models can exacerbate or create biases out of imperfections. 54:27.440 --> 54:36.340 Even things that aren't normally thought of as imperfections, just the choice of your loss function implicitly decides how you're going to weigh different samples. 54:36.340 --> 54:52.740 And this is crude, but if 90% of the people in a data set are white and 10% of the people in the data set are black, then an algorithm that gets a little gain in performance at the cost of a little loss on a white person, at the cost of a little loss on a black person, 54:52.740 --> 54:59.080 is going to perform better on average, if it just mostly pays attention to white people. 55:00.520 --> 55:05.900 So, you know, lost the rest of my train of thought. 55:07.220 --> 55:18.200 How, so it's not, it's not just that when we talk about bias in algorithms, we're not just talking about the data coming in, but kind of algorithms can create their own sorts of little biases. 55:18.800 --> 55:21.120 Like, what does it look like to combat that? 55:21.120 --> 55:23.280 And kind of how does that happen? 55:23.660 --> 55:28.480 You know, from the perspective of a data scientist who's looking at something, it's like, you know, how do you know that's happening? 55:28.480 --> 55:30.080 And how do you combat that? 55:32.020 --> 55:35.260 Just a quick anecdote, because I don't have an answer to your question. 55:35.940 --> 55:48.820 A quick anecdote was one of the resume screening algorithms, it wasn't Amazon, but it was like a different company, when they reverse engineered it to see, like, why was it preferring men over women? 55:48.820 --> 55:53.580 One of the factors that they found, one of the strongest predictors was like, the name. 55:53.580 --> 56:02.400 So if somebody was named Jared, in particular, it was a very strong predictor of success, as well as if they were a lacrosse player or not on their resume. 56:03.020 --> 56:12.220 And so it's like, just a sort of very interesting thing, where, in a way, what happens there, like, they're not even they're considering, well, I don't know, I don't really know the model. 56:12.220 --> 56:16.740 But like, I don't think they were directly considering race or gender in that case. 56:16.740 --> 56:24.180 But if you look at those things, like Jared is a very white, like kind of Anglo name. 56:25.160 --> 56:32.460 Lacrosse is a very elite, white, sort of private school sport in the United States, despite the fact that it was invented by Native Americans, actually. 56:32.720 --> 56:39.920 And so like, those things sort of, they seep in, and then they become these factors, and they become predictive factors. 56:39.920 --> 56:46.980 But like, behind those is always like, is race and gender and all of those things are like, kind of still underlying. 56:48.640 --> 56:58.500 Yeah, these systems learn the associations, and you were both asking what kind of metrics we should, for example, develop to understand how to improve these systems. 56:58.500 --> 57:12.040 So I was wondering, maybe this exists in literature, but do we have, for example, diversity dispersion measurement metrics to understand how diverse a group is, so that we can try to improve diversity within one social group? 57:12.040 --> 57:26.940 And then the other thing is, Stella, you mentioned objective functions, or machine learning algorithms that are trying to optimize based on the data representation, where people are categorized into groups, racial groups, or gender. 57:26.940 --> 57:29.860 And it's not even clear how ethical this approach is. 57:29.860 --> 57:32.260 Should people be represented in categories? 57:32.260 --> 57:38.640 Is this the correct proxy to a person's identity or being? 57:38.640 --> 57:44.180 And how should we deal with system design when we are developing new machine learning algorithms? 57:44.180 --> 57:52.140 Because each algorithm depends on these data types, the way they are represented, numeric values, ordinal, or whatever. 57:52.140 --> 58:03.340 And should we come up with more creative ways to design new algorithms for machine learning to encounter, to counter balance, these problems? 58:04.800 --> 58:20.580 And Stella, I mean, I want to rewrite this question back to Catherine, because when I was reading a chapter from her book, Catherine, you spoke about concepts which maintain power, like fairness, bias, and ethics. 58:20.580 --> 58:24.220 And you give this amazing framework about concepts of challenge power. 58:24.220 --> 58:26.100 I didn't even think about that. 58:27.200 --> 58:29.880 Is that something you want to talk about? 58:29.880 --> 58:36.400 I feel it's a really cool framework for engineers to think beyond the fact star or fact framework that we're used to. 58:36.940 --> 58:37.620 Sure. 58:37.620 --> 58:37.980 Yeah. 58:37.980 --> 58:41.980 I think that's in chapter one or two. 58:43.200 --> 58:44.600 I can't remember. 58:45.600 --> 59:00.940 But yeah, I mean, so one of the things I think that Lauren and I were frustrated by is that we just felt like fairness, and this is not, I guess I should say a little bit of a disclaimer , because I actually think the work that's going on in the fact star community is super exciting. 59:01.160 --> 59:06.620 And so like, this is not to say that, like, no one should do that work, because I am really happy that work is being done. 59:07.280 --> 59:14.300 But that's just some of the organizing framing concepts for what we're talking about aren't, they don't go far enough. 59:14.300 --> 59:30.540 And so it's like, instead of fairness, we propose justice, or no, instead of ethics, we propose justice, like, we're really thinking about what does it mean to make a just system, which means repairing past harms, like harm has been done in the world, 59:30.540 --> 59:49.140 we come to the world in the bodies, in which harm has been inflicted at the group level, for some of us for centuries, you know, and so like, how do we build systems that not only kind of treat us all equally as if we're all the same, but actually take that harm into account. 59:49.180 --> 59:54.200 And so we propose other kinds of like, justice, concepts like... 59:55.380 --> 59:57.420 My favorite one is transgression. 59:58.060 --> 59:59.340 That's what I was gonna say. 59:59.340 --> 01:00:08.640 Yeah, like, kind of like, why settle for accountability, because often accountability is after the fact, you know, like, why can't we build systems that liberate us? 01:00:08.640 --> 01:00:11.060 Why can't we build systems that heal us? 01:00:11.060 --> 01:00:16.560 Like, I feel like we, we have the power to imagine just much more expansively. 01:00:16.560 --> 01:00:25.620 So like, it doesn't always have to be after the fact that we're trying to, like, you know, repair things or whatever, we can really, we can really aspire to do much more. 01:00:25.620 --> 01:00:31.160 So co-liberation means, it's this idea that none of us are free until all of us are free. 01:00:31.160 --> 01:00:46.840 So we work with the most marginalized groups and center, like, this is a case where we would like, build a data set around whoever is most marginalized in whatever the application area is, and we would work with them directly, and they would be part of the process. 01:00:47.040 --> 01:00:54.080 And whatever is built, the system that is built would like, actively serve their needs in a really culturally grounded way, you know. 01:00:54.420 --> 01:00:57.480 And so I think that's totally feasible. 01:00:57.480 --> 01:00:59.600 It's just not the way that things are done right now. 01:01:00.440 --> 01:01:01.920 I fully agree. 01:01:01.920 --> 01:01:04.380 I think these are great points, right? 01:01:04.380 --> 01:01:07.600 Ultimately, we have to kind of rework the whole framework. 01:01:07.800 --> 01:01:15.720 But I think to begin with, even, right, like, the way these systems are now just kind of put out with almost no testing, right? 01:01:15.720 --> 01:01:32.060 And, and then only kind of somebody like, kind of, right, brings up the problem, usually a minority researcher, and then the company is doing everything to push back and use all the resources to discredit the researcher, I think, at least we should change that conversation. 01:01:32.060 --> 01:01:49.800 And I think that can happen, especially, you know, with the viewers of this panel, we need more hackers into these AI systems, go hack every AI system out there, reveal all the vulnerabilities, take it down, rebuild a new one. 01:01:49.800 --> 01:01:54.440 Yeah, you can have like a brigade of like algorithmic bias assessors. 01:01:54.440 --> 01:01:58.300 I mean, good trolls, good trolls. 01:01:59.840 --> 01:02:17.280 To kind of move this towards, you know, advice or takeaways from the audience, because this has been a question popping up in the chat a lot, you know, so you guys, you know, you guys are talking about how there's definitely steps that people can take to make progress in the way that they design their algorithms to be more inclusive and stuff like that. 01:02:17.280 --> 01:02:31.160 And one thing that kind of stood out to me is being implicitly present in a lot of what you guys said, but I don't think anyone said this explicitly, is that kind of the default view of AI is it's supposed to be descriptive, you take a description of the world as it was, 01:02:31.160 --> 01:02:36.020 and you put that through an algorithm, and the role of the algorithm is to make the future look like the past. 01:02:36.020 --> 01:02:40.120 And so in a very real sense, AI is intrinsically regressive. 01:02:40.460 --> 01:02:55.200 But at the same time, nobody, there's nobody saying you have to do that, you know, you can decide that you want to choose to value certain things more than others, and you want to kind of bring about certain impacts, and you can proscriptively put those into your algorithms. 01:02:56.580 --> 01:03:06.520 Personally, I have tried to advocate for this, and largely speaking, failed, because people are scandalized by the idea that you might tell the data how it's supposed to look. 01:03:09.300 --> 01:03:28.400 But you know, so if you're a, if you're a data scientist, or a machine learning research engineer, listening to this chat, you know, can, can, how can they proscriptively design their, their algorithms for equity, for justice, for liberation, for whatever all these very nice words are, 01:03:28.400 --> 01:03:32.160 that are better than bias, or fairness, I suppose. 01:03:32.660 --> 01:03:36.440 You know, what does that, what does that look like to design an algorithm in that fashion? 01:03:41.100 --> 01:03:44.120 I mean, I'll start, and then hopefully others can build. 01:03:44.120 --> 01:03:53.160 But like, I mean, I think what it would look like is to do, to really think about where, where what your work is doing is being applied, and on whom, right? 01:03:53.160 --> 01:04:07.180 And like, who you're making it for, like, who's your boss, and then like, who are ultimately like the end users, or who is kind of impacted, who are the decisions being made about because of the system, and to do basically like a power analysis of that system. 01:04:07.780 --> 01:04:24.820 And then think about what is, that application area, who has been systematically disenfranchised, and then work with the groups who are working towards a better future in that domain to co-create, like what the metrics should be for the future. 01:04:24.820 --> 01:04:33.680 You know, I mean, I think that we're not asking, I don't, I don't think it's like appropriate that like, like a machine learning engineer would like invent the metrics out of their head or something. 01:04:33.680 --> 01:04:34.220 You know what I mean? 01:04:34.220 --> 01:04:49.780 Like in a domain space, they may not know very much about, but I think it is feasible that like a company could work really well with a group of folks, say, working on prison abolition or something like that, and could craft like here are the metrics for what the world that we would like to see. 01:04:49.780 --> 01:04:52.740 And this is like the kind of outcomes that we want to see. 01:04:52.740 --> 01:04:56.520 And so that's, that would be my kind of generalized recommendation. 01:04:56.520 --> 01:05:05.500 But that said, and this is like one of the major tensions of this space, is that often people are building tools that are more generic, right? 01:05:05.500 --> 01:05:13.060 Like you're looking for generalizability, you're looking for abstraction and like that could be applied in a wide variety of different application domains. 01:05:13.060 --> 01:05:14.640 And that I think is a tension. 01:05:14.660 --> 01:05:22.700 I actually think that is like a major tension for AI doing good in the world and being culturally grounded and informed. 01:05:22.700 --> 01:05:23.860 I don't know how to resolve that. 01:05:23.860 --> 01:05:31.840 And I would love to hear what like the computer scientists and researchers have to say about if they see that tension as well or not. 01:05:32.740 --> 01:05:34.440 I fully agree. 01:05:34.440 --> 01:05:39.020 And I think to me that foundations of that goes back to the education, right? 01:05:39.020 --> 01:05:47.420 So computer science and engineering has had the military background, you know, it started as a way to get into the military and that's there today. 01:05:47.420 --> 01:05:50.580 Like, you know, we are told to keep our emotions aside. 01:05:50.620 --> 01:05:56.880 In fact, that was one of a lot of gaslighting on Twitter is about, oh, why are we emotionally charged about machine learning? 01:05:56.880 --> 01:05:58.940 Just focus on the math, you know? 01:05:58.940 --> 01:06:05.560 And so that's something that's pervasive in the educational system and then beyond in the companies, right? 01:06:05.560 --> 01:06:07.360 Like, you don't worry about this. 01:06:07.360 --> 01:06:09.400 You do, you know, you derive math. 01:06:09.400 --> 01:06:10.960 You just, you know, program. 01:06:10.960 --> 01:06:14.520 So you're asked to become a robot without any emotions. 01:06:14.520 --> 01:06:18.200 And I think that's just really dangerous, right? 01:06:18.200 --> 01:06:21.120 Because it's not every engineer who's the problem. 01:06:21.120 --> 01:06:34.900 I mean, there's absolutely a decision maker who is probably, right, kind of, you know, may or may not understand the implications, but it's kind of directing all this and leading to the pervasive bias and injustice. 01:06:35.240 --> 01:06:37.140 I think we have to start with education. 01:06:37.140 --> 01:06:38.740 That's where I would pin my hopes. 01:06:38.740 --> 01:06:43.360 For instance, at Caltech, we have a course on ethics for AI. 01:06:43.360 --> 01:06:45.720 I think every university is now doing that. 01:06:45.720 --> 01:06:48.500 Ultimately more humanistic education. 01:06:48.680 --> 01:06:51.080 You know, we are kind of, you know, I went to technical school. 01:06:51.080 --> 01:06:53.280 Now I'm in a technical school, right? 01:06:53.280 --> 01:06:57.440 The role of humanities there needs to be even further emphasized. 01:06:57.440 --> 01:07:04.200 You know, it's much better than when I went to school, but it's still a long way to go to integrate humanities and engineering together. 01:07:04.600 --> 01:07:06.700 I completely agree with Anima. 01:07:06.700 --> 01:07:08.580 I'm also teaching bias in AI. 01:07:08.580 --> 01:07:23.360 I think it needs to be part of the curriculum, not for computer scientists only, but also engineers, humanities, social scientists, probably as well, because data scientists or essentially anyone that is interested in computing is using these tools. 01:07:23.360 --> 01:07:28.700 So they need to understand that all of these tools exist within a context . 01:07:28.700 --> 01:07:33.520 They don't just exist in an abstract, generalizable way, as Catherine mentioned. 01:07:33.520 --> 01:07:43.720 It's correct that these tools or algorithms were developed so that they can be applied to hypothetical data sets so that the theory or foundations are presented. 01:07:43.720 --> 01:07:55.360 But then when we take social data and apply these tools for machine learning tests in the social domain, then we are dealing with all kinds of human complications. 01:07:55.360 --> 01:08:10.040 So people need to always keep in mind that these systems, yes, they might look objective and neutral, but they are not in the social context because they exist with all the problems or side effects that human decision making brings into the game. 01:08:10.040 --> 01:08:17.460 But moreover, AI also brings its own kinds of problems, such as new types of biases we are not even aware of yet. 01:08:17.460 --> 01:08:20.540 And these keep getting amplified in society. 01:08:20.540 --> 01:08:42.220 So before we start introducing more of those, we need to understand what we are bringing to the world and what we should be careful about, because all of these systems are deployed right now without understanding what side effects they might have or what the implications are on society, 01:08:42.220 --> 01:08:44.920 economy, democracy, and so on. 01:08:44.920 --> 01:08:50.860 And now we would need to deal with all of these consequences on top of all the other things. 01:08:53.780 --> 01:09:09.800 It seems like a major takeaway you guys all advocate for is to kind of broaden the scope of who gets included in the conversation about AI design, about AI data collection, you know, bringing in activists in the relevant spaces, bringing in other stakeholders. 01:09:10.880 --> 01:09:19.900 And, you know, one thing that I would like to add to that is that the notion of stakeholder is really politically fraught. 01:09:19.980 --> 01:09:35.820 In industry context, stakeholders are generally conceived of as like, you know, so I suppose I'm at a ML company developing an algorithm that I'm going to sell to another company to use for financial profits. 01:09:35.820 --> 01:09:48.120 You know, the stakeholders in this process are generally conceived as being me and my team, my superiors at my company, the people I'm selling the algorithm to, and the stockholders of their company. 01:09:48.700 --> 01:09:55.140 And then I think that misses a really crucial point, which is the stakeholders are the people whose data is going to be analyzed. 01:09:55.600 --> 01:10:04.780 And their voice is very, very frequently just completely left out of any discussion about how things are designed, why things are designed, and where they come from. 01:10:04.800 --> 01:10:17.260 And I mean, to an extent, partially by design, because a lot of companies exist out of the fact, because of the fact that they can make money analyzing user data, like that's their sole reason, that's their reason for existing. 01:10:17.260 --> 01:10:18.620 There are lots of these things. 01:10:19.700 --> 01:10:29.120 But broadening, I think broadening the notion of a stakeholder , in particular, viewing anyone whose data is going to be processed by an AI as a stakeholder is really important. 01:10:32.960 --> 01:10:35.120 Yeah, I just I totally agree with that. 01:10:35.120 --> 01:10:52.540 We have a chapter in our book, which is about the principle embrace pluralism, which is basically this idea that, you know, we create more complete knowledge, like the path to objectivity is not through individual genius, but it's through bringing multiple different perspectives to the table. 01:10:52.820 --> 01:11:01.680 But I agree with you, I think like the it's important to think about like, because we can say, oh, it was participatory, like that company with all the money and the other company with all the money. 01:11:02.580 --> 01:11:12.180 So I do think and this is because participation, I mean, there's this whole world, you know, participatory design and participatory methods. 01:11:12.700 --> 01:11:24.980 And so like, there's even folks have talked about participation washing, where, you know, you're like, oh, yeah, we did a, you know, this also happens in urban planning, like, oh, yeah, we did a public meeting and it was at like 10pm and there was no childcare. 01:11:27.460 --> 01:11:31.440 So like, what's the quality and the character of the participation? 01:11:31.440 --> 01:11:32.660 Like who shows up? 01:11:32.660 --> 01:11:34.340 Who are these stakeholders? 01:11:34.340 --> 01:11:37.740 And who are the stakeholders that are not convenient to have? 01:11:37.740 --> 01:11:39.680 And how do you make folks listen to them? 01:11:39.680 --> 01:11:45.480 I mean, I think that's a big, that's a big question when ultimately, like the goal is to optimize profit. 01:11:45.480 --> 01:11:47.760 That to me is a conundrum. 01:11:50.090 --> 01:11:53.930 I think how do you make people listen, I think is a really interesting question. 01:11:53.930 --> 01:12:06.930 You know, a lot of people, I imagine a lot of people feel like they would like to do something good, you know, they would like to be more just and do their design the right way. 01:12:07.090 --> 01:12:19.310 But you know, how do you convince your boss or your boss's boss who controls the money and the funding, or the money and the decision making power, and who's maybe more interested in what's profitable? 01:12:19.350 --> 01:12:23.510 You know, how do you do any of you guys have advice about how to approach that conversation? 01:12:23.890 --> 01:12:37.330 Or how to how to advocate for justice in AI design, in context where that input is not necessarily the primary concern of the people who pull the strings? 01:12:38.530 --> 01:13:00.950 I'm interested in that as a research question, if we can figure out a way to simulate AI systems to show their long term gains and losses for all the parties that are involved, I think we would be able to understand how this affects economy in very from various perspectives. 01:13:00.950 --> 01:13:03.490 But again, this is a hypothetical research question. 01:13:03.490 --> 01:13:06.610 I'm sure there's already some great research on this yet. 01:13:06.610 --> 01:13:15.430 But maybe long term goals might be able to convince people that are profit driven as well. 01:13:17.230 --> 01:13:41.190 I mean, especially if there's concern, I really like the idea a lot, especially if there's concern, I mean, as there's this growing sort of industry of sort of impact assessment, or algorithmic bias assessment, and things like this, this is like, bringing folks in and doing a more participatory design process is a preventative for what could happen later, 01:13:41.190 --> 01:13:49.990 and so I think maybe that's the, like, if the case needs to be economic rather than moral, I mean, you can make the moral case, but probably what that listens to is the economic case, right? 01:13:50.010 --> 01:14:06.230 But if the case needs to be economic, I think that, you know, it would be lovely to have some quantifiable research that you could cite to say that like, yeah, there's this like, very strong economic case that this will reduce the number of lawsuits later, 01:14:06.230 --> 01:14:10.070 that you'll be fully compliant with all federal regulations, but you know what I mean? 01:14:10.070 --> 01:14:24.410 Like this, these kinds of things, I think that that might be the path for convincing the employers, and like, ultimately, like, you've invested in prevention, so that you don't have to clean up a big mess after the fact. 01:14:27.970 --> 01:14:33.070 That's why academia and the independent researchers have such an important role, right? 01:14:33.070 --> 01:14:39.410 Because they can talk openly about these issues, like, you know, what happened with the studies, right? 01:14:39.410 --> 01:14:50.810 Even though internally, there was a lot of conversation at Amazon, when I was there, you know, having that external light into it, and then the wider public part spreading in it, I think is critical. 01:14:51.570 --> 01:14:56.650 Because, you know, like being in these companies, it's just the whole system is so short term driven, right? 01:14:56.650 --> 01:15:02.690 The Wall Street expects quarterly results, that's not delivered, the company won't even exist, right? 01:15:02.690 --> 01:15:06.690 So the whole system is not built for the long term implications. 01:15:06.690 --> 01:15:09.310 And I think that's what creates these challenges. 01:15:09.790 --> 01:15:26.090 And so somehow, maybe then having more participation from the public, right, can show that also, you know, bad AI can lead to bad PR and loss of profits, even in the short term, I think that activism and awareness is critical to change this. 01:15:28.630 --> 01:15:30.210 Ram, you want to jump in? 01:15:30.210 --> 01:15:36.850 Yeah, I was just framing about what Anima said. 01:15:37.010 --> 01:15:51.330 But the way I think about this is, at least again, through lived experience, I can say that what has changed for me is that if you think of like inclusion, as after the fact, that's kind of like really weird. 01:15:51.410 --> 01:15:57.810 One of the things that I'm very lucky is that inclusion is part of my core priorities that I set. 01:15:57.930 --> 01:16:01.210 Like, it's just not for me, it's for everybody in the company. 01:16:01.450 --> 01:16:04.010 Like, you've got to set that, what is your inclusion priority? 01:16:04.010 --> 01:16:09.990 For me, it might be like, you know, I'm going to take care of Pride Month, like, you know, talks or, you know, whatever things, right? 01:16:09.990 --> 01:16:12.130 So obviously, it doesn't solve all problems. 01:16:12.130 --> 01:16:24.370 So first of all, I fundamentally believe that if your organization is questioning why you're doing something ethical, that itself is a big red flag for me. 01:16:24.990 --> 01:16:33.650 Right, like, and for the business case, I do agree, like, you know, there's this, like, you know, how Catherine said, there could be some financial motivation. 01:16:33.650 --> 01:16:42.810 I saw like an HPR article about like, how diversity leads to more surplus, because you're selling your product to more people. 01:16:43.030 --> 01:16:45.330 That could be like an other way. 01:16:45.630 --> 01:16:50.110 The PR angle, like, I mean, everybody is afraid of like bad marketing. 01:16:50.150 --> 01:16:51.830 I'd never actually thought about that. 01:16:51.830 --> 01:16:53.370 So thanks for bringing that up. 01:16:53.430 --> 01:16:59.310 But for me, intrinsically, you know, I have never been told you can't publish this. 01:16:59.310 --> 01:17:02.310 I mean, but again, it's a very different setting, like MSR. 01:17:02.310 --> 01:17:04.650 And like, you know, product setting is very different. 01:17:05.150 --> 01:17:15.830 But yeah, I would be very uncomfortable if I have to make a case to my manager as to why I'm doing something ethical in the first place. 01:17:15.830 --> 01:17:22.770 But like Anima pointed out, like, business, I've only worked in one organization for like, my working life. 01:17:22.770 --> 01:17:24.770 I do not know how others operate. 01:17:24.770 --> 01:17:28.370 I know some of the CEOs out there question why they should use pronouns. 01:17:28.610 --> 01:17:36.530 So or like, why they should like, you know, why they do not like fundamentally care about democracy and just want to dismantle it. 01:17:36.530 --> 01:17:40.110 So I don't know, I do not know how things work there from my lived experience. 01:17:40.290 --> 01:17:42.870 I don't know if this might actually be helpful for others. 01:17:45.840 --> 01:17:52.600 All right, so it seems like the kind of two major points you guys have raised is that you can view it as a risk mitigation factor. 01:17:53.600 --> 01:17:57.260 Because it will stop things from it can stop things from going wrong in the future. 01:17:57.260 --> 01:17:59.500 And it's in some sense insurance on that. 01:17:59.500 --> 01:18:04.160 And it's also a PR thing, which is partially a risk mitigation factor. 01:18:04.160 --> 01:18:13.320 You really don't want the New York Times running an article saying so and so's company develops algorithms systematically discriminates against women in hiring. 01:18:13.780 --> 01:18:16.240 Oh, I actually saw a headline. 01:18:16.240 --> 01:18:16.620 Yeah. 01:18:16.620 --> 01:18:17.660 Oh, sorry. 01:18:17.660 --> 01:18:18.160 Yeah. 01:18:18.160 --> 01:18:19.340 I actually saw an email. 01:18:19.340 --> 01:18:24.040 I only saw it through your tweet about gentrify or genderify. 01:18:24.400 --> 01:18:35.440 Yeah, another thing after all the right outcry got withdrawn, right? 01:18:35.440 --> 01:18:43.140 The fact that it even got launched is just appalling the kind of lack of awareness these companies. 01:18:43.500 --> 01:18:44.920 Yeah, it was shocking. 01:18:44.920 --> 01:18:47.240 You know, there was another company. 01:18:49.100 --> 01:18:51.320 I don't know who alerted me to it. 01:18:51.320 --> 01:19:03.060 But essentially, like their solution for more representation on stock photos was to kind of like artificially generate like photos using GANs. 01:19:03.060 --> 01:19:06.620 So you will not actually pay money to hire a black model. 01:19:06.620 --> 01:19:12.020 Oh, let's just use StyleGAN and like to kind of like, like, how do you get through? 01:19:12.020 --> 01:19:17.960 Like, who's funding this and goes back to an email like initial comment about VCs getting excited. 01:19:17.960 --> 01:19:20.860 So that I feel is a bigger like thing. 01:19:20.860 --> 01:19:32.240 I would be very less less worried about managers and more worried about like plutocratic like Silicon Valley folks to then like actually managers. 01:19:32.660 --> 01:19:44.100 So to kind of contextualize this for people in the audience who aren't aware, Genderify is a product that existed for all of about four hours. 01:19:46.900 --> 01:19:56.960 What their business model was, was that you can send them user data, you can send them photos, you can send them people's names, and they will 01:20:00.440 --> 01:20:20.620 they say they will determine the gender of the people that you in that you sell to or the people that are in your data set with an aim of doing better predictive marketing and stuff based on the... I'm having trouble coming up with a word other than assumed gender of the of the people in your data set. 01:20:21.240 --> 01:20:40.240 And this was this was funded by some VCs and it went online and everybody including lots of people in the ML space who don't really care very much, who don't talk very much publicly about about ML bias lost their shit because it was a completely absurd idea. 01:20:40.620 --> 01:20:47.140 And just to give an example, right, the word professor was only 1.9% females. 01:20:47.160 --> 01:20:50.980 So that's the assigned percentage for me in terms of gender. 01:20:50.980 --> 01:20:53.600 So I think that that was the problematic thing. 01:20:53.600 --> 01:21:01.340 Like how could you just type any name, like any word, not just name, any word in the vocabulary and assign a gender to that? 01:21:01.340 --> 01:21:04.140 I mean, and only binary genders, right? 01:21:04.140 --> 01:21:06.680 It's just appalling the whole concept. 01:21:06.760 --> 01:21:14.180 Yeah, my friend was saying the only time that's allowed it's in French, any other language, like, you know, because, you know, nouns have gender. 01:21:14.260 --> 01:21:17.820 Any other time table doesn't have like a male or female. 01:21:17.820 --> 01:21:18.600 Yes. 01:21:18.700 --> 01:21:19.560 Anyway. 01:21:19.700 --> 01:21:23.620 Yeah, but I'll know that like there are like this product is insane. 01:21:23.620 --> 01:21:30.120 But there are other products like there's all these gender detection libraries out there, both for faces. 01:21:30.140 --> 01:21:44.760 So like facial detection, which the researcher Oz Keys has written this amazing paper called Misrecognition Machines, which is about how harmful face like gender detection of faces is for trans folks. 01:21:44.860 --> 01:21:53.340 But then there's also all sorts of libraries that infer gender from name using various kinds of like name databases. 01:21:53.340 --> 01:22:04.600 And like, maybe I'll be like, sort of a naysayer here, because like, in general, I feel like these applications are very well, again, it depends on the application domain. 01:22:04.600 --> 01:22:17.840 So like, if you're doing this for predictive marketing, or like Oz talks about how they're doing it for bathroom access, which is like, disgusting, you know, like, are you the right gender to get into the bathroom or whatever? 01:22:18.480 --> 01:22:20.800 Those are super harmful and awful. 01:22:20.800 --> 01:22:22.840 And like those kind of applications shouldn't exist. 01:22:22.840 --> 01:22:41.480 At the same time, these gender detection libraries are helpful for doing like, bibliometric studies, where you're looking across like a huge corpus of say, like, I don't know, like several million authors and trying to see like, what's the over representation of men in that data set or something like that, 01:22:41.480 --> 01:22:50.340 you know, and so like, that that's why like, again, I think it's hard to just say categorically, hmm, something is bad or good. 01:22:50.340 --> 01:23:01.600 Although I do have certain things that I do think that way about, but like, that could enable really interesting research that monitors back on the patriarchy, for example. 01:23:01.620 --> 01:23:06.000 And so like, again, it's like, who's using it? 01:23:06.000 --> 01:23:06.740 For what? 01:23:07.740 --> 01:23:09.760 Yeah, like, that's what matters. 01:23:09.760 --> 01:23:15.860 And just to give an example, we have used the gender detector, but that was to learn about the MeToo conversation. 01:23:15.860 --> 01:23:23.520 So we, you know, collect lots of MeToo data from Twitter, men talking about it, how are women talking about it? 01:23:24.160 --> 01:23:26.500 How are Democrats, Republicans? 01:23:26.800 --> 01:23:34.040 So yeah, so that kind of, so you know, and this is in collaboration with Professor in Sociology, Mike Alvarez. 01:23:34.040 --> 01:23:36.840 So you know, like, the context certainly matters. 01:23:36.840 --> 01:23:45.820 We can't just ban all applications of gender detection, but really contextualize and look at whether it's for harm or for good. 01:23:45.860 --> 01:23:46.120 Yeah. 01:23:46.120 --> 01:24:03.100 And in medical applications, we especially need very accurate biased representations, because we know that for certain disease, the diagnosis and prognosis depends on ethnic factors, gender, and socioeconomic background as well. 01:24:03.100 --> 01:24:08.820 Of course, there are biases representing injustices in these medical data sets. 01:24:08.820 --> 01:24:16.580 But at the same time, there are biases that we need to learn from for more accurate medical treatment. 01:24:16.580 --> 01:24:26.600 And I guess that's why, for example, in the medical domain, machine decision making is becoming much more accurate than human decision making. 01:24:26.600 --> 01:24:32.220 But again, yet we need to deal with all the other disadvantages that come with this. 01:24:32.220 --> 01:24:39.680 So again, I guess everyone keeps mentioning this, it really depends on the context where we are using these things. 01:24:39.680 --> 01:24:45.820 And name-based gender detection is a kind of an outrageous example. 01:24:45.820 --> 01:25:04.500 But then there is a supercharged version of this in country X, where machine learning is used to predict the criminality of a person from their face, or actually give people social scores so that they can be sent to workers camps for their entire lives, 01:25:04.500 --> 01:25:06.480 because the machine said so. 01:25:06.480 --> 01:25:12.380 So there's a spectrum of these effects that we can discuss. 01:25:12.380 --> 01:25:19.660 And then what foundations should we be building on, given these examples that already exist in the world? 01:25:21.980 --> 01:25:23.780 Well, thanks for sharing that. 01:25:23.780 --> 01:25:26.720 I was just, I completely dismissed it, Catherine. 01:25:26.720 --> 01:25:31.620 And thank you, Anima and Eileen for, you know, correcting this. 01:25:31.620 --> 01:25:32.660 Thank you. 01:25:33.180 --> 01:25:35.720 I didn't mean to correct anything, by the way. 01:25:35.720 --> 01:25:38.520 Oh no, it was like, you know, I learned something. 01:25:38.520 --> 01:25:40.180 So thanks for telling me about this. 01:25:40.180 --> 01:25:40.920 Yes. 01:25:41.280 --> 01:25:42.280 I appreciate it. 01:25:42.280 --> 01:25:44.140 What did you learn, Ram? 01:25:44.220 --> 01:25:48.180 Well, I just completely dismissed like gender detection. 01:25:48.180 --> 01:25:51.160 I was like, oh, this is like, has literally no use. 01:25:51.160 --> 01:25:56.620 If I want to ask somebody's gender, I will just ask them if it's part of my business. 01:25:56.620 --> 01:25:57.900 I'll say it's none of my business. 01:25:57.900 --> 01:25:58.900 I'm not going to ask. 01:25:58.900 --> 01:26:04.900 But when, I was like, okay, that kind of makes sense. 01:26:04.920 --> 01:26:08.260 And when Eileen spoke about the medical contacts, that makes sense. 01:26:08.320 --> 01:26:12.680 So yeah, this was, like I said, I already learned something from this panel. 01:26:12.680 --> 01:26:16.680 And within like two more minutes for the panel to get over, I learned something. 01:26:16.680 --> 01:26:17.920 I learned a lot of things. 01:26:20.960 --> 01:26:30.660 Yeah, the example that Eileen brought up of criminality prediction is unfortunately running rampant in the AI literature right now. 01:26:31.160 --> 01:26:45.940 I feel like there is a half dozen papers that have been published and then promptly retracted after a massive public outcry, because they're basically trying to like reinvent phrenology, but doing it with machine learning, and therefore it's good. 01:26:47.000 --> 01:26:55.120 There's a whole lot that can be said about this that could and actually almost was its own panel at DEF CON. 01:26:56.100 --> 01:27:16.240 But there's a really good paper on this called Criminality from Facial Illusion, which I'll send out on Discord and on Twitch to people interested in reading about this problem that does a really good job of picking apart a lot of the issues with both morally and also technologically, 01:27:16.240 --> 01:27:22.500 because a lot of these papers are fundamentally technologically unsound with these kinds of works. 01:27:23.300 --> 01:27:26.880 So we're approaching an hour and a half. 01:27:27.540 --> 01:27:29.460 This is the last event of the day. 01:27:29.460 --> 01:27:31.480 I'm enjoying talking to you guys a lot. 01:27:31.480 --> 01:27:35.760 I don't know if you have somewhere you need to be, but I understand that. 01:27:35.760 --> 01:27:41.600 But if you guys are interested in continuing to talk, I am more than happy to do so. 01:27:42.580 --> 01:27:45.480 I have to go make dinner for my family. 01:27:49.740 --> 01:27:54.440 And it's actually sunny for a very short period of time in Seattle. 01:27:56.360 --> 01:27:59.560 Trying to sneak in like a run. 01:27:59.560 --> 01:28:06.660 Stella, before we all disperse, I first want to thank you so much for organizing this. 01:28:06.660 --> 01:28:10.320 You know, you have put so much thought and intention into this. 01:28:10.320 --> 01:28:14.940 Like, I remember like asking you like, Hey, I don't want to be the male in this. 01:28:14.940 --> 01:28:18.760 Like, have you thought about inclusive panel? 01:28:18.760 --> 01:28:19.940 I don't want to be a male. 01:28:19.940 --> 01:28:25.740 You wrote like, perhaps the most comprehensive male, you know, about this. 01:28:25.740 --> 01:28:28.980 So thank you so much for, you know, moving with intention, Stella. 01:28:28.980 --> 01:28:32.420 It clearly shows when you organize this panel. 01:28:32.420 --> 01:28:33.260 Thank you. 01:28:34.680 --> 01:28:35.380 Thank you. 01:28:35.380 --> 01:28:35.860 Second. 01:28:35.860 --> 01:28:36.560 Thank you. 01:28:36.560 --> 01:28:37.520 This is amazing. 01:28:37.520 --> 01:28:41.420 I learned so much and it's great to connect with all of you. 01:28:41.420 --> 01:28:42.540 And thanks, Stella. 01:28:42.540 --> 01:28:44.840 I enjoyed discussing with all of you. 01:28:44.840 --> 01:28:46.260 I think this was productive. 01:28:46.260 --> 01:28:49.400 And now I have more research ideas that I can go on. 01:28:51.560 --> 01:29:03.460 So quickly before you guys run, I want to take advantage of the fact that I have several experts here to get your thoughts on something I've been working on recently, which we've, we've almost touched on this several times. 01:29:03.460 --> 01:29:09.140 And it's that a lot of times machine learning algorithms have a underlying ontology. 01:29:09.140 --> 01:29:20.600 So Catherine mentioned, you know, that some gender algorithms exclude transgender people, you know, but even when you're talking about like all sorts of other applications you know, I work a lot with U.S. 01:29:20.600 --> 01:29:22.080 government data scientists. 01:29:22.080 --> 01:29:24.420 And so they're working with data sets that come from the U.S. 01:29:24.420 --> 01:29:24.780 census. 01:29:24.780 --> 01:29:25.120 And the U.S. 01:29:25.120 --> 01:29:33.260 census is a very specific notion of what races, and there isn't an option to use anything else because you're, you know, you're the U.S. 01:29:33.260 --> 01:29:34.600 government and you're using U.S. 01:29:34.600 --> 01:29:35.600 census data. 01:29:36.260 --> 01:29:50.340 But kind of how you, how you define your categories, you know, what, how you choose who gets labeled as white, who gets labeled as Asian, really makes a big impact on what your algorithm does and how it behaves. 01:29:50.340 --> 01:29:55.120 And whether you include certain categories, you know, if you include transgender people or not. 01:29:55.220 --> 01:30:09.260 I mean, if transgender, if non-binary transgender people, people who don't identify as, as, as a man or as a woman is not a possible label, then you have a guaranteed 100% incorrect assignment rate for them. 01:30:10.660 --> 01:30:17.660 So, you know, do you have any, any thoughts or advice about how to kind of tackle, you know, how to, how to approach this problem? 01:30:17.660 --> 01:30:24.740 Obviously, originally, we can go and talk to the impacted communities and try to figure out what the right buckets are. 01:30:24.740 --> 01:30:30.720 But oftentimes we start from a position where our data has been pre-collected for us by someone else. 01:30:30.720 --> 01:30:39.520 And so if you have data that has this kind of problem in the way the questions were asked originally, you know, is there something you can do with it still? 01:30:40.300 --> 01:30:43.180 I think that's a great question, right? 01:30:43.180 --> 01:30:46.640 And a difficult one because, you know, the systems are already built. 01:30:46.640 --> 01:30:48.680 Like you were saying, the categories are there. 01:30:48.680 --> 01:30:50.340 This is the label data. 01:30:50.620 --> 01:31:00.380 So I think towards getting to solutions for this, I think an important step is being able to trust the system in terms of confidence levels, right? 01:31:00.380 --> 01:31:04.620 Like, can the system say truly how confident it is? 01:31:04.620 --> 01:31:07.620 And the standard neural networks don't have this property. 01:31:07.620 --> 01:31:10.180 So it can make high confidence errors. 01:31:10.900 --> 01:31:14.680 You know, it can be like 95% confident and be wrong, right? 01:31:14.680 --> 01:31:23.500 And that especially happens on these kind of hard examples, which hasn't seen in training data because it also gets the calibration wrong. 01:31:23.520 --> 01:31:26.860 In one of our papers, we call it angular visual hardness. 01:31:26.860 --> 01:31:39.060 We proposed a new angular measure for detecting these hard examples that was more robust than the standard accuracy scores and the confidence scores that the systems are built on. 01:31:39.060 --> 01:31:42.400 And we also have some mathematical intuitions for that. 01:31:42.400 --> 01:31:50.600 But the outcome of this is we can now look at, you know, examples which are hard for the current classifier, right? 01:31:50.600 --> 01:31:59.280 So in this gender detection example, it would be most likely the transgender people who would fall as hard examples because it's not getting this accurate, right? 01:31:59.280 --> 01:32:08.540 No matter what binary gender the person is assigned to and if they don't kind of, you know, display that gender, right? 01:32:08.540 --> 01:32:13.780 So there is that issue that can be detected through these mathematical means. 01:32:13.840 --> 01:32:17.880 And then I think at least there is now the problem that has come up, right? 01:32:17.880 --> 01:32:24.020 That not every example is equally treated and there are hard examples for this classifier. 01:32:24.020 --> 01:32:27.440 We can go back and then design a system to correct that. 01:32:27.860 --> 01:32:39.380 And on a related note, I will give a vague answer to this before I actually ask an even more daunting or deeper question. 01:32:40.140 --> 01:32:49.360 Maybe an email, for example, idea of using more unsupervised approaches might be some kind of a solution for future generations of AI algorithms. 01:32:49.360 --> 01:33:01.300 Maybe you wouldn't need to categorize yourself as a person, but instead some kind of more unsupervised continuous representation of social entities might be a way to approach this problem. 01:33:01.300 --> 01:33:13.200 This is not going to happen anytime soon, so I don't really have an answer for today's problems because we know that gender is reflected to other features, all other features as proxies as well. 01:33:13.200 --> 01:33:18.680 Because when we think about gender, that's the largest social group in society. 01:33:18.680 --> 01:33:22.080 So it's one of the biggest proxies in the data sets. 01:33:22.080 --> 01:33:30.260 But my more concerning question is semantics, gender, and race is defined in categories. 01:33:30.260 --> 01:33:47.820 So it's a proxy, but when you think about it, essentially race doesn't really exist or gender might not even exist, but we have it in our language, in our linguistic system, and that's the way we are constructed to deal with the world. 01:33:47.820 --> 01:33:56.960 So how are we going to change these problems that are social constructs in AI systems that we are building on these social constructs? 01:33:56.960 --> 01:34:06.080 So it's a longer-term problem that might require longer-term solutions, which might not be very helpful for your question. 01:34:07.100 --> 01:34:17.980 Yeah, that really gets us something unfortunate, and a lot of people in the Discord are asking applied questions that I haven't raised because I know that the answer is, we don't know how to do that. 01:34:18.040 --> 01:34:20.780 We would love to know how to do that too. 01:34:21.500 --> 01:34:33.440 So one example question is, if you can't bring in experts or social scientists or stakeholders, is there an algorithmic way to kind of pick up on the bias? 01:34:33.440 --> 01:34:37.780 And the answer is, well, there's 50 and we don't know which ones work better in different contexts. 01:34:38.080 --> 01:34:43.720 I mean, but there is an algorithmic way to educate yourself, and that's like search engines, right? 01:34:44.380 --> 01:34:45.360 That's what? 01:34:45.560 --> 01:34:52.920 That's like, that could be search engines, just like, you know, go type, like, you know, if you want to learn about bias, go read Aileen's papers. 01:34:52.920 --> 01:34:58.320 If you want to learn about, like, you know, bias, go watch, like, Timnit Gupru's, like, workshops. 01:34:58.400 --> 01:35:10.020 So I think a lot of this hands-on things, people, smart people have already put out content, and it's just like a matter for us to spend, like, cycles to educate ourselves on. 01:35:11.060 --> 01:35:22.660 Yeah, and as a, like, very basic kind of check, like, you can look at the output of your system, if it has to do with people, and just look across intersectional groups. 01:35:22.660 --> 01:35:41.040 So, like, you do, like, assuming that you have gender data and race data, maybe you have income data, or maybe you have age, or maybe you have, like, some other kinds of, you know, dimensions that describe people, is just looking at disparate outcomes within each thing, 01:35:41.040 --> 01:35:46.960 and then combining those in various different ways to see if there are disparate outcomes amongst the categories. 01:35:46.960 --> 01:35:56.140 Because, like, the theory of intersectionality from black feminism says that, like, these categories compound in, like, a non-linear way, right? 01:35:56.140 --> 01:36:04.380 So being, experiencing sexism as a white woman is fundamentally very different from experiencing sexism and racism as a black woman. 01:36:04.380 --> 01:36:16.100 So, I mean, I think that's a very basic kind of self-audit that you can do when you're working with data that's about people, that, like, you don't need all that much, like, special training, necessarily. 01:36:16.100 --> 01:36:17.860 But, I mean, I think other people are working on this, right? 01:36:17.860 --> 01:36:30.680 I mean, I feel like you guys would know this better than me, but, like, folks have been working on things, like, where you would substitute out, like, you do, like, a sort of counterfactual, and you look at, well, what if this data point, like, they have this outcome, 01:36:30.680 --> 01:36:34.980 but what if they had, their race had been white, would they have had the same outcome with my system? 01:36:34.980 --> 01:36:38.360 So I think people are working on those kinds of methods. 01:36:38.360 --> 01:36:41.940 I think just a really basic one is just do a kind of an intersectional audit. 01:36:44.480 --> 01:36:49.860 Well, Stella, I, for one, look forward to reading your work when it comes out. 01:36:50.280 --> 01:36:51.800 Thank you so much for joining. 01:36:51.980 --> 01:36:53.920 Thank you guys so much for joining us. 01:36:53.920 --> 01:37:02.660 We've gone horribly over, but I've had a wonderful time talking to all of you, and I know that the people listening really enjoyed it. 01:37:02.660 --> 01:37:17.660 If anyone wants to continue this conversation who's listening, if you head over to the DEF CON Discord, there's a voice channel where I'm happy to continue to talk to anyone who wants to ask questions or get references to papers that people brought up and such. 01:37:17.880 --> 01:37:20.780 Thank you again to all of our wonderful panelists. 01:37:20.780 --> 01:37:23.480 This has been amazing. 01:37:23.480 --> 01:37:24.900 So thank you very much. 01:37:24.900 --> 01:37:25.860 Thank you. 01:37:25.860 --> 01:37:26.540 Thank you. 01:37:27.100 --> 01:37:29.740 Wonderful to talk with all of you. 01:37:29.780 --> 01:37:30.480 All right. 01:37:30.480 --> 01:37:31.560 Bye. 01:37:33.020 --> 01:37:45.900 I was also going to say we've also, we're going to have a journal club focused on ethics and privacy with AI starting on the 18th. 01:37:45.900 --> 01:37:54.220 It's organized by Suhaka on Thing, and you can check out our website for a link to a description of the whole thing.