All right, welcome. This is the AI Village panel on AI ethics, where we're going to discuss the
ethical implications of AI technologies and especially their deployment both in general
and also specifically in data science and computer security. We have four lovely panelists
from a variety of backgrounds here to discuss the topic with us. Catherine is a HackerMOMA
scholar and artist who focuses on feminist technology, data literacy, and civic engagement.
She wrote a 2020 book called Data Feminism, co-authored with Lauren Klein, where she
charts a course for more ethical and empowering data science practices. Catherine is an assistant
professor of urban science and planning at the Department of Urban Studies and Planning at MIT,
where she's also the director of their Data Plus Feminism Lab. Ram is a data cowboy within the
trustworthy ML team at Microsoft, working on the intersection of machine learning and security.
His team's goal is to empower engineers to develop and deploy ML systems securely.
Ram is also an affiliate at the Berkman Klein Center at Harvard University, where he explores
the societal implications of adversarial AI. Ai-Lin is an assistant professor of computer
science at George Washington University and at the Institute for Data, Democracy, and Politics.
She is interested in AI ethics, bias in AI, machine learning, and the implications of machine
learning on privacy, security, fairness, and society. Much of her research is on methods to
interpret AI and gain insight into the bias in both artificial and natural intelligence.
And she has worked to demonstrate how human-like biases can crop up in natural language analysis
due to the statistical regularities of language. And finally, Anima is the Bren Professor at Caltech
and the Director of ML Research at NVIDIA. She was previously a principal scientist at Amazon
Web Services. She's received several honors, such as the Alfred P. Salone Fellowship,
NSF Career Award, Young Investigator Awards from DoD, and several more. She is part of the
World Economic Forum's Expert Network and is passionate about designing principled AI
algorithms and applying them to interdisciplinary outcomes. She has been outspoken on AI bias and
the need for democratization and focuses her research on unsupervised AI optimization and
tensor methods. They are all wonderful people and I can personally vouch for the fact that
super cool research and I'm more than thrilled to talk with them all.
So to kind of get us started, there's been a lot of conversation, at least recently, about AI bias
and misuse of AI in the news in the United States recently. A lot of tech companies have talked
about how they're going to stop using facial recognition algorithms or stop providing them
to police to use. And there's also been a lot of conversation both in the academic literature and
also in the national news media about biased AI, the AIs that learn to be racist because they look
at racist data or learn to be sexist because they look at sexist data. But AI ethics is kind of a
much broader topic than that. So to kick us off, can each of you say a little bit about what the
scope of the field is to you and how it relates to the work that you do? Ram, why don't you give us a start?
Oh, thank you. First of all, I'd like to say that I'm not an AI ethics expert. I'm so glad that Anima,
Catherine, Eileen are here along with Stella. So you should be getting like solid like stellar
advice from them. So whatever I'm going to talk is through my lived experience and interacting
with folks like Kendra Albert, Jonathan Penny from Berkman and along with Mary Gray and a whole
host of other folks. So that's my, you know, that's my preface for this. So for me, like
I've been looking at Berkman at the, you know, what is adversarial ML? How does that impact the
experience of citizenship? So one of the things that I, you know, for me, like how I, you know,
how we've been interpreting this topic is moving away from just like an, you know, like the
constraint of like ethics and thinking about power structures, especially in the context of adversarial
machine learning. So turns out that, you know, machine learning is political in a Langdon Winner
sense and security has a lot of like implications, like, you know, if you think of the moral character
of cryptography by Rod Wave. And adversarial machine learning, which is what I work on as a
researcher, sits in the intersection of these two topics. So thinking through who is securing like
the ML systems? What is securing? Who are we securing it against? Who is the attacker?
Are we, is the attacker just like anybody who compromises the confidentiality, integrity,
and availability? So those are kind of the broader questions that I'm looking at at Berkman.
All right, Ellen.
Oh, Stella, thank you for the introduction. And I'm very excited to be part of this amazing panel
so that I will get to hear all of your ideas and vision on this topic. My research focuses on,
working on the implications of machine learning on privacy and security.
Since machine learning or AI technologies are double-edged swords, when they have security
enhancing properties, they usually end up infringing upon our privacy. And starting from
there, I started slowly seeing the link between privacy and fairness and how when our data is
exploited, abused, or collected by third parties, it can lead to fairness concerns at the group
level. And nowadays I'm focusing on understanding how AI systems learn bias or essentially how
systems can learn some form of intelligence or learning ability and how that relates to bias in
natural and artificial intelligence. And when we looked at natural language systems,
for example, word embeddings or today's neural language models that can generate text that is
almost human level, we see that these systems are learning human-like biases from society
very accurately, especially when compared to validated social psychological methods such as
implicit association tests. And then these systems are deployed at the world scale impacting
billions of people every single day with their consequential decisions or biased outputs.
And this ends up turning into this AI bias lifecycle, which is also a feedback cycle
of AI ethics as well. And I'm interested in understanding all of the components in this AI
ethics lifecycle, the role humans play in this lifecycle, how we can discover biases or harmful
side effects of these AI systems, and what we can do to mitigate these harmful effects and prepare
for upcoming problems that might have already been caused by AI systems. And during today's panel,
we will go into the details of all of these single components, starting from the most technical
atomic details of machine learning, going up to probably some philosophical questions at the
high level abstraction. Thank you Aylin. Anima? Yeah, I completely agree with Aylin and Ram.
You know, it's to me exciting times to be an AI researcher, but also, right, I'm filled with
dread that where the world is moving and how fast we are, you know, moving in directions that I find
problematic. And to me, the source of it all comes to the foundations under which this deep learning
revolution is built on, right? So I'd like to emphasize when I'm giving talks on AI about
this trinity, you know, the data, algorithms, and computer infrastructure coming together.
And that's what allowed us to have this fast progress in deep learning. But most of that
progress has been with supervised learning, right? The availability of large-scale label data starting
from the ImageNet data set was the start of this revolution. And in a way, that is quite
straightforward. Once you know that you want to optimize for accuracy and you have the label data,
you can kind of quickly build these systems. But the problem is sometimes we want not just the
accuracy, but like robustness, right? Like what Ram was saying, what are the different changes
beyond the training data? And when our training data is biased, how does this affect on minority
populations that are not represented in training data? And part of it is not just the data imbalance,
but also the algorithmic bias. You know, if you are on Twitter like me, and if you follow the
Twitter storm on the presence of algorithmic bias on deep learning, you know what I'm talking about.
And just to the gist of it, the idea is I think some proponents of deep learning do not want
anything negative being said around deep learning. But I do think these very large-scale models,
highly nonlinear models, are the source of the problem. Because the more nonlinear it gets,
the more brittle they are, the more they amplify the bias, right? There is no free lunch. You know,
you cannot have it all in one mechanism unless we are thoughtful in terms of the objectives we build
in, the kind of data we use, and also the kind of testing we do. And so what I work on is unsupervised
learning. And I think to me, this is the core for us to move away from these highly biased models.
You know, to take a simple example, you know, even before the deep learning revolution,
I was working on topic models, right? Where we wanted to categorize large corpus of documents
and there's no label data. And so now with these unsupervised learning, we are suggesting these
are possible topics, right? And these are the words representing the topics. And then it's up
to the human to interpret and judge what are relevant topics, you know, because there are
weird correlations in text. So there can be lots of topics that are not human interpretable. And so
human in the loop with unsupervised learning, I think is the key to get out of this place where
we are stuck with biased models. And, you know, the current revolution now is unsupervised. Now
we're seeing a lot of progress in self-supervised learning and robustness. Hopefully that'll lead
us the way forward. Thank you. All right. Thank you. And finally, Catherine,
where are you coming from with this? Yeah. Thanks so much. Thanks, Stella,
for the invitation. It's a super pleasure to be here with you all. Yeah. So I guess
where I'm coming from is not as much of a focus on AI, although some of the things that we say,
many of the things we say in our book, Data Feminism, apply, but we sort of enlarge the
scope more broadly to be data writ large. And thinking about also, you know, one of the things
that's happened as this conversation has grown around fairness, accountability, transparency,
bias, ethics, and so on. You know, there's been a lot of pushback on just like AI and data are
great. They're working great for everybody. And our contribution with Data Feminism is to say,
hey, actually, feminism as a kind of body of knowledge, as a kind of intellectual heritage,
which has theorized, philosophized, and also done a lot of writing and sort of action,
like social movements, action around the topic of inequality has a lot to offer us in terms of
conceptual models for how we can think about this current state of affairs. And so that was sort of
the premise of Data Feminism is to say, hey, let's look over here at feminism. What do we,
what can we bring from feminism to understand how we might work with data science in a more
ethical way, and specifically taking structural inequality, which is basically like power
differences between social groups as kind of our main locus of attention. And so when you look at
that, and you look at things like, you know, one of the things I guess that was irritating, I guess
I'll say as we were writing the book, which I'm sure folks here are also irritated by, is that often,
you know, when we have a biased, whatever, algorithm or something like this, like a resume
screener, let's say, that is favoring men over women, you know, the media reports on it with this
thing of like surprise, like, oh my gosh, like, how could this be possible? Like, how could there be
bias in a machine? And we're like, oh my gosh, Data Feminism has so much to say about this.
And so ultimately, like what the book does is it kind of backs us up. It's in a way less in the
details. It's not providing you with a way of like systematically fixing your data set or anything
like that. But it backs us up, like way further back in the processing pipeline to say, well,
like, here's why these things surface. They surface because of hundreds of years of inequality.
They surface because of really flawed models of people like women not being considered fully human
or people of color not being considered fully human. And to really take, you know, if we aspire
to build something that is ethical, we have to look and we have to undertake that with a different
kind of time horizon. And that has to be built into everything that we do. So we're trying to
kind of both say like, hey, let's look back at the root causes. What causes structural inequality?
Like, why do we end up with biased, you know, crime data sets in the first place before we start
trying to like fix them or something? And then how do we use that information to decide which
problems do we work on? Who do we work with? Who do we want to benefit with our work? And who do
we want to just like refuse to work with entirely? Because there are some really bad actors out there.
And so that's... and then we talk a lot about democratizing data, data literacy, and AI as well.
Because that, I think, is really part of the key of how we're gonna get through this. So we're
trying to balance being not only doom and gloom with like, how do we move forward? And we propose
a set of seven principles for how we move forward with a kind of feminist approach to data science.
All right, thank you. So I think a common thread in what all of y'all said is that
there are problems both... problems in machine learning exist both in the pipeline process,
where that is coming from, how it's processed, but also... but that that isn't all of the...
all of the problems. A lot of the time in the media and in conversations, people have this idea
that data is like some kind of objective truth, and that when you come up with biased results or
incorrect results, it's because the analyst messed up somewhere. So, you know, do you think that's
true? Spoiler alert, I'm pretty sure you're all going to say no. But, you know, what would you say
to someone who thinks that way? And like, what is... what are the problems with thinking about
data that way? Whoever wants to kind of get started. I can quickly start with a positive
note. Data certainly reflects the state of the society, but it does it in a defective way. So
we are not getting a perfect representation of society, especially with regards to minorities,
because they are underrepresented. But when we have such data sets, at the same time,
one advantage is that we can also study the historical injustices these minorities,
underrepresented or disadvantaged groups have been going through, because it's reflected in
the data set. As I already mentioned, since these data sets are a reflection of society,
and then when we start building models on these systems, then we have data that is reflecting
these biases. But at the same time, it also reflects some facts about the world, such as
statistical biases, inductive biases, or some kind of facts that can also be useful.
But that does not justify the fact that there are all these harmful information that gets
very accurately embedded in these systems. So this is my double-edged view on this
data topic for now. I can follow up with that. I completely agree with Arlene. I think
the problem is how the data on the internet is reflecting the society, but only in very
balanced ways. And this gets amplified when you want large data sets. The latest
one in this aspect is the GPT-3. It's trained on links from Reddit. I'm almost never on Reddit.
I've only been a victim of trolling and gun threats on Reddit. I stay away. I don't know
any woman in machine learning that I'm aware of who is active on Reddit.
So you can see that even among all the imbalance in the internet,
the Reddit is even further amplified. So the decision to go for Reddit links and claim
that's what we want to build the largest language model on, we can spend tens of millions of dollars
on this data set and nothing else, is to me extremely disappointing and disheartening that
the VCs, the decision makers, they're always going for the worst of the toxic forms of
data that's available on the internet. And I really can't justify this. I don't know how they
sleep every night. I don't know how they can say with a straight face that this is the path
towards AGI. I mean, this is nonsense, right? So I get frustrated all the time. I know Aileen
was trying to paint a more positive picture. I'm just extremely disappointed that I'm sorry for
being open about that. Yeah, well, maybe I'll pop in on the negative note.
I mean, what happens that I think we're pointing to here is that data are very resource intensive,
right? And so it's only certain actors, it's something we say in the book, it's only certain
actors in the world that have the resources to collect, store, maintain, utilize, analyze, deploy
in some product or whatever, these kinds of data. And so at root, you only have certain actors in
the world that are able to actually do that work. And because data are so expensive, we see all
these kinds of shortcuts, like exactly like this, where it's like, oh, like, where can we get a big
data set? Okay, Reddit, or like, oh, you know, where can we get images to train image, you know,
face recognition algorithm? Oh, okay, like celebrity faces or whatever. And just like not
connecting the dots of again, like, in what ways like who is not represented in those data sets.
And there's like all these multiple points of kind of where inequality kind of enters that
pipeline and like not catching it at any of those points, because then the people that are working
with the data are often they are the white elite, non-immigrant men, you know, and they
are representative of that demographic that are celebrities, right? So like, you know, that you
have these kind of multiple stages at which those things don't get caught. But I mean, at the root
of it, I often think of the constraint in a lot of ways is, it's financial, like folks are looking
for a shortcut. I mean, I'm so glad, you know, Catherine, you brought the issue of like celebrity
data set. So in the news event, you know, you've heard about Fox, which is kind of like a cloaking
tool, you know, for privacy, it uses like very fundamental techniques and adversarial machine
learning so that, you know, people can kind of, you know, like save their privacy from like terrible
tools like Clearview AI. And Kashmiri Hill, kind of like the New York Times reporter, tried it out.
It was so weird, you know, she got she got some really weird photos. And when she asked the PI
of that project, like, hey, why are these photos looking really awry? I thought the whole point is
to keep the face same in the cloaked picture. And he said something to the effect, and I'm
perhaps going to misquote him, he said, Oh, yeah, women get mustaches and like, you know, men get
eyelashes. That's how you kind of like cloak. You know, that's kind of like an outcome. I'm sure
there's a whole like, semester of SDS gender studies to unpack there, because they use like
celebrity faces. And like, how many transgenders are represented in like celebrity data sets?
My first thought was, have you seen the winner of Eurovision 2019? Like, shame on you. Have you
seen like RuPaul's Drag Race? Shame on you. But, but yeah, it is. And I'm so sorry, Eileen, that,
you know, we're not like raining on your parade. But even though, even though somebody starts with
really good intent, and I'm sure like the Fox team did it with very good intent of like, I am going
to, you know, forward privacy, I'm going to like, you know, empower people. But like you said, like
when they don't connect the dots, or like how Anima said, when they take like shortcuts,
yeah, it doesn't lead to less than like harmful outcomes.
So just as a quick reminder for the people listening, Fox is the paper that we discussed
yesterday at Journal Club. So if you were there for that, awesome. If not, you can read about the
paper. You can listen to me and a couple others talk about the paper I was talking about on our
YouTube channel, where it should be up by now. Sorry, Eileen, did I cut you off?
Oh, I saved my pessimistic comments to the end.
But now we have our feedback loop completed. So I'm really curious about all the data practices
that are going on out there, or the data impractices or abuse, because as I said,
I'm coming from a privacy background, and how people's data is abused, so that they can be
de-anonymized, they can be tracked, and essentially, entities that hold power can use this for
surveillance or for large scale manipulation. And unfortunately, as citizens, or as human beings
living in the world, we don't consent to our data being collected, but we don't have an option to
opt out. So all of our data, under the premise of free applications, free products, or just walking
on the streets in country X, is being collected by these entities that hold power to build large
databases of any kind of information. And in some cases, this data is not being even used,
because they think that we don't have a way to use it right now. But in the future, we will
figure it out, figure out a way to make use of this data to make one type of consequential
decision again. And this is extremely problematic, because the citizen is not informed about these
practices and what the long term effects might be on individuals and society. And on the other hand,
when we look at other power holders, such as proprietary software, or all companies that are
providing these services for the entire world that are free, supposedly free, are making so much
money because they are making use of the data that we are providing them. And after that, they keep
building stronger machine learning models so that they can keep amplifying these biases in their
direction with their targeted ads, or whatever their incentive is. And here there is a serious
problem about the incentives of these systems. And both Catherine and Anima mentioned that,
for example, we have these objective functions optimizing for accuracy in these systems
that actually are helping these power entities. But the objective function could also be how can
we use these data sets for social good, as opposed to optimizing for accuracy. So there are so many
different faceted and different levels of abstraction of problems involved in this
data ecosystem again. And I guess we can keep discussing these in more details,
but today the goal is kind of give a glimpse of what is going on in this area.
So Aileen, you picked out... guess wrong? Oh sorry, I just have like a quick follow-up for
Aileen and Anima. I know that both of you have kind of revealed the shortcomings of ML systems.
Anima, you've done extremely awesome work about how GPT-3 is flawed. And Aileen, your groundbreaking
work on word embeddings. And I know that both of you have communicated that research to the
community in a very approachable fashion. But I don't know about you, Aileen, but I visibly see
the kind of like negative, rancorous feedback that you get. And from very well-intentioned
research, I mean I don't know well-intentioned, from like people who are in the ML community,
why do you think that happens? And Catherine, like from a feminist perspective, why are people from
the own educated community like unloading on these researchers? I would love to get your
perspectives on this. Sorry, Stella, I didn't mean to like... No, that's fine. That's very close
to where I was going to go anyway. I mean, just maybe I can get started on this, right? I mean,
to me, I think it's at the root of human psychology. Like no one wants to be the bad guy.
No one wants to think of themselves as the bad guys. So they believe, right, they are going
towards AGI through this GPT-3 model, for instance, right? I mean, I think they truly believe.
So let's assume that. And I think the thing is they really don't want to also hear anything
that may harm that, right? So they believe anything that may harm that path is that are
their enemies. People who are voicing opinions against it are their enemies. And so that's why
we see this vicious pushback on social media platforms like Twitter and, you know, and then
they have huge follower counts. So that builds up, right? And all they have to do is
do it, just kind of give their followers a sense to go after people who are bringing up these
difficult issues. And honestly, because there's also a lot of insecurity, they don't know the
answers to this. They don't know a way out. And they worry that their entire existence is in
question, which I think that's why, you know, to me, I feel like we are on the right track. So we
have to keep pushing this on and increasing awareness, have more allies. I think we will
get more support. And it's already been amazing over the last few years, right? I mean, it's
happened so quickly, even just the mention of AI bias was not there. It was completely dismissed
in machine learning conferences. People working on it were considered not technical, you know,
were not respected. There was a lot of gaslighting, you know, because it happens to be women, it
happens to be minorities. So there was even more of all that racism and sexism coming into play when
such research was reported. But now, you know, especially with the Black Lives Matter movement,
that's when these companies finally said, we won't be selling face recognition to law enforcement,
right? We've been asking for this, you know, we've signed letters, we've gone to the press,
we've been doing this for a while. And it's been led by black women, which is amazing.
But, you know, it took a broader global movement to make this happen. And I think we are building
this momentum up, because it's not just about AI, it's not just about data ethics. It's a much
broader question for humanity. And I think we're, you know, I have to, I remain hopeful.
So since Anima brought this up, I wanted to take a moment and ask you guys, do any of you,
as yourselves personally, or the companies or institutions that you represent, have any
policies about what types of organizations or what types of tasks you will not do ML research
or data research for? You know, be it police applications, military applications are two of
the ones I know are really popular. I mean, speaking personally, disclaimer, sorry, before
I say this, my company has a large number of lawyers who'd like me to tell you that my reviews
on the stream represent myself and in no way represent my company at all. I work for a US
government contractor. And a lot of the people who work at my company do ML research for the
Department of Defense and the US military. And that's a very contentious topic, both within my
company and I know within the ML community at large. So, you know, I have been personally trying
to decide how to navigate that line myself. But I was wondering if any of you guys had
particular stances you've developed? I mean, I'll just say, for my own work,
and I'm not speaking about MIT, because obviously MIT is all wound up with all sorts of different
things. But at least as a professor, you have autonomy to do what you want. And so at least
from my perspective, I've decided to only work with folks who are doing some kind of counter
hegemonic work. So activists, journalists, social movements, community based organizations.
That's also kind of the, like, that's also in my research interest is to think about, like,
how do we think about building data literacy in those kinds of sectors as well. So it's like a
really nice fit. And that happens to be where my interests are. But I'm sympathetic to the stance
that like, yeah, when you work in a large organization, as many of these places are,
you have a lot less autonomy and thinking about, well, like, how do you navigate that as an
individual? How do you navigate that as a team? And how do you navigate that at the level of the
institution? But I will say, like, I am really energized by all of the organizing and activism
that's happened within large companies right now as well. And so I think that is a huge,
potentially, is a huge force for change as well. Yeah, I agree. And I want to add what Stella
said earlier, right? I am not representing NVIDIA or Celtic for that matter. But I see a lot of
support building up from the grassroots in these organizations. And NVIDIA being the platform for
AI computing, you know, we are not directly selling these kind of AI applications, right? So it's
really our customers and partners. And so there is less of a control in sense of everything the
others do, but there's always periodic reviews. And, you know, there have been discussions
internally in terms of how to make sure that our platforms are being used for good causes rather
than bad ones. But I agree, there is a lot more work to be done. But I'm very energized by activism
internally at NVIDIA, and also at Celtic, especially the students, everything, you know,
from the recent Black Lives Matter, how it was really the students who have been
energizing this movement. And so that gives me a lot of hope.
Yeah, and I'm happy. Oh, sorry. Go ahead, Aileen. Please.
I was just going to say that I'm not representing anyone except my humble opinion that might not be
very well formed. But I try to build my own ethical framework when approaching these issues,
since we don't have the AI ethics framework that is widely accepted by anyone. I basically
try not to work on any confidential project. I don't try, I haven't worked on any confidential
or secret projects. And other than that, I try to get involved in pro bono activities to help
companies, entities, research their systems to understand what kind of negative side effects
they might have, especially with regards to bias, can we come up with new forms of ways to measure
bias, develop methods. And that has been my approach to this issue right now. But if we had
a well accepted code of ethics among researchers, I think everyone would benefit from such a framework.
I mean, I totally agree. I think one of the things that's kind of like helped me personally,
again, you know, boilerplate, boilerplate, I'm only representing myself. When Anima said like
reviews that kind of like really resonated with me. There is like, within Microsoft,
there's this like committee called the AI Ethics and Research Organization. It's called ETHER.
I'm perhaps butchering the acronym. But one of the things that they do, and it is like,
you know, folks from MSR, you know, and it's just not the ML folks, but MSR is such a vibrant
community. So, you know, you've got like sociologists, you know, folks from like,
who think about bias from MSR New York, in Eric Horowitz's organization, kind of like,
you know, making sure for critical like ML applications, you know, you need to get ETHER's
sign off. So that has been a, you know, a guiding light. The second thing for me is like, you know,
it's, I completely agree with Eileen, if there's some like, like a framework, and I, you know,
that way you can meet it out to engineering engineers. The thing that's kind of worked for
me is that the standards of business conduct at Microsoft is surprisingly good, which is a very,
which is a very good thing to say, because they make it, they shoot it like Netflix style videos.
So it's not the boring training that you have to kind of go through. I mean, I mean, I've worked
in Microsoft, like, in 2013, it was just like the most like boring thing, but you still have to do
it. It's like a read through it. But now it's like an actual like Netflix series, they've got the
same actors coming in different years. But I mean, long story short, there was a module about how
this guy tries to train his model on this, like private customer data, and how he gets into such
so much of, you know, like, like a big, like big soup, he can't get out of it, you know,
he's trying to demo it and he has to like talk to his manager and managers like, oh my gosh,
you're gonna like, you know, lose your job. So they really rain it down on you in a more relatable
fashion. So all of this to say that, you know, having this, you know, this committee to kind of
like guide for critical like applications, and kind of having this like relatable, you know,
ethics discussion has kind of like helped at least me and my team personally a lot,
quickly ask Ram, so can they release this to the public? I think
immensely from this as well. Yeah, I really hope that they do because it's,
it's really captivating, like you're not bored. You know, when you're watching that.
So I have a question from the discord that is something you guys have all touched on a little
bit, but I think it's worth like directly addressing, which is that so we've been talking
about how we were, we all represent large companies, or major universities, you know,
institutions that have power and money and resources. And, you know, how, what is it?
So what the question asked on discord was, was, you know, if, what's the role that
these large institutions is powerful and franchise institutions play? And, you know,
if the ability to collect large data sets or train these very large expensive models
were somehow democratized, would that improve the situation? Ram, I think we should throw this over
to you, because I know you've recently written a paper on this topic. Yeah, that's a really
hard question for me. You know, just because, you know, again, from, from a, from a, from a
winner sense, like, is machine learning inherently authoritarian, you know, because like, or is it
inherently democratic? And, you know, there, there are arguments to be made on both sides,
like, you know, like Anima started off with data compute, and the algorithms, that knowledge is
kind of centralized in a lot of, you know, in a lot of aspects, right? You, you know,
like, $10 million or so with their TPU training pods, which, you know, like a mom and pop shop,
like an individual researcher is not going to have. And data, as we all know, like Catherine
kind of like outlines in her book beautifully, like how it is really, it's situated with power,
almost. So, you know, in a, in a, in a, in a land and winner sense, it really does,
you know, one may almost say that, you know, machine learning is authoritarian. On the other
hand, actually, one of the other papers that we read was Anima's Born Again Neural Nets paper,
where she shows that, you know, a student can actually in certain like conditions outperform
the teacher. And, and if you think about the revolution that transfer learning is bringing
along, you know, you, a company can put out this like pre-trained model, and then you can just
retrain the last layer for your like custom, like applications. So, and, and now with like,
you know, systems like fast.ai and like, you know, they're bringing like machine learning,
like two programmers, which again is not making it democratic. So it's almost like,
you know, you kind of trying to install solar panels in your, in your house, as opposed to
depending on like a nuclear power station. So I completely struggle with this question. I keep
vacillating back and forth. And unfortunately I do not have like, I'm not, I'm not, I'm not like
settled on an answer. And perhaps like other panelists will guide us, will guide us through
this cave. Thank you Ram for bringing up the paper. I think to me, I, you know, I again pin my
hope back on the algorithms, right? So I think we are just at the infancy in terms of algorithmic
development for deep learning. We've been using the very basic algorithms. The question is,
you know, where to go next? You know, I mentioned unsupervised learning earlier as a broad topic.
I think within that, what we want is more controllability, right? Like for instance,
generating text. If you could control what needs to be generated, then you can try, you know,
head towards removing these biases, at least in what's generated, even if it's inherent in the
model. And in fact, that's what we need in practical applications. You know, we don't want
it to write weird poetry. We want it to maybe, right, give us knowledge facts or ultimately
program, you know, do program synthesis and so on. And so that's something we've been working on.
Same for text, same for images, right? Ultimately the GAN should be able to have controllable
generation and it can control various attributes, right? And that way then it's really at the hands
of the human on how to use these models. And currently we don't have good frameworks for that.
And I think unsupervised learning is the key to enable that because you have to learn what are
the hidden variations in data and then how to control and change that and be able to generate
new examples. Yeah, so I think that's the direction to go forward. I completely agree
with Anima, by the way. She has been mentioning unsupervised learning. I also think that the
future is probably in unsupervised, self-supervised learning that is a bit maybe more intelligent,
depends on how you define intelligent, because again, this is semantics and a vague term, but
it seems like once we get there so that the system is able to figure out this universal
learning pattern somehow, and then starts figuring out different kinds of tests by itself,
we might be able to somehow incorporate some kind of awareness to the system. And one option would
be a human in the loop, for example. And again, Anima mentioned this a few minutes ago, saying
that human psychology and awareness plays a very important role here. Like, we are not able to
agree on many of these questions, because we are dealing with ethical questions. Everyone has
their own understanding, their own answer, and their own desired fairness notion. And we can't
satisfy all of these at the same time. However, if we are able to bring some kind of awareness
in these systems, then we might be able to change actions and decisions. Because,
the evolution of how people are reacting to biases depends on how aware we are of our biases.
It's not like people are trying to discriminate other groups. It's mostly caused by favoring the
in-group. Like, there are no harms that are explicitly trying to be imposed upon other
groups. But this unintentional bias ends up causing all of these harms. So, in an unsupervised way,
we can figure out some kind of an efficient learning pattern, where we can also incorporate
ethical mechanisms, then we will start getting there. But of course, all of these will require
a lot of informed discussions from many different perspectives, and many different disciplines as
well, because this is affecting all disciplines, all the sciences, and every social fabric.
And yeah, maybe I'll just, I mean, I think the question of, you know, is machine learning
authoritarian or not is a super interesting one. But ultimately, the way that we would answer it
from a feminist perspective is, in whose hands is it? So, you know, so feminism, basically,
what feminism brings is who questions. So, like, who is using it, for whom, to support whom,
to oppress whom, and so on. And so, I mean, I think that, for me, is the,
those are the questions to be asking when we're thinking about, like, democratizing AI. Like,
I think it's lovely to think about making it easier to use models, democratizing access to
data and computational power. Those are all great things as, like, a kind of infrastructure
of democratization. But on their own, they won't create a democratic kind of usage of AI,
because we also need to do the bridge to the groups that could really actually benefit from AI.
And for me, like the, again, like I'm saying, like, I'm interested in sort of counter-hegemonic
groups and thinking about, like, what are groups that actually hold power accountable? They're
groups like lawyers and legal scholars, journalists, community organizations, social movements. Like,
these are the folks who need to be kind of trained and brought to the table, if we're
actually talking about a comprehensive effort to democratize AI. Because those are the folks that
you want to understand, like, what are actually the creative, creative potentials. And there are
many, like, some of the journalists that have started to discover the power of machine learning
have been doing amazing things, like detecting, using image recognition to detect amber mining
from satellite imagery in the Ukraine, for example. Like, you can do this, like, kind of amazing
accountability work, but not every person is going to have that idea, because they haven't been
exposed to working in this way. So, yeah. So, if we're talking about, you know, AI algorithms
being developed by institutions of power, and the fact that, like, who is using algorithms matters a
lot, is there a role in, like, adversarial AI? Adversarial AI is usually cast as a bad thing,
because machine learning engineers do good, and those pesky evildoers are trying to ruin it. But
if machine learning algorithms are being used for bad, you know, does that mean that adversarial AI
techniques can be used for good? Oh, I love that. Totally. I totally think so, yes, because, like,
there are so many reasons why groups or individuals would need to be protecting themselves, like,
from, you know, we were just talking about Reddit, for example, you know, like, there's so many reasons
why, you know, like, an act of, kind of, an adversarial act is not necessarily bad. Like,
for example, I have a friend who, as a way of, kind of, fooling the just rampant data capture
that's done by promotional ads and things in the browser, and so, like, the browser is always sending
the IP address and whatnot, and what it does is, instead of, like, sending what it clicks on,
the browser clicks on, it actually just sends every link on the page. So it's, like,
hacking it through obfuscation, you know, and so it's, like, I think that's a brilliant technique
for then preserving privacy. So you're obfuscating what the user is actually doing, and screwing
with the ad systems or whatever. And so I think, especially in situations where power is not equal,
which is every situation right now, like, we also need, like, good, like, white hat adversarial ML,
basically. I love this, Catherine. I got so excited for the ad blocker research. I mean,
right now, in the field of adversarial machine learning, the use of adversarial machine learning
to enhance the experience of citizenship is still nascent, but, you know, there's fantastic work
that's happening. I'm so excited for this. Bogdan Kulinich, I'm sorry if I'm pronouncing his last
name incorrectly, developed this paper called POTS, which is Protective Optimization Techniques.
We're into, kind of, like, empower. So what they do is, it's really cool, it turns out that,
obviously, credit card scoring is biased against minority population, and they show, like, how
you can poison the system tactically, like, do it, literally launch, like, a poisoning attack
to, kind of, like, you know, make it better for the target population. There's really another
cool work from some researchers who found that Facebook's ads platform is micro-targeting,
and they, again, use adversarial ML to show, like, you know, faults in that. So that opens up, like,
exactly, like, how Catherine says, white hat adversarial examples. Unfortunately,
you know, it's not the dominant thinking. In fact, DARPA, when it released its
guard challenge, which is, you know, securing against, like, AI challenge,
they said, we need to protect ML system from attackers. And they do not define who attackers
are. So that's... And I completely agree with Ram and Catherine. And I think there's also,
right, because we don't know who are the attackers, I feel like there's also people
are going in all kinds of weird directions with adversarial AI research, right? Because they don't
know what kind of attack to, you know, and they're like, okay, maybe I'll do this norm bound or some
other. And then it's like, okay, why does this matter? We don't know, right? So I think perceptual
attacks for images are much more meaningful. I mean, you know, if those that kind of preserve the
for humans, the perception, but, you know, change the statistics of data, right. And to me, I think
the next step is to really have the regulatory agencies, you know, take on these tests. I mean,
we have a long way to go. Because if you look at the NIST, which is the standards data sets for
face recognition, that itself is heavily biased against minorities. But ultimately, the hope is
regulatory agencies in each domain will be using these as tests, you know, especially for autonomous
driving, this is a must, right? I mean, given the long tail, the only way to do is also have very
heavy testing would need a lot of compute, but only then certify that this is roadworthy and
otherwise be very careful about that.
And Anima, to my understanding, NIST has been tasked to develop these AI standards. So they
are especially working on those given the example with the face recognition or computer vision data
sets they had as well. So that was a great point that you identified. And again, there are all of
these privacy enhancing or fairness enhancing adversarial applications of machine learning,
but they might not be very common right now. But the most simplistic version we can think of is
synthetically generating data to oversample minority populations so that we can improve
their representation. Because at least in real world statistics, it's well known that oversampling
minority groups is a helpful approach. But at the same time, again, this calls for the need of some
standardization in this process. And again, this ties back into the question of is only data bias
or algorithms also biased? Why are we creating these algorithms? What are the goals of these
algorithms? Who are the bias power holders implementing these algorithms? And again,
so it really matters. And this might bring us to the Kant question here. Should we have this
ontological approach? Or should we go with utilitarianism? And so on. So there are many
interesting questions in this direction as well that we are starting to just scratch the surface
nowadays, because it's becoming a very hot topic. So why are we even building these systems? Why are
they going to empower at the end? Absolutely, I just want to quickly add to Alan's point on the
oversampling for minorities. I think, yeah, if you don't have more data, I think that's a quick fix
right? That's the starting point. But that itself can also create problems because GANs have mode
collapse. So you just have very less diversity among, say, the minority class, right? For instance,
in fact, it was shown when the images that are generated based on the celeb data set,
the celebrity data set, black women tended to have all the makeup on, right? So it was and
with these face recognition systems, when black women didn't have makeup, it was even worse.
You know, so then there is kind of further bias even within the class because of the lack of
diversity and variation. And that's why I think we need to disentangle all these factors and do
unsupervised learning and learn, is there also enough diversity within the class in terms of
these variations? Yeah, go ahead Stella, sorry. So that sort of feeds into something that someone
in the Slack asked about as well. And it's been in the academic discussion of AI bias a lot
recently, which is how models can exacerbate or create biases out of imperfections. Even things
that aren't normally thought of as imperfections, just the choice of your loss function implicitly
decides how you're going to weigh different samples. And this is crude, but if 90% of the
people in a data set are white and 10% of the people in the data set are black, then an algorithm
that gets a little gain in performance at the cost of a little loss on a white person, at the cost of
a little loss on a black person, is going to perform better on average, if it just mostly
pays attention to white people. So, you know, lost the rest of my train of thought.
How, so it's not, it's not just that when we talk about bias in algorithms, we're not just
talking about the data coming in, but kind of algorithms can create their own sorts of little
biases. Like, what does it look like to combat that? And kind of how does that happen? You know,
from the perspective of a data scientist who's looking at something, it's like, you know, how do
you know that's happening? And how do you combat that? Just a quick anecdote, because I don't have
an answer to your question. A quick anecdote was one of the resume screening algorithms,
it wasn't Amazon, but it was like a different company, when they reverse engineered it to see,
like, why was it preferring men over women? One of the factors that they found, one of the strongest
predictors was like, the name. So if somebody was named Jared, in particular, it was a very
strong predictor of success, as well as if they were a lacrosse player or not on their resume.
And so it's like, just a sort of very interesting thing, where, in a way, what happens there,
like, they're not even they're considering, well, I don't know, I don't really know the model. But
like, I don't think they were directly considering race or gender in that case. But if you look at
those things, like Jared is a very white, like kind of Anglo name. Lacrosse is a very elite,
white, sort of private school sport in the United States, despite the fact that it was
invented by Native Americans, actually. And so like, those things sort of, they seep in,
and then they become these factors, and they become predictive factors. But like, behind those
is always like, is race and gender and all of those things are like, kind of still underlying.
Yeah, these systems learn the associations, and you were both asking what kind of metrics we
should, for example, develop to understand how to improve these systems. So I was wondering,
maybe this exists in literature, but do we have, for example, diversity dispersion measurement
metrics to understand how diverse a group is, so that we can try to improve diversity within one
social group? And then the other thing is, Stella, you mentioned objective functions,
or machine learning algorithms that are trying to optimize based on the data representation,
where people are categorized into groups, racial groups, or gender. And it's not even clear how
ethical this approach is. Should people be represented in categories? Is this the correct proxy
to a person's identity or being? And how should we deal with system design when we are developing
new machine learning algorithms? Because each algorithm depends on these data types, the way
they are represented, numeric values, ordinal, or whatever. And should we come up with more creative
ways to design new algorithms for machine learning to encounter, to counter balance,
these problems? And Stella, I mean, I want to rewrite this question back to Catherine, because
when I was reading a chapter from her book, Catherine, you spoke about concepts which
maintain power, like fairness, bias, and ethics. And you give this amazing framework about concepts
of challenge power. I didn't even think about that. Is that something you want to talk about?
I feel it's a really cool framework for engineers to think beyond the fact star or fact framework
that we're used to. Sure. Yeah. I think that's in chapter one or two. I can't remember. But
yeah, I mean, so one of the things I think that Lauren and I were frustrated by
is that we just felt like fairness, and this is not, I guess I should say a little bit of a
disclaimer, because I actually think the work that's going on in the fact star community is
super exciting. And so like, this is not to say that, like, no one should do that work, because
I am really happy that work is being done. But that's just some of the organizing framing concepts
for what we're talking about aren't, they don't go far enough. And so it's like, instead of fairness,
we propose justice, or no, instead of ethics, we propose justice, like, we're really thinking
about what does it mean to make a just system, which means repairing past harms, like harm has
been done in the world, we come to the world in the bodies, in which harm has been inflicted at
the group level, for some of us for centuries, you know, and so like, how do we build systems that
not only kind of treat us all equally as if we're all the same, but actually take that harm into
account. And so we propose other kinds of like, justice, concepts like...
My favorite one is transgression.
That's what I was gonna say. Yeah, like, kind of like, why settle for accountability,
because often accountability is after the fact, you know, like, why can't we build
systems that liberate us? Why can't we build systems that heal us? Like, I feel like we,
we have the power to imagine just much more expansively. So like, it doesn't always have
to be after the fact that we're trying to, like, you know, repair things or whatever, we can really,
we can really aspire to do much more. So co-liberation means, it's this idea that
none of us are free until all of us are free. So we work with the most marginalized groups
and center, like, this is a case where we would like, build a data set around whoever is most
marginalized in whatever the application area is, and we would work with them directly, and they
would be part of the process. And whatever is built, the system that is built would like,
actively serve their needs in a really culturally grounded way, you know. And so I think that's
totally feasible. It's just not the way that things are done right now.
I fully agree. I think these are great points, right? Ultimately, we have to kind of rework the
whole framework. But I think to begin with, even, right, like, the way these systems are now just
kind of put out with almost no testing, right? And, and then only kind of somebody like, kind
of, right, brings up the problem, usually a minority researcher, and then the company is
doing everything to push back and use all the resources to discredit the researcher, I think,
at least we should change that conversation. And I think that can happen, especially, you know,
with the viewers of this panel, we need more hackers into these AI systems, go hack every AI
system out there, reveal all the vulnerabilities, take it down, rebuild a new one.
Yeah, you can have like a brigade of like algorithmic bias assessors.
I mean, good trolls, good trolls.
To kind of move this towards, you know, advice or takeaways from the audience, because
this has been a question popping up in the chat a lot, you know, so you guys, you know,
you guys are talking about how there's definitely steps that people can take to make progress in the
way that they design their algorithms to be more inclusive and stuff like that. And one thing that
kind of stood out to me is being implicitly present in a lot of what you guys said, but I don't think
anyone said this explicitly, is that kind of the default view of AI is it's supposed to be
descriptive, you take a description of the world as it was, and you put that through an algorithm,
and the role of the algorithm is to make the future look like the past. And so in a very real sense,
AI is intrinsically regressive. But at the same time, nobody, there's nobody saying you have to
do that, you know, you can decide that you want to choose to value certain things more than others,
and you want to kind of bring about certain impacts, and you can proscriptively put those
into your algorithms. Personally, I have tried to advocate for this, and largely speaking,
failed, because people are scandalized by the idea that you might tell the data how it's supposed to
look. But you know, so if you're a, if you're a data scientist, or a machine learning research
engineer, listening to this chat, you know, can, can, how can they proscriptively design their,
their algorithms for equity, for justice, for liberation, for whatever all these very nice
words are, that are better than bias, or fairness, I suppose. You know, what does that, what does that
look like to design an algorithm in that fashion? I mean, I'll start, and then hopefully others can
build. But like, I mean, I think what it would look like is to do, to really think about where,
where what your work is doing is being applied, and on whom, right? And like, who you're making
it for, like, who's your boss, and then like, who are ultimately like the end users, or who
is kind of impacted, who are the decisions being made about because of the system,
and to do basically like a power analysis of that system. And then think about what is,
that application area, who has been systematically disenfranchised, and then work with the groups
who are working towards a better future in that domain to co-create, like what the metrics should
be for the future. You know, I mean, I think that we're not asking, I don't, I don't think it's like
appropriate that like, like a machine learning engineer would like invent the metrics out of
their head or something. You know what I mean? Like in a domain space, they may not know very
much about, but I think it is feasible that like a company could work really well with a group of
folks, say, working on prison abolition or something like that, and could craft like here
are the metrics for what the world that we would like to see. And this is like the kind of outcomes
that we want to see. And so that's, that would be my kind of generalized recommendation. But that
said, and this is like one of the major tensions of this space, is that often people are building
tools that are more generic, right? Like you're looking for generalizability, you're looking for
abstraction and like that could be applied in a wide variety of different application domains.
And that I think is a tension. I actually think that is like a major tension for AI doing good
in the world and being culturally grounded and informed. I don't know how to resolve that. And
I would love to hear what like the computer scientists and researchers have to say about
if they see that tension as well or not. I fully agree. And I think to me that
foundations of that goes back to the education, right? So computer science and engineering has
had the military background, you know, it started as a way to get into the military and that's
there today. Like, you know, we are told to keep our emotions aside. In fact, that was one of a lot
of gaslighting on Twitter is about, oh, why are we emotionally charged about machine learning? Just
focus on the math, you know? And so that's something that's pervasive in the educational
system and then beyond in the companies, right? Like, you don't worry about this. You do, you know,
you derive math. You just, you know, program. So you're asked to become a robot without any emotions.
And I think that's just really dangerous, right? Because it's not every engineer who's the problem.
I mean, there's absolutely a decision maker who is probably, right, kind of, you know, may or may
not understand the implications, but it's kind of directing all this and leading to the pervasive
bias and injustice. I think we have to start with education. That's where I would pin my hopes.
For instance, at Caltech, we have a course on ethics for AI. I think every university is now
doing that. Ultimately more humanistic education. You know, we are kind of, you know, I went to
technical school. Now I'm in a technical school, right? The role of humanities there needs to be
even further emphasized. You know, it's much better than when I went to school,
but it's still a long way to go to integrate humanities and engineering together.
I completely agree with Anima. I'm also teaching bias in AI. I think it needs to be part of the
curriculum, not for computer scientists only, but also engineers, humanities, social scientists,
probably as well, because data scientists or essentially anyone that is interested in
computing is using these tools. So they need to understand that all of these tools exist within a
context. They don't just exist in an abstract, generalizable way, as Catherine mentioned.
It's correct that these tools or algorithms were developed so that they can be applied
to hypothetical data sets so that the theory or foundations are presented. But then when
we take social data and apply these tools for machine learning tests in the social domain,
then we are dealing with all kinds of human complications. So people need to always keep
in mind that these systems, yes, they might look objective and neutral, but they are not in the
social context because they exist with all the problems or side effects that human decision
making brings into the game. But moreover, AI also brings its own kinds of problems, such as
new types of biases we are not even aware of yet. And these keep getting amplified in society. So
before we start introducing more of those, we need to understand what we are bringing to the world
and what we should be careful about, because all of these systems are deployed right now without
understanding what side effects they might have or what the implications are on society,
economy, democracy, and so on. And now we would need to deal with all of these consequences
on top of all the other things. It seems like a major takeaway you guys all advocate for is to
kind of broaden the scope of who gets included in the conversation about AI design, about AI data
collection, you know, bringing in activists in the relevant spaces, bringing in other stakeholders.
And, you know, one thing that I would like to add to that is that the notion of stakeholder
is really politically fraught. In industry context, stakeholders are generally conceived of as
like, you know, so I suppose I'm at a ML company developing an algorithm that I'm going to sell to
another company to use for financial profits. You know, the stakeholders in this process are
generally conceived as being me and my team, my superiors at my company, the people I'm selling
the algorithm to, and the stockholders of their company. And then I think that misses a really
crucial point, which is the stakeholders are the people whose data is going to be analyzed.
And their voice is very, very frequently just completely left out of any discussion about how
things are designed, why things are designed, and where they come from. And I mean, to an extent,
partially by design, because a lot of companies exist out of the fact, because of the fact that
they can make money analyzing user data, like that's their sole reason, that's their reason
for existing. There are lots of these things. But broadening, I think broadening the notion of a
stakeholder, in particular, viewing anyone whose data is going to be processed by an AI as a
stakeholder is really important. Yeah, I just I totally agree with that. We have a chapter in
our book, which is about the principle embrace pluralism, which is basically this idea that,
you know, we create more complete knowledge, like the path to objectivity is not through
individual genius, but it's through bringing multiple different perspectives to the table.
But I agree with you, I think like the it's important to think about like, because we can
say, oh, it was participatory, like that company with all the money and the other company with all
the money. So I do think and this is because participation, I mean, there's this whole
world, you know, participatory design and participatory methods. And so like, there's
even folks have talked about participation washing, where, you know, you're like, oh,
yeah, we did a, you know, this also happens in urban planning, like, oh, yeah, we did a public
meeting and it was at like 10pm and there was no childcare. So like, what's the quality and
the character of the participation? Like who shows up? Who are these stakeholders? And who
are the stakeholders that are not convenient to have? And how do you make folks listen to them?
I mean, I think that's a big, that's a big question when ultimately, like the goal is to
optimize profit. That to me is a conundrum. I think how do you make people listen, I think
is a really interesting question. You know, a lot of people, I imagine a lot of people feel like
they would like to do something good, you know, they would like to be more just and do their
design the right way. But you know, how do you convince your boss or your boss's boss who
controls the money and the funding, or the money and the decision making power, and who's maybe
more interested in what's profitable? You know, how do you do any of you guys have advice about
how to approach that conversation? Or how to how to advocate for justice in AI design,
in context where that input is not necessarily the primary concern of the people who pull the strings?
I'm interested in that as a research question, if we can figure out a way to simulate AI systems to
show their long term gains and losses for all the parties that are involved,
I think we would be able to understand how this affects economy in very from various perspectives.
But again, this is a hypothetical research question. I'm sure there's already some great
research on this yet. But maybe long term goals might be able to convince people that are profit
driven as well. I mean, especially if there's concern, I really like the idea a lot, especially
if there's concern, I mean, as there's this growing sort of industry of sort of impact assessment,
or algorithmic bias assessment, and things like this, this is like, bringing folks in and doing
a more participatory design process is a preventative for what could happen later,
and so I think maybe that's the, like, if the case needs to be economic rather than moral,
I mean, you can make the moral case, but probably what that listens to is the economic case, right?
But if the case needs to be economic, I think that, you know, it would be lovely to have some
quantifiable research that you could cite to say that like, yeah, there's this like,
very strong economic case that this will reduce the number of lawsuits later,
that you'll be fully compliant with all federal regulations, but you know what I mean? Like this,
these kinds of things, I think that that might be the path for convincing the employers,
and like, ultimately, like, you've invested in prevention,
so that you don't have to clean up a big mess after the fact.
That's why academia and the independent researchers have such an important role,
right? Because they can talk openly about these issues, like, you know, what happened with the
studies, right? Even though internally, there was a lot of conversation at Amazon,
when I was there, you know, having that external light into it, and then the wider public part
spreading in it, I think is critical. Because, you know, like being in these companies, it's
just the whole system is so short term driven, right? The Wall Street expects quarterly results,
that's not delivered, the company won't even exist, right? So the whole system is not built
for the long term implications. And I think that's what creates these challenges. And so
somehow, maybe then having more participation from the public, right, can show that also,
you know, bad AI can lead to bad PR and loss of profits, even in the short term,
I think that activism and awareness is critical to change this.
Ram, you want to jump in?
Yeah, I was just framing about what Anima said. But the way I think about this is, at least again,
through lived experience, I can say that what has changed for me is that if you think of like
inclusion, as after the fact, that's kind of like really weird. One of the things that I'm very
lucky is that inclusion is part of my core priorities that I set. Like, it's just not for me,
it's for everybody in the company. Like, you've got to set that, what is your inclusion priority?
For me, it might be like, you know, I'm going to take care of Pride Month, like, you know,
talks or, you know, whatever things, right? So obviously, it doesn't solve all problems.
So first of all, I fundamentally believe that if your organization is questioning
why you're doing something ethical, that itself is a big red flag for me.
Right, like, and for the business case, I do agree, like, you know, there's this, like, you know,
how Catherine said, there could be some financial motivation. I saw like an HPR article about like,
how diversity leads to more surplus, because you're selling your product to more people.
That could be like an other way. The PR angle, like, I mean, everybody is afraid of like bad
marketing. I'd never actually thought about that. So thanks for bringing that up. But for me,
intrinsically, you know, I have never been told you can't publish this. I mean, but again,
it's a very different setting, like MSR. And like, you know, product setting is very different.
But yeah, I would be very uncomfortable if I have to make a case to my manager as to why I'm doing
something ethical in the first place. But like Anima pointed out, like, business, I've only
worked in one organization for like, my working life. I do not know how others operate. I know
some of the CEOs out there question why they should use pronouns. So or like, why they should
like, you know, why they do not like fundamentally care about democracy and just want to dismantle
it. So I don't know, I do not know how things work there from my lived experience. I don't know if
this might actually be helpful for others. All right, so it seems like the kind of two
major points you guys have raised is that you can view it as a risk mitigation factor.
Because it will stop things from it can stop things from going wrong in the future. And it's
in some sense insurance on that. And it's also a PR thing, which is partially a risk mitigation
factor. You really don't want the New York Times running an article saying so and so's company
develops algorithms systematically discriminates against women in hiring. Oh, I actually saw
a headline. Yeah. Oh, sorry. Yeah. I actually saw an email. I only saw it through your tweet
about gentrify or genderify. Yeah, another thing after all the right outcry got withdrawn,
right? The fact that it even got launched is just appalling the kind of lack of awareness
these companies. Yeah, it was shocking. You know, there was another company.
I don't know who alerted me to it. But essentially, like their solution for more representation
on stock photos was to kind of like artificially generate like photos using GANs. So you will not
actually pay money to hire a black model. Oh, let's just use StyleGAN and like to kind of like,
like, how do you get through? Like, who's funding this and goes back to an email like
initial comment about VCs getting excited. So that I feel is a bigger like thing. I would be
very less less worried about managers and more worried about like plutocratic like Silicon
Valley folks to then like actually managers. So to kind of contextualize this for people
in the audience who aren't aware, Genderify is a product that existed for all of about four hours.
What their business model was, was that you can send them user data,
you can send them photos, you can send them people's names, and they will
they say they will determine the gender of the people that you
in that you sell to or the people that are in your data set with an aim of doing better
predictive marketing and stuff based on the... I'm having trouble coming up with a word other
than assumed gender of the of the people in your data set. And this was this was funded by some VCs
and it went online and everybody including lots of people in the ML space who don't really care
very much, who don't talk very much publicly about about ML bias lost their shit because it
was a completely absurd idea. And just to give an example, right, the word professor was only 1.9%
females. So that's the assigned percentage for me in terms of gender. So I think that that was
the problematic thing. Like how could you just type any name, like any word, not just name,
any word in the vocabulary and assign a gender to that? I mean, and only binary genders, right?
It's just appalling the whole concept. Yeah, my friend was saying the only time that's allowed
it's in French, any other language, like, you know, because, you know, nouns have gender.
Any other time table doesn't have like a male or female. Yes. Anyway. Yeah, but I'll know that like
there are like this product is insane. But there are other products like there's all these gender
detection libraries out there, both for faces. So like facial detection, which the researcher
Oz Keys has written this amazing paper called Misrecognition Machines, which is about how
harmful face like gender detection of faces is for trans folks. But then there's also all sorts
of libraries that infer gender from name using various kinds of like name databases. And like,
maybe I'll be like, sort of a naysayer here, because like, in general, I feel like these
applications are very well, again, it depends on the application domain. So like, if you're doing
this for predictive marketing, or like Oz talks about how they're doing it for bathroom access,
which is like, disgusting, you know, like, are you the right gender to get into the bathroom or
whatever? Those are super harmful and awful. And like those kind of applications shouldn't exist.
At the same time, these gender detection libraries are helpful for doing like,
bibliometric studies, where you're looking across like a huge corpus of say, like,
I don't know, like several million authors and trying to see like, what's the
over representation of men in that data set or something like that, you know, and so like,
that that's why like, again, I think it's hard to just say categorically,
hmm, something is bad or good. Although I do have certain things that I do think that way about,
but like, that could enable really interesting research that monitors back on the patriarchy,
for example. And so like, again, it's like, who's using it? For what? Yeah, like, that's what
matters. And just to give an example, we have used the gender detector, but that was to learn
about the MeToo conversation. So we, you know, collect lots of MeToo data from Twitter,
men talking about it, how are women talking about it? How are Democrats, Republicans? So yeah,
so that kind of, so you know, and this is in collaboration with Professor in Sociology,
Mike Alvarez. So you know, like, the context certainly matters. We can't just ban all
applications of gender detection, but really contextualize and look at whether it's for
harm or for good. Yeah. And in medical applications, we especially need very accurate
biased representations, because we know that for certain disease, the diagnosis and prognosis
depends on ethnic factors, gender, and socioeconomic background as well. Of course,
there are biases representing injustices in these medical data sets. But at the same time,
there are biases that we need to learn from for more accurate medical treatment. And I guess
that's why, for example, in the medical domain, machine decision making is becoming much more
accurate than human decision making. But again, yet we need to deal with all the other disadvantages
that come with this. So again, I guess everyone keeps mentioning this, it really depends on the
context where we are using these things. And name-based gender detection is a kind of an
outrageous example. But then there is a supercharged version of this in country X,
where machine learning is used to predict the criminality of a person from their face,
or actually give people social scores so that they can be sent to workers camps for their entire
lives, because the machine said so. So there's a spectrum of these effects that we can discuss.
And then what foundations should we be building on, given these examples that already exist in
the world? Well, thanks for sharing that. I was just, I completely dismissed it,
Catherine. And thank you, Anima and Eileen for, you know, correcting this. Thank you.
I didn't mean to correct anything, by the way. Oh no, it was like, you know,
I learned something. So thanks for telling me about this. Yes. I appreciate it.
What did you learn, Ram? Well, I just completely dismissed like gender detection. I was like,
oh, this is like, has literally no use. If I want to ask somebody's gender, I will just ask them if
it's part of my business. I'll say it's none of my business. I'm not going to ask. But when,
I was like, okay, that kind of makes sense. And when Eileen spoke about the medical contacts,
that makes sense. So yeah, this was, like I said, I already learned something from this panel. And
within like two more minutes for the panel to get over, I learned something. I learned a lot of
things. Yeah, the example that Eileen brought up of criminality prediction is unfortunately
running rampant in the AI literature right now. I feel like there is a half dozen papers that
have been published and then promptly retracted after a massive public outcry,
because they're basically trying to like reinvent phrenology, but doing it with machine learning,
and therefore it's good. There's a whole lot that can be said about this that could and actually
almost was its own panel at DEF CON. But there's a really good paper on this called
Criminality from Facial Illusion, which I'll send out on Discord and on Twitch to people interested
in reading about this problem that does a really good job of picking apart a lot of the issues
with both morally and also technologically, because a lot of these papers are fundamentally
technologically unsound with these kinds of works. So we're approaching an hour and a half.
This is the last event of the day. I'm enjoying talking to you guys a lot. I don't know if you
have somewhere you need to be, but I understand that. But if you guys are interested in continuing
to talk, I am more than happy to do so. I have to go make dinner for my family.
And it's actually sunny for a very short period of time in Seattle.
Trying to sneak in like a run. Stella, before we all disperse, I first want to thank you
so much for organizing this. You know, you have put so much thought and intention into this. Like,
I remember like asking you like, Hey, I don't want to be the male in this. Like,
have you thought about inclusive panel? I don't want to be a male. You wrote like,
perhaps the most comprehensive male, you know, about this. So thank you so much for, you know,
moving with intention, Stella. It clearly shows when you organize this panel. Thank you.
Thank you. Second.
Thank you. This is amazing. I learned so much and it's great to connect with all of you.
And thanks, Stella.
I enjoyed discussing with all of you. I think this was productive. And now I have more research
ideas that I can go on. So quickly before you guys run, I want to take advantage of the fact
that I have several experts here to get your thoughts on something I've been working on
recently, which we've, we've almost touched on this several times. And it's that a lot of times
machine learning algorithms have a underlying ontology. So Catherine mentioned, you know,
that some gender algorithms exclude transgender people, you know, but even when you're talking
about like all sorts of other applications you know, I work a lot with U.S. government data
scientists. And so they're working with data sets that come from the U.S. census. And the U.S.
census is a very specific notion of what races, and there isn't an option to use anything else
because you're, you know, you're the U.S. government and you're using U.S. census data.
But kind of how you, how you define your categories, you know, what, how you choose who
gets labeled as white, who gets labeled as Asian, really makes a big impact on what your algorithm
does and how it behaves. And whether you include certain categories, you know, if you include
transgender people or not. I mean, if transgender, if non-binary transgender people, people who don't
identify as, as, as a man or as a woman is not a possible label, then you have a guaranteed 100%
incorrect assignment rate for them. So, you know, do you have any, any thoughts or advice about how
to kind of tackle, you know, how to, how to approach this problem? Obviously, originally,
we can go and talk to the impacted communities and try to figure out what the right buckets are.
But oftentimes we start from a position where our data has been pre-collected for us by someone else.
And so if you have data that has this kind of problem in the way the questions were asked
originally, you know, is there something you can do with it still? I think that's a great question,
right? And a difficult one because, you know, the systems are already built. Like you were saying,
the categories are there. This is the label data. So I think towards getting to solutions for this,
I think an important step is being able to trust the system in terms of confidence levels,
right? Like, can the system say truly how confident it is? And the standard neural networks
don't have this property. So it can make high confidence errors. You know, it can be like 95%
confident and be wrong, right? And that especially happens on these kind of hard examples, which
hasn't seen in training data because it also gets the calibration wrong. In one of our papers,
we call it angular visual hardness. We proposed a new angular measure for detecting these hard
examples that was more robust than the standard accuracy scores and the confidence scores
that the systems are built on. And we also have some mathematical intuitions for that.
But the outcome of this is we can now look at, you know, examples which are hard for the
current classifier, right? So in this gender detection example, it would be most likely the transgender
people who would fall as hard examples because it's not getting this accurate, right? No matter
what binary gender the person is assigned to and if they don't kind of, you know, display that
gender, right? So there is that issue that can be detected through these mathematical means.
And then I think at least there is now the problem that has come up, right? That not every
example is equally treated and there are hard examples for this classifier. We can go back and
then design a system to correct that. And on a related note, I will give a vague answer to this
before I actually ask an even more daunting or deeper question. Maybe an email, for example,
idea of using more unsupervised approaches might be some kind of a solution for future generations
of AI algorithms. Maybe you wouldn't need to categorize yourself as a person, but instead some
kind of more unsupervised continuous representation of social entities might be a way to approach this
problem. This is not going to happen anytime soon, so I don't really have an answer for today's
problems because we know that gender is reflected to other features, all other features as proxies
as well. Because when we think about gender, that's the largest social group in society. So
it's one of the biggest proxies in the data sets. But my more concerning question is semantics,
gender, and race is defined in categories. So it's a proxy, but when you think about it,
essentially race doesn't really exist or gender might not even exist, but we have it in our
language, in our linguistic system, and that's the way we are constructed to deal with the world. So
how are we going to change these problems that are social constructs in AI systems that we are
building on these social constructs? So it's a longer-term problem that might require longer-term
solutions, which might not be very helpful for your question. Yeah, that really gets us something
unfortunate, and a lot of people in the Discord are asking applied questions that I haven't raised
because I know that the answer is, we don't know how to do that. We would love to know how to do
that too. So one example question is, if you can't bring in experts or social scientists or
stakeholders, is there an algorithmic way to kind of pick up on the bias? And the answer is, well,
there's 50 and we don't know which ones work better in different contexts. I mean, but there
is an algorithmic way to educate yourself, and that's like search engines, right? That's what?
That's like, that could be search engines, just like, you know, go type, like, you know,
if you want to learn about bias, go read Aileen's papers. If you want to learn about, like, you know,
bias, go watch, like, Timnit Gupru's, like, workshops. So I think a lot of this hands-on
things, people, smart people have already put out content, and it's just like a matter for us to
spend, like, cycles to educate ourselves on. Yeah, and as a, like, very basic kind of check,
like, you can look at the output of your system, if it has to do with people, and just look across
intersectional groups. So, like, you do, like, assuming that you have gender data and race data,
maybe you have income data, or maybe you have age, or maybe you have, like, some other kinds of,
you know, dimensions that describe people, is just looking at disparate outcomes within each thing,
and then combining those in various different ways to see if there are disparate outcomes
amongst the categories. Because, like, the theory of intersectionality from black feminism
says that, like, these categories compound in, like, a non-linear way, right? So being,
experiencing sexism as a white woman is fundamentally very different from experiencing
sexism and racism as a black woman. So, I mean, I think that's a very basic kind of self-audit
that you can do when you're working with data that's about people, that, like, you don't need
all that much, like, special training, necessarily. But, I mean, I think other people are working on
this, right? I mean, I feel like you guys would know this better than me, but, like, folks have
been working on things, like, where you would substitute out, like, you do, like, a sort of
counterfactual, and you look at, well, what if this data point, like, they have this outcome,
but what if they had, their race had been white, would they have had the same outcome with my
system? So I think people are working on those kinds of methods. I think just a really basic
one is just do a kind of an intersectional audit. Well, Stella, I, for one, look forward
to reading your work when it comes out. Thank you so much for joining. Thank you guys so much
for joining us. We've gone horribly over, but I've had a wonderful time talking to all of you,
and I know that the people listening really enjoyed it. If anyone wants to continue this
conversation who's listening, if you head over to the DEF CON Discord, there's a voice channel
where I'm happy to continue to talk to anyone who wants to ask questions or get references to
papers that people brought up and such. Thank you again to all of our wonderful panelists.
This has been amazing. So thank you very much. Thank you. Thank you. Wonderful to talk with
all of you. All right. Bye. I was also going to say we've also, we're going to have a journal
club focused on ethics and privacy with AI starting on the 18th. It's organized by Suhaka
on Thing, and you can check out our website for a link to a description of the whole thing.