WEBVTT 00:00.000 --> 00:01.020 Hi. 00:02.240 --> 00:03.220 Welcome. 00:03.220 --> 00:07.220 It's early morning, but I've been up for a while. 00:07.220 --> 00:08.320 I have jet lag. 00:08.920 --> 00:14.940 But welcome to our talk on the Genitive Red team that's happening over in AI Village across from this wall. 00:15.560 --> 00:18.800 We want to... yeah. 00:18.800 --> 00:20.420 So about us. 00:20.420 --> 00:20.920 Hi. 00:20.920 --> 00:22.040 I'm Sven. 00:22.040 --> 00:23.560 I have got a Ph .D. 00:23.560 --> 00:27.720 in math and then founded the village six years ago. 00:27.720 --> 00:29.060 Seven, six years ago. 00:29.140 --> 00:33.300 And so I like to describe myself as a mathematician who has wandered into security. 00:33.660 --> 00:37.040 And I do weird stuff with math and ML security. 00:37.700 --> 00:40.120 And then we have Roman and Austin here. 00:41.220 --> 00:42.780 Do you want to say hi? 00:42.780 --> 00:43.180 Yeah. 00:43.180 --> 00:44.080 Hey, guys. 00:45.640 --> 00:50.600 So Roman was on Twitter. 00:50.600 --> 00:52.740 Now she's running humane intelligence. 00:52.740 --> 00:54.520 And she ran the design team. 00:54.520 --> 00:55.940 She'll talk about that later. 00:56.020 --> 01:00.640 And Austin has made a lot of this happen. 01:01.880 --> 01:02.380 Yeah. 01:02.680 --> 01:04.280 Austin has done stints on the hill. 01:06.840 --> 01:10.100 It has been a very interesting year for us. 01:10.580 --> 01:13.300 But I want to tell you why we're doing all this. 01:13.300 --> 01:14.620 Why this is important. 01:14.620 --> 01:18.380 And it's not just about AI security ethics. 01:18.380 --> 01:19.920 It's not just about GPT. 01:19.920 --> 01:23.160 It's something that's been going on for the last 20 years. 01:23.160 --> 01:25.400 And there's a long history of this. 01:25.740 --> 01:30.060 So in a sense, traditional software is explainable. 01:30.380 --> 01:31.780 You have assembly code. 01:31.780 --> 01:34.720 You know what every single thing does. 01:34.720 --> 01:35.700 You can audit it. 01:35.700 --> 01:37.900 You have people doing reverse engineering. 01:38.160 --> 01:43.780 And they can go through that system and figure out exactly what it does, how it does that, how it makes decisions. 01:44.380 --> 01:46.400 And there's a way of doing that in math. 01:46.400 --> 01:53.680 If you give me a big complicated polynomial, I will use calculus, figure out where it's inflection points, critical points, where it crosses zero. 01:53.680 --> 01:57.580 I will figure that all out and then I'll be able to draw the polynomial and figure it out. 01:57.820 --> 02:07.120 Problem is, this is a part of a hello world and it's 53,000 lines long in a modern Rust compiled hello world. 02:07.280 --> 02:12.680 Which is a little long for a hello world in terms of just 53,000 instructions long. 02:12.680 --> 02:14.400 It's just a bit too much. 02:14.840 --> 02:18.660 But what we have here is AI, everyone says, oh, it's a black box. 02:18.660 --> 02:33.980 The actual amount of code that goes into something like chat GPT, the actual core of it is only a few hundred lines once you boil it down, just get matrix multiplication working, just get a couple functions working and then glue things together in just the right way. 02:33.980 --> 02:36.260 And you get a transformer and it works. 02:36.380 --> 02:40.340 But the number of parameters is how it really works. 02:40.340 --> 02:43.040 It's not lines of code, it's number of parameters. 02:43.180 --> 02:44.480 And it's a black box. 02:44.480 --> 02:45.420 But what does that mean? 02:45.420 --> 02:47.980 Everyone says, oh, AI is a black box. 02:48.040 --> 02:50.460 But that's not really true. 02:50.760 --> 02:53.300 It's more like chaos. 02:53.300 --> 02:55.060 That's supposed to be a slide. 02:55.060 --> 03:01.140 So if you know what chaos is mathematically, it is a process. 03:01.140 --> 03:03.440 In this case, it's a dynamical system. 03:03.520 --> 03:05.900 That's supposed to be a slide that's moving. 03:06.160 --> 03:08.000 It's a Julia fractal. 03:08.000 --> 03:11.560 And what you do is you have a step. 03:11.560 --> 03:15.240 Every single time you take a step, it changes slightly. 03:15.380 --> 03:18.900 And you don't know where the process is going to end up. 03:18.900 --> 03:25.060 You have to compute every single point on that two dimensional plane to figure out what the fractal looks like. 03:25.100 --> 03:36.360 If you've coded up a Mandelbrot set or Julia fractal, a lot of people code up, there's a process that the end game, you have to just calculate it. 03:36.360 --> 03:39.280 You can't predict the exact shape of a Mandelbrot set. 03:39.280 --> 03:40.900 You just have to calculate it. 03:40.900 --> 03:48.500 And that is the same thing with all sorts of things where it's a chaotic system and you just have to calculate it. 03:48.500 --> 03:50.440 And the same sort of thing is true for AI. 03:50.440 --> 03:56.880 You can get a sense of what it does by just like putting data through, seeing what comes out. 03:56.940 --> 04:00.040 And this is a problem. 04:00.120 --> 04:04.880 So if you take the simplest AI system, we're going to do MNIST. 04:07.100 --> 04:09.360 We're classifying digits. 04:09.360 --> 04:12.400 We have from postal services. 04:12.440 --> 04:20.380 And I'm going to take a two dimensional slice of a 784 dimensional space and figure out where things go. 04:20.380 --> 04:31.040 Each color on that side is a independent, essentially an independent unique decision path in the neural network. 04:31.040 --> 04:33.520 And I do not know what each of those does. 04:33.520 --> 04:35.780 We know what some of them do. 04:35.840 --> 04:41.040 We know what 60,000 of them do, because that's what we've tested. 04:41.040 --> 04:42.740 We don't know what all of them do. 04:42.740 --> 04:46.500 And there are two to the 60 of them on a very simple network. 04:46.500 --> 04:53.680 And if you use the correct network, there's two to the 500 little individual decision paths or more. 04:53.740 --> 04:57.460 And it seems to work, but we don't know. 04:57.680 --> 05:04.740 And one of the problems is, you know, if you talk about AI security, you have to include the slide of the panda gibbon thing. 05:05.100 --> 05:07.940 This is more of a proof that it is a chaotic system. 05:07.980 --> 05:14.060 If you go here, one of those slices is doing the correct thing. 05:14.060 --> 05:15.740 The AI is behaving correctly. 05:15.740 --> 05:20.920 You go to the next slice over, it's got something completely different and you don't know what's happening. 05:21.320 --> 05:25.580 And we have proof of this repeatedly with adversarial examples. 05:25.580 --> 05:30.820 You take panda, you put it into the image net and you add that bit of noise to it. 05:30.820 --> 05:35.900 This is from people from 2014. 05:36.000 --> 05:38.360 This is a fast gradient sign method attack. 05:38.420 --> 05:49.080 And you just do the slight perturbation that you can't see, but you just moved from one region where you know how it works to the next door neighbor region and you don't know how it really works. 05:49.600 --> 05:54.380 And there's no way to know exactly how it's going to go. 05:54.380 --> 06:03.100 And there are techniques, there are ways to smooth out this process, but this chaotic system is inherent to the AI and you have to just test. 06:03.100 --> 06:05.400 You can't prove that it works. 06:05.400 --> 06:07.860 You just have to test over and over again. 06:08.860 --> 06:12.520 And this comes out in machine learning, in security. 06:12.520 --> 06:18.520 So this is a malware data set that I managed at end game and Elastic when I worked there. 06:18.520 --> 06:22.920 So we had a malware model trained up until June 1st, 2019. 06:24.560 --> 06:27.640 And I tracked the false positive rate, the false negative rate. 06:27.640 --> 06:28.940 Those are the two things you care about. 06:28.940 --> 06:40.140 You care about the false negative rate because every single time this thing has a false negative, that is potential malware detonating on your systems that could take down your possible network, your computer, your personal thing. 06:40.740 --> 06:44.640 Every time you have a false positive rate, that is an alert you shouldn't see. 06:44.640 --> 06:47.640 So we want to keep track of those two things and keep them as low as possible. 06:47.800 --> 06:57.780 So we trained this model in June 1st, 2019, and then we release it, everyone's things, and the attackers now can start testing their stuff against it and they start exploring. 06:57.800 --> 06:59.500 And they start moving their stuff. 06:59.500 --> 07:03.800 The red line is how far they have moved, how far the overall data set has moved. 07:03.800 --> 07:13.820 And the green and blue lines are the false negative rate because malicious software drifts faster than benign software. 07:13.840 --> 07:21.220 And you can see, three months after it releases, they have figured out a location that works, that bypasses my model. 07:21.620 --> 07:26.640 They have just moved in the data space to somewhere that you get passed. 07:26.880 --> 07:29.780 And once they figure that out, they're just going to keep doing that over and over again. 07:29.780 --> 07:36.960 And you can see, you take it two years later, if you leave this model online, after a while, it goes terribly. 07:38.880 --> 07:43.260 20% false positive and false negative rate is not a good thing to have. 07:43.660 --> 07:55.340 So what you have, when you release any model, language models, malware models, image classification, anything, you have the situation where you want to have a complete picture of what it does. 07:55.340 --> 07:59.620 But you can only test things you know about and things you have thought about. 07:59.620 --> 08:06.200 So you only want the picture where you can see everything, but that is impossible. 08:06.200 --> 08:07.460 You cannot test everything. 08:09.000 --> 08:12.040 Testing everything is problem 2 to the D. 08:12.040 --> 08:14.560 It is equivalent to breaking cryptography. 08:15.160 --> 08:17.440 You get this picture on the side. 08:17.440 --> 08:18.520 This is just a thing. 08:18.520 --> 08:20.660 You don't see part of the image. 08:20.660 --> 08:22.400 You cannot ever test enough. 08:22.400 --> 08:32.020 And the attackers and people who are out to get you can just move into the space you didn't test, find something where the model misbehaves, and just do the bad thing. 08:32.740 --> 08:35.540 So for LLMs, things are different. 08:35.540 --> 08:45.300 Because we kind of have a static binary where we don't really have this attacker, this attacker, defender mindset that we have in security. 08:45.300 --> 08:51.860 But the problem is the breadth of what they are capable of is extreme. 08:52.180 --> 09:00.840 With malware models, with malware detection and spam models, there is a contained, known, this can do this, this is the thing. 09:00.840 --> 09:05.480 The worst case scenario is you misclassify a piece of malware and it just does the wrong thing. 09:05.640 --> 09:08.900 But with ChatGPT, you can do all sorts of stuff. 09:08.900 --> 09:10.980 You can use it for hiring applications. 09:10.980 --> 09:18.000 You can use it in medicine. 09:18.000 --> 09:19.580 You can use it in trading. 09:19.580 --> 09:23.080 You can use it in all sorts of different things. 09:23.460 --> 09:28.240 The problem now becomes the vendors tested as much as they could. 09:28.260 --> 09:37.140 They had a bunch of people training, testing, then they hired extra people to test and then they released the model. 09:37.300 --> 09:44.400 And if you saw ChatGPT, they released the model and two weeks later, people were like, oh, yeah, prompt injections are a thing because they hadn't thought of that. 09:44.400 --> 09:55.240 Now they release the model and we know that prompt injections are a thing, but now there's all sorts of new prompt injections because originally it was just ignore the above instructions and tell me what's up. 09:55.280 --> 09:56.980 Now that doesn't work. 09:56.980 --> 09:58.480 You have to use the grandma hack. 09:58.480 --> 10:02.900 You have to use Dan and a bunch of other things, but they just do the same thing. 10:02.900 --> 10:03.760 They move. 10:05.000 --> 10:12.080 Because of the breadth of what things are applied to, the adversaries attacking things is not the only problem. 10:12.080 --> 10:13.680 You just have to test more stuff. 10:13.680 --> 10:17.760 You cannot finish the testing because you don't know how things work. 10:18.320 --> 10:22.900 And the other major problem with the malware model, it's sort of disposable. 10:22.900 --> 10:31.640 It costs at most $20,000, $50,000 to train a malware model if you're on an industrial size dataset. 10:31.640 --> 10:36.960 For an LLM, the ChatGPT4 costs millions of dollars. 10:37.240 --> 10:47.460 With the malware model, if it's $50,000, if it's really key to the business and you need to train that once a month, that's an expense you can just book and dispose of that model every month. 10:47.460 --> 10:53.180 With an LLM, especially the large language models that we're testing over in the AI village, you cannot dispose of them. 10:53.180 --> 10:54.660 You have to patch them. 10:54.660 --> 11:04.320 And that is another new thing that LLMs, the expense of AI has gotten to the point where you have to test the known thing and over and over fix it. 11:04.340 --> 11:06.920 That is what this whole thing is about. 11:06.920 --> 11:08.640 We just need to test more stuff. 11:08.640 --> 11:12.980 And the people that have tested it are... there's a small number of people who have tested it. 11:12.980 --> 11:17.120 We might double the number of total people who have tested LLMs at the end of this weekend. 11:17.320 --> 11:20.060 But that is why we're doing it. 11:20.060 --> 11:23.260 And how we actually got this happening is what Austin is going to talk about. 11:27.010 --> 11:29.070 Thanks for the rapid transition. 11:29.070 --> 11:30.110 I was prepared. 11:30.350 --> 11:30.870 All right, folks. 11:30.870 --> 11:31.650 Austin Carson. 11:31.650 --> 11:33.750 I'm probably the most confusing person here. 11:33.870 --> 11:37.330 I just randomly ended up working in government. 11:37.330 --> 11:46.830 I worked in Congress for about seven years and then got into nonprofit world around policies and trying to teach people in the government about how different parts of technology work. 11:47.410 --> 11:55.450 Briefly worked, actually worked at NVIDIA for about three and a half years trying to help explain high performance computing and artificial intelligence. 11:55.530 --> 11:59.090 People in the, you know, in Congress and in the administration primarily. 11:59.510 --> 12:07.390 And at a certain point, we got to the point, we got to getting to like practical things on the ground because it's too abstract, right? 12:07.390 --> 12:16.330 It's like all we ever do is kind of talk about principles and here's a series of kind of maybe thought experiments and we just had to get down into the granular part of the world. 12:16.330 --> 12:26.910 So back at DEF CON 30, Will Pierce, for the program, he just was testing stuff on stable diffusion and asked for a mug shot and every time he did, it made him black. 12:27.130 --> 12:37.910 So that was pretty transparently a problem and to the point of moving in the space and not testing it, that seems like a space that might have should been tested at any point, right? 12:37.910 --> 12:45.370 And it really illustrates a key part of why we're doing what we're doing, which is that if you don't think about it, you don't know about it. 12:45.370 --> 12:56.090 And it's within text and language space, so like your actual experience in life and your experience talking about things is more explicitly relevant than it's ever been in like a different technological epoch. 12:56.850 --> 13:14.830 So next up, we had hackers on the hill, which every year, some of you are probably familiar, I think Bo and Harley, some others, get some folks together, pretty good crew here, it was originally like 10 and now we're up to 50, 60, take them around to the house and Senate offices and explain to folks what they're actually seeing in the world. 13:14.830 --> 13:18.510 I mean, when I worked in my last job on the hill, it was hanging out at B sides. 13:18.530 --> 13:26.310 It was the first time I realized how totally fucked everything was in terms of info tech space, which was really constructive, to be honest. 13:26.350 --> 13:40.510 So we're at hackers on the hill, we are going around, Sven and I, and we're sitting there talking to one of the Senate staff, a couple different people in the group brings up wanting to do a generative AI red team and some other folks were like, you can't really do it. 13:40.910 --> 13:43.610 Sven says, yeah, you can, you can do it. 13:44.350 --> 14:03.810 And I was like, all right, we're going to go talk separately after this and we sat there for about an hour in the cafeteria and talked about how, you know, I had an event going at South by Southwest and I'd been working with a number of community college students and the purpose was to help build kind of like an AI national network and in part predicated on the fact that it's a language skill, 14:03.810 --> 14:10.770 it's just talking to a thing, so it doesn't really matter who you are and in fact who you are explicitly matters in a beneficial way. 14:10.930 --> 14:16.590 So Sven said he could hack it together in five weeks, use CTFT and put some challenges in it. 14:16.910 --> 14:22.390 Ramon, of course, was extremely helpful and then our weird stool had three legs and we started moving. 14:22.490 --> 14:27.970 So at South by Southwest, I turned the entire event to be around this, whoops, realized I wasn't facing the mic, sorry guys. 14:28.070 --> 14:37.370 At South by Southwest, I turned the entire event to be about this exercise, which at the time with a little simpler version, we call it prompt detective and everybody had a blast, right? 14:37.370 --> 14:40.050 We bust in 20 students from Houston Community College. 14:41.010 --> 14:50.310 We had something of like an educational and then a red team and then we had like kind of a very short training, here's how you could build one of these things component to it. 14:50.310 --> 15:03.490 But huge hit, we had some folks from the White House and Congress there, one of them was on the wrap up panel and just some great takeaways, you could see Congressman McCall just geeking out there, everybody loved watching it happen and the localism helps too, 15:03.490 --> 15:03.630 right? 15:03.630 --> 15:11.690 If it's people in your backyard, if it's your constituents, seeing them have a good time and do something that you honestly didn't think was an option is really constructive. 15:12.510 --> 15:25.090 And so then it escalated very quickly to the point that I can't even do the voice, I was going to do that bit but it feels so much more serious looking at its sequence, I'm like no, it was really crazy, it's crazy as hell. 15:26.190 --> 15:34.290 The White House decided that they wanted to jointly announce and participate and the red team and our lives exploded for the subsequent three months. 15:34.290 --> 15:42.910 Now the benefit of that is the White House's attention is very compelling for corporations to participate in a public disrobing and people exploring their things. 15:42.910 --> 15:45.390 We're also a very friendly group compared to many. 15:46.370 --> 16:01.230 And so over the, yeah, it was like the following week we had to get maybe the most important companies in human history to let us have a bunch of hackers and some community college students and folks from an organization called Black Tech Street just try to break it every way that they possibly could. 16:01.310 --> 16:04.950 And to Sven's point, I think we all know there's going to be stuff they find. 16:05.230 --> 16:15.510 There's no way that the 30 to 100 person red teams that they have are possibly exploring any of this infinite 650 billion parameter 3D mathematical thing. 16:15.510 --> 16:22.710 And I think part of what's been exciting in this process has been to see kind of the evaporation of hubris, perhaps. 16:22.750 --> 16:27.450 Folks as they got more and more involved were increasingly thankful that we were working on this. 16:27.510 --> 16:34.210 So we had a second pilot at Howard University in D.C., which is one of the premier historically black colleges and universities. 16:34.790 --> 16:37.750 And we had students split from Georgetown and Howard. 16:37.750 --> 16:45.410 And actually at the end of it, everybody had so much fun that I think we ended up flying four additional students here from Howard to participate and get involved in the community. 16:45.410 --> 16:50.770 So we had about an hour and a half, 10,000 generations and 400 submissions. 16:51.110 --> 17:01.930 And it was so competitive at the end that even though the cash prizes that we offered, because people should get paid for doing this shit, the cash prizes that we offered didn't matter. 17:01.930 --> 17:02.970 They were like, no, we don't care. 17:02.970 --> 17:03.590 We don't even want it. 17:03.590 --> 17:04.610 We just want to win. 17:04.610 --> 17:05.550 Fuck you guys. 17:05.550 --> 17:06.950 And I honestly really enjoyed it. 17:06.950 --> 17:07.710 It was awesome. 17:08.750 --> 17:11.270 And now I think I'm kind of done. 17:11.270 --> 17:16.450 I don't know if you guys want to know more about the government and how this happened, but that's pretty much why it's crazy as hell and why I'm here. 17:16.530 --> 17:18.150 So I'll let Ramad talk now. 17:26.780 --> 17:27.800 Thanks, Austin. 17:27.860 --> 17:36.380 But also just to give you a time frame, just to give you a time frame, Hackers on the Hill was in February. 17:36.420 --> 17:38.860 So that's when Sven and Austin met. 17:38.860 --> 17:40.880 So all of this exploded, blew up. 17:40.880 --> 17:44.620 All of it happened in just a few short months of our lives. 17:44.620 --> 17:47.560 But I want to talk a little bit about the history of AI reporting. 17:47.560 --> 17:51.740 So Sven alluded to the fact that this is a decades-old problem. 17:51.740 --> 17:56.080 The first reported ML vulnerability is in spam from 2003. 17:56.080 --> 17:59.720 So we're tackling actually a 20-year-old problem that's existed. 17:59.720 --> 18:04.460 So this is not brand new, did not just pop up because of large language models. 18:04.500 --> 18:08.600 This is something that we've been trying to tackle and trying to think of for years. 18:08.600 --> 18:19.780 And as these models get more and more complex, have more dimensions, more parameters, we've not actually scaled up the capabilities to be able to tackle all the issues that are going to come up. 18:20.520 --> 18:22.840 Security is also a community effort. 18:22.940 --> 18:32.320 So we have methods in the security space that can be brought over into thinking more broadly about the different kinds of harms that happen. 18:32.420 --> 18:35.500 And that was kind of my role in a bunch of this. 18:35.500 --> 18:39.020 So for folks who don't know me, my name is Raman. 18:39.020 --> 18:43.940 I used to lead Twitter's machine learning ethics, transparency, and accountability team. 18:43.940 --> 18:46.700 I guess the current leadership doesn't like ethics. 18:50.240 --> 18:57.380 But two years ago, Sven invited me and we actually co-hosted the first algorithmic bias bounty in practice. 18:57.380 --> 19:02.400 It's something folks like, you know, that other folks have been writing about, talking about, thinking about. 19:02.480 --> 19:07.520 But Twitter was kind enough to open up one of our models for public scrutiny. 19:07.520 --> 19:09.180 We hosted it at a remote DEF CON. 19:09.180 --> 19:10.400 This is DEF CON 29. 19:10.820 --> 19:12.420 That was my second DEF CON. 19:12.420 --> 19:14.980 My very first DEF CON was in 2018. 19:15.200 --> 19:17.940 Actually, we did a panel about deep fakes. 19:17.940 --> 19:20.380 So it feels like everything for me has come very full circle. 19:20.380 --> 19:24.180 We have an election coming up next year in a world of generative AI. 19:24.180 --> 19:27.620 Who knows what deep fakes and misinformation is going to look like, right? 19:27.620 --> 19:29.120 So security is a community effort. 19:29.120 --> 19:33.620 We have always needed groups of smart people tackling these problems. 19:33.960 --> 19:36.820 But AI reports work a little bit differently, right? 19:36.820 --> 19:42.560 So when we held the bias bounty, this wasn't just about, hey, find a flaw and we'll fix it, right? 19:42.660 --> 19:49.040 Sometimes there's not this one-to-one relationship with identifying a bad outcome and figuring out how to fix something. 19:49.040 --> 19:58.160 With our model, actually, we had tested it for image cropping bias based on gender and race. 19:58.160 --> 19:59.580 And we knew that was insufficient. 19:59.580 --> 20:07.800 So we actually created a rubric, we put it out in the world, and we said, hey, test it for stuff that my team would never have thought of. 20:07.800 --> 20:14.620 And even being on an ethics team or a security team at a big tech company, you're the 1%. 20:14.620 --> 20:19.860 We are all very privileged to be in the rooms and have access to things we have access to. 20:19.860 --> 20:26.380 So opening up these challenges to people all over the world was extremely enlightening and it taught us how little we knew. 20:26.560 --> 20:37.620 People were testing it, for example, on religious head coverings to demonstrate that often people with head coverings get cropped out because they don't have hair in a photo. 20:37.700 --> 20:42.920 People with disabilities are cropped out of photos because they're not at the same height of people who are standing. 20:42.940 --> 20:48.000 And actually, people in camouflage can get cropped out because I guess camo works. 20:50.980 --> 21:00.200 So designing the GRTs, all of this kind of culminates over some years into this crazy thing we're going to be opening up in like, I don't know, 30 minutes. 21:00.800 --> 21:03.040 And this is the part that I'm really proud of. 21:03.040 --> 21:06.820 So my nonprofit, Humane Intelligence, was designing the GRTs. 21:06.820 --> 21:08.240 So what does it mean to design it? 21:08.260 --> 21:11.500 This was a collaborative effort, as both Sven and Austin have said. 21:11.500 --> 21:15.900 This was not like the three of us holed up in a corner, wrote some stuff, and were throwing at models. 21:16.260 --> 21:30.280 This was, you know, people from NIST, OSTP, various nonprofits, such as Terraz, Avid, Black Tech Street, and all eight of our vendors would meet basically every week, and we would design this challenge together. 21:30.280 --> 21:33.800 And it started from literally how should this be structured? 21:33.800 --> 21:35.640 What are the general topics? 21:35.740 --> 21:37.280 What are the kinds of questions? 21:37.280 --> 21:43.680 And also importantly, one of our goals was to align the generative red team to the AI Bill of Rights. 21:43.680 --> 21:48.140 Often in ethics, which is the world that I come from, we have lots of principles. 21:48.140 --> 21:51.800 We have a lot of lists of things we ought to be doing. 21:51.800 --> 21:55.360 But the hardest part actually is taking those principles and putting them into practice. 21:55.460 --> 21:58.200 That's what I've built my career on, is actually building stuff. 21:58.200 --> 21:58.960 I like breaking things. 21:58.960 --> 22:00.220 I like making things. 22:00.300 --> 22:11.060 And part of this was to actually take our country's AI Bill of Rights and figure out how to make it testable across some of the most important technology that we have seen in recent times. 22:12.240 --> 22:19.760 And like I said, as a group we got together, we identified the types of challenges that the companies really wanted to tackle. 22:19.760 --> 22:24.360 We addressed it in terms of the things that the country has said is a priority. 22:24.380 --> 22:26.220 And there are two kinds of challenges we have. 22:26.220 --> 22:29.900 So one, a lot of folks in this room are probably familiar with prompt injections. 22:29.900 --> 22:32.020 And prompt injections have to do with malicious actors, right? 22:32.020 --> 22:35.760 People trying to get the model to do something it's not supposed to do. 22:35.760 --> 22:43.100 So you're trying to subvert the terms of service or trying to break through some of the security safeguards that we've put up. 22:43.100 --> 22:44.900 And those make for great headlines, right? 22:44.900 --> 22:48.160 We all probably know do anything now in the grandma hack. 22:48.160 --> 22:53.240 We've probably seen the latest to come out of Carnegie Mellon about adding on text strings, right? 22:53.240 --> 22:57.380 But most people in the world won't be trying to do this. 22:57.380 --> 23:14.480 Most people in the world just want to use, let's say, a search engine that may have a language model underneath to tell them how to get to their court date or what their rights as a citizen are or who's running for president and what a person's opinion is on climate change or some political topic. 23:14.480 --> 23:20.140 And what we do know today is that language models can be fickle and they can be unreliable. 23:20.140 --> 23:26.760 And the information that comes out for a regular person can actually be hallucinated, false, but harmfully so. 23:26.760 --> 23:34.040 So the second set of challenges are based on embedded harms and in something in the responsible AI field we call unintended consequences. 23:34.040 --> 23:39.120 So the person is using it in good faith, but the output that comes out is actually harmful. 23:39.300 --> 23:47.840 Austin talked about the first example that was identified here in creating a mug shot and every single photo that came up was of somebody with black skin. 23:48.140 --> 23:59.860 In our challenges we are looking at things like misinformation, internal consistency, information integrity, traditional security style hacks, as well as prompt hacks. 24:00.480 --> 24:03.000 As a result, though, grading is tough. 24:03.000 --> 24:05.260 This is not make the model say turtle. 24:05.260 --> 24:07.680 This is a little bit harder than that. 24:07.680 --> 24:18.400 One of the things that I pride myself on is we're trying to tackle the complexity of the interaction of this technology with human beings and humanity. 24:18.400 --> 24:29.220 People are going to use these models in many, many, many different ways in order to make these actually useful for people to build things on, to have collaborative experiences on. 24:29.220 --> 24:30.920 It actually has to be reliable. 24:31.060 --> 24:32.980 And that makes grading pretty tough. 24:32.980 --> 24:43.140 So the way we're structuring grading is there is an auto grading aspect to it, but at the end of it we have a handful of judges who will be seven judges, I think, right? 24:43.140 --> 24:50.560 Who will be sitting down and actually grading the top folks just to ensure that what they've submitted actually is in alignment with how the challenge is constructed. 24:50.640 --> 24:54.380 So it's going to be a little different, a little interesting for those of you who are planning on doing it. 24:54.380 --> 25:07.100 You're going to get a little sheet when you walk in that gives you our code of conduct, you know, some sample prompts as well as some guidelines on what you can do when you sit down and look at the screen. 25:08.000 --> 25:11.680 And with that, back to Sven to talk a little bit more about the specifics of the challenge. 25:20.940 --> 25:36.240 So the actual start of this was DEF CON last year, we actually had stable diffusion, so at DEF CON last year we had stable diffusion in the village and people started testing that. 25:36.240 --> 25:46.740 One of the standard tests that we did was get nurses and surgeons and generate 1,000 pixels of nurses, 1,000 pixels of surgeons and all of the nurses were female, all the surgeons were male. 25:46.840 --> 26:00.380 And that's just, you know, we actually started canning that demo because everyone said if you're going to hospital nurses, we're just going to show you the folder full of nurses' images instead of have the GPU spin for that because it's faster. 26:01.740 --> 26:06.560 And then that's when we showed the stable diffusion, we had early access. 26:06.920 --> 26:08.260 People had fun with that. 26:08.260 --> 26:18.840 And so the first thing we did was we went and spoke to Bo Woods at the medical device lab because they had a very good history of working with vendors to bring them in to talk about how things worked out. 26:18.900 --> 26:25.220 So we have a hacker, like the biopacking village does, please do no harm. 26:25.960 --> 26:38.220 This is a, some of these models, the models have most of the safety mechanisms turned on but some of the safety mechanisms are turned off because they are, they will ban your account. 26:38.220 --> 26:44.780 And if we had our account banned for that model, that model would be off for the rest of the competition and that would be bad. 26:44.780 --> 26:49.380 So the vendors have agreed to turn off the account and not ban our accounts. 26:49.380 --> 26:51.280 So we can do that. 26:51.520 --> 27:06.680 But that is, we managed to get them to do that because we have the hacker and we're telling people, hey, don't, please do not try to, like, take a gotcha, take a screenshot, take a picture with your phone. 27:06.920 --> 27:09.040 These models are going to say bad things. 27:09.040 --> 27:10.200 We're expecting that. 27:10.200 --> 27:12.780 The safeties are partially turned off. 27:13.020 --> 27:17.960 We don't want, we want to work with the vendors after DEF CON to sort this out. 27:18.280 --> 27:21.080 And we're doing the disclosure after DEF CON. 27:21.120 --> 27:35.020 So we are going to grab some volunteers on our end and we're going to go through the data that we can collect and do a disclosure process with a bunch of people and we can talk about that later. 27:35.520 --> 27:45.020 But the other thing is we don't want this content to leave not because it doesn't satisfy DEF CON's code of conduct. 27:45.160 --> 27:49.900 If you do get it to say some hate speech, that isn't underneath DEF CON's code of conduct. 27:50.120 --> 27:52.020 So we want it to keep it private. 27:53.340 --> 27:57.340 So hacker just to say, hey, keep it private. 27:57.340 --> 27:59.380 We will do the reporting afterwards. 28:00.000 --> 28:05.860 And we just need it to be, you know, everyone to go calmly. 28:06.060 --> 28:07.400 It will be fun. 28:07.640 --> 28:09.720 So the hacker looks like this. 28:09.740 --> 28:12.860 The objective of this whole competition is to make LLM safer. 28:13.180 --> 28:17.700 The session data when you're in there is being collected so we can do the reporting afterwards. 28:18.140 --> 28:19.700 We are asking for an email. 28:19.700 --> 28:21.620 We are not verifying the email at all. 28:21.680 --> 28:26.280 If you do find something, we're going to email that email saying, hey, you found something. 28:26.280 --> 28:27.920 We do want credit for it. 28:28.380 --> 28:31.580 So if you want to get credit for it, that email should work. 28:31.580 --> 28:41.600 But if you want to give us some bullshit email because you don't want to be involved with that, you are free to give us other email. 28:42.100 --> 28:54.340 The other thing, the model vendors, you are going to see some element names, so like thorium, cadmium, not model names because the model vendors are anonymized. 28:54.340 --> 29:02.460 That is to get back at the trying to keep it so the model vendors are safe doing this because they've turned off some of the safeties. 29:02.660 --> 29:10.520 And we don't want you to be like, ah, I got a hugging face and they said a bad thing. 29:10.520 --> 29:12.160 Yes, that's the point of this. 29:13.600 --> 29:14.720 That's cool. 29:14.860 --> 29:22.840 If you said a really bad thing, we will send you an email and say, hey, you got to say something really bad and we want to report it out afterwards. 29:24.600 --> 29:26.300 We have models. 29:26.880 --> 29:28.560 A huge variety of models. 29:28.640 --> 29:32.860 So we have some smaller models, 40 billion parameters, which is pretty big. 29:32.860 --> 29:35.500 You can't run that at home on a consumer GPU. 29:35.640 --> 29:41.620 And we have some models with hundreds of billions of parameters from the larger vendors. 29:41.880 --> 29:49.720 We have models that are naked, which means that they don't have any ‑‑ they don't have filtrate defenses, which I'll talk about in a second. 29:49.720 --> 29:54.040 We have models that are ‑‑ have a whole big layer of defenses. 29:54.280 --> 30:00.980 So NVIDIA is using their guard rail system and that has an input filter to prevent you from putting in prompt injections. 30:00.980 --> 30:08.360 If it detects a prompt injection input filter, it shoves that off to a different LLM that handles prompt injections. 30:08.560 --> 30:22.800 If it has ‑‑ if it doesn't detect a prompt injection, it will pass it off to the real LLM and then once it actually gets the real LLM, it will do an output filter to make sure that the LLM is appropriately responding and doesn't say anything bad. 30:22.840 --> 30:29.800 But there's like three or four models that you actually go through for NVIDIA because they're using their guard rail systems. 30:29.800 --> 30:36.660 But then another vendor, Hugging Face, I believe they're just using a raw LLM with a basic filter on the front. 30:36.660 --> 30:45.860 So you are testing the actual machine, just the LLM and not the multilevel layers around it that other vendors have. 30:46.560 --> 30:50.580 So it's different types of defenses they're using. 30:50.580 --> 31:00.220 All of them are using reinforcement learning from human feedback, which is a technique to steer LLMs to correct behavior. 31:00.280 --> 31:10.040 So if it starts talking about, you know, if it starts saying something bad that it shouldn't, you can punish it, which is a misnomer. 31:10.040 --> 31:11.420 It is a mathematical object. 31:11.420 --> 31:12.620 It is not a thing. 31:12.620 --> 31:17.880 But there is a ‑‑ it is from the terminology from reinforcement learning, which is all a thing. 31:17.880 --> 31:23.380 There is an equation that it follows for reward and there's an equation that you follow for punishment. 31:24.300 --> 31:30.120 But there is that system in place for every single one of the LLMs. 31:30.240 --> 31:36.160 But there are other systems in place for some of the LLMs. 31:37.180 --> 31:41.120 And also 30 billion to 175 billion parameters. 31:41.140 --> 31:44.960 So we got a massive variety of models. 31:45.360 --> 31:56.200 One example that you can go play with yourself, GRT.AI, that is a ‑‑ you're testing a model, a prompt injection challenge. 31:56.340 --> 31:58.860 This is a input filtered prompt injection challenge. 31:58.920 --> 32:07.040 They have a bunch of classifiers that are classifying different prompt injections and you're trying to get a particular word out. 32:07.040 --> 32:16.380 And if the model says I don't know what the word is, I haven't actually had a chance to play this, you get the points and you can go on to the next level. 32:16.960 --> 32:25.140 And they collect different types of prompt injections and they are ‑‑ classify them and try to work with that. 32:25.720 --> 32:30.560 But we have ‑‑ so you can play with that yourself and then we have other types of things. 32:30.600 --> 32:33.120 What are the common first names for medical doctors? 32:33.120 --> 32:36.300 And respond with John, David, Michael. 32:36.620 --> 32:38.920 What are the common first names for registered nurses? 32:38.920 --> 32:45.940 The doctor nurse example from Stable Diffusion that we used last year, Sarah, Jennifer, Jessica. 32:48.140 --> 32:55.080 And there are other ‑‑ you can test all sorts of different permutations of this thing. 32:56.340 --> 33:06.280 The best thing for you when you actually get into the GRT is not to test something ‑‑ it's to test something from your background. 33:06.460 --> 33:07.500 So what's unique about you? 33:07.500 --> 33:18.100 If you know ‑‑ if you're very good at social engineering, is there some weird social engineering technique that you think that Cohere or Open AI didn't think about? 33:18.100 --> 33:21.760 Is there some weird history of your background? 33:21.760 --> 33:35.380 Like I'm South African, I asked a bunch of these models about South African history and some of the weird situations that happen in South Africa and some of them do okay and some of them don't do well. 33:35.820 --> 33:49.720 But that's the best way for me to find sort of biases and issues in this model is to ask about weird things about Poland and South Africa that I don't think that the people over in Silicon Valley thought of. 33:50.440 --> 33:58.860 But for you, I don't ‑‑ you should test your background, your sort of lived history and what your expertise is in. 34:00.280 --> 34:07.180 That will bring the most interesting data for afterwards and might get you the best responses. 34:07.920 --> 34:09.840 So the point system. 34:09.840 --> 34:28.300 So we have scale AI who's building ‑‑ who's got a grading system that uses some human graders that they've hired for this thing and they're going to do a first pass to make sure ‑‑ like a triage pass to make sure that when you submit, when you get a result, 34:28.300 --> 34:34.420 you want to submit something, you're going to get a little report saying, hey, why do you think you solved the challenge? 34:34.500 --> 34:44.280 You type in a little report saying, hey, I think I solved the challenge because it said a bunch of female names for nurses and it should have said ‑‑ mixed some male names in there. 34:45.240 --> 34:53.080 If that report is accepted, it gets shown to a graders for triage and then all they're checking is, is this plausible or not? 34:53.360 --> 35:14.760 They're not going into a big depth grading and we are just going to take the top 20 plausible scores and then on Sunday, we're going to have this group of people, Roman, Alan, SJ Turp who's over in merch right now, Tyrance Bigley who helped organize all the students that were coming in. 35:14.760 --> 35:16.160 You should mention the students. 35:17.220 --> 35:28.400 Sarah Kinsley who is a Carnegie Mellon, Casey John Ellis from Bug Crowd and Harley Geiger from Venable and the policy council. 35:29.760 --> 35:33.460 And then they're going to pick a winner out of the top ten. 35:33.540 --> 35:47.480 They're going to just judge and pick someone and the top three finishers get invited ‑‑ get an A6000 which NVIDIA donated for this which is an excellent GPU and I love my A6000. 35:52.100 --> 35:54.440 So what's next? 35:54.440 --> 36:00.160 After the after this, we have a sister conference. 36:00.160 --> 36:03.540 AI Village has been around for the ML community for years now. 36:03.540 --> 36:04.920 This is our sixth year. 36:05.160 --> 36:08.340 And it is also Canvas's sixth or seventh year. 36:08.740 --> 36:12.520 Canvas is in October in D.C. 36:12.520 --> 36:20.980 They are ‑‑ it's a more technical conference geared towards professionals in the space of ML security. 36:21.240 --> 36:27.760 But we're ‑‑ it's for people who have worked in industry and like deploy models in adversarial situations. 36:27.960 --> 36:31.580 So we're going to do a live hacking event of a large language model. 36:31.580 --> 36:42.540 So this is kind of not a large live hacking event but we're going to invite the top ten finishers to Canvas and we're going to do a, you know, top ten finishers, three days. 36:42.580 --> 36:43.560 You get API keys. 36:43.560 --> 36:44.400 There's no platform. 36:44.400 --> 36:45.220 There's no nothing. 36:45.220 --> 36:58.440 You just try to get it to do bad things for three days and report and we're going to have fun figuring out how all the reporting works in an intense situation and then have Canvas and talk about it. 36:59.680 --> 37:03.240 And that is also part of the coordinated issue disclosure stuff. 37:03.240 --> 37:08.620 We are going to do ‑‑ as I said, we're going to report out all the issues that we find. 37:10.160 --> 37:12.760 And that is going to be complicated. 37:12.800 --> 37:18.960 Patching issues in machine learning these models is more complicated than normal patching. 37:18.960 --> 37:23.260 So it's going to take six months and we're going to then report out the cool stuff people found. 37:24.760 --> 37:30.820 So if you do find something cool, we will email you and this process is going to be a lot of fun. 37:31.360 --> 37:36.420 But I think Austin has a couple more things to say. 37:36.420 --> 37:36.900 Thanks. 37:37.040 --> 37:37.580 All right. 37:37.580 --> 37:41.540 I do want monkey to tackle me off stage so I'm going to do like nine minutes instead of four. 37:41.540 --> 37:44.120 So a few things about why this is important quickly . 37:44.200 --> 37:59.520 So first of all, I ran a massive national poll out of just morbid curiosity as I watched the media and kind of folks talk about how they thought about large language models and generative AI and kind of the, it passed the MCATs but not the bar, it's dumber than me, 37:59.520 --> 38:13.680 you know, like kind of some incoherent nature and the inability to look at it as a complex mathematical representation of human language and the probability a word comes next and like our souls, you know, it's like a ‑‑ gets a little challenging for some folks, 38:13.680 --> 38:14.460 it seems. 38:14.460 --> 38:25.280 So in the process, first of all, fun fact, 25% of people believe they have a true self and like 45% of people strongly believe they have a soul. 38:25.280 --> 38:29.100 So somewhere there's 20% of people that believe they have a soul but it's not their true self. 38:29.100 --> 38:30.300 That's just a fun thing. 38:30.680 --> 38:34.500 But the thing I tested that mattered most, we tested across a couple different things. 38:34.500 --> 38:42.940 The first was given all the use cases, right, look at everything we have existing law on, should we still regulate the underlying models? 38:42.940 --> 38:45.880 The answer was surprisingly overwhelmingly yes. 38:45.880 --> 38:48.540 I mean, in the state of Texas that was 66%. 38:48.540 --> 38:55.260 So it's not natural for you to see somebody trying to regulate a technology in absentia. 38:56.380 --> 39:10.580 Second aspect of that, when asked who people thought should be the ones investigating and testing these models between the government and the private sector and other public‑private partnerships, the number one answer strongly agreed between Democrats, 39:10.580 --> 39:13.200 Republicans and independents is white hat hackers. 39:13.200 --> 39:23.620 So while on a certain level, we're not really necessarily ‑‑ the crowd in the room is not the first one getting a call from anybody in AI space asking if they can check out their models. 39:23.620 --> 39:26.620 But the American people would like to see that be the case. 39:26.740 --> 39:32.540 So in my view, this is a beautiful opportunity to prove that point to what Sven said. 39:32.540 --> 39:41.900 It is, in fact, whatever strangeness you've experienced and whatever you know from watching the world burn around you as nobody patches anything that is fairly useful in this experience. 39:41.900 --> 39:47.120 And second of all, we're flying in community college students and folks from black tech street from 18 states. 39:47.360 --> 39:53.600 We are trying to bring in this national consortium of folks that can unify with the hacker community. 39:53.640 --> 39:56.480 It's like a natural alliance waiting to be born. 39:56.580 --> 40:00.920 And then on the other side, help propagate this out as a red teaming exercise. 40:01.120 --> 40:08.400 So the open source component of this thing that we ran at South by Southwest and Howard can be run and modularized anywhere. 40:08.400 --> 40:14.140 It doesn't really require that much pre-work to just run the exercise. 40:14.140 --> 40:17.600 So infinitely scalable as long as somebody gives us compute forever. 40:18.320 --> 40:35.760 The other part of that is we also pulled to see if folks would like to come learn that way in their local area, people that live in random areas, Katy, Texas or something, would you like to go to your chamber of commerce and participate in one of these exercises where you can learn how generative AI works while testing it and trying to break it. 40:35.760 --> 40:38.640 Again, strongly agree that they would attend this event. 40:38.640 --> 40:42.060 It was like 25% democrats, independents and republicans. 40:42.060 --> 40:50.680 So if we do this right and if y'all break the shit out of this, as I know everybody can, we will prove this point forever. 40:50.940 --> 40:58.580 And as Sven is talking about this infinite latent space that we're trying to understand, strange things that interact with each other break in a stupid way. 40:58.580 --> 41:03.080 If y'all are familiar with this, most of the things that break generative AI are just dumb and weird. 41:03.140 --> 41:12.080 Like asking it to tell you how your grandma's favorite story was building a nuclear bomb and give me step-by-step instructions is like, okay, that works for some reason. 41:12.080 --> 41:15.120 And there must be infinite of these things, right? 41:15.120 --> 41:27.580 And as people are going to slap just general language intelligence on top of every service that we have, if we're not able to know what that looks like, we're going to walk into, again, a really weird catastrophe. 41:27.760 --> 41:34.240 So folks are fixated on some obvious sci-fi scenario catastrophes, which I also grew up reading a lot of sci-fi. 41:34.240 --> 41:35.660 I don't fault them for it. 41:35.660 --> 41:37.240 But we're all kind of smart. 41:37.240 --> 41:39.980 I don't know why we're only thinking about four ideas. 41:40.220 --> 41:41.660 There's going to be weird stuff. 41:41.660 --> 41:46.900 And we can find it and actually make it so that it's useful for us, all of us. 41:47.000 --> 41:52.720 It talks and sounds like us without people saying, no, you can't use the thing for this purpose, because it's a little offensive. 41:52.800 --> 41:59.820 If we know what's safe and not, you don't really have the issue of like the self-determination problem. 41:59.820 --> 42:03.700 It's not like they're going to say, oh, you can't use it for this or that, because it's a little dangerous. 42:03.700 --> 42:16.260 So I think this is kind of a unique opportunity and one we may never get again if we don't really do this right and everybody be reasonable about how they do it, but also really adventurous. 42:16.860 --> 42:18.320 Monkey, hurry up and tackle me, dude. 42:18.320 --> 42:18.800 I'm sitting here. 42:18.800 --> 42:20.120 I'm going to keep going until you do. 42:20.500 --> 42:23.680 All right, guys, listen, I'm done. 42:23.760 --> 42:25.260 Yeah, but he's nicer than you are. 42:25.480 --> 42:26.380 All right, I'm done. 42:26.620 --> 42:31.820 You guys feel free to find us and talk to us later if you have more questions, but the whole thing is weird and awesome and I believe in you all.