[00:00.540 --> 00:10.100]  From your perspective, what does CVD mean to you? And kind of what are your motivations
[00:10.100 --> 00:13.200]  when you're working on an issue and you need to disclose something?
[00:14.460 --> 00:20.500]  So I think the main point of CVD is, of course, to protect users.
[00:23.580 --> 00:31.540]  And my main thought is a bit historical. So I've been around in the 90s where disclosure was
[00:31.540 --> 00:37.760]  you find a mailing list where other hackers hang out and you zero-day everybody. And nobody
[00:37.760 --> 00:44.200]  really cared. And that became responsible disclosure, which is the predecessor for
[00:44.200 --> 00:51.840]  coordinated vulnerability disclosure. The problem with responsible disclosure is that it was really
[00:51.840 --> 00:59.080]  made to shame hackers that didn't follow the desired protocol of industry text.
[00:59.080 --> 01:06.560]  And the current iteration, coordinated vulnerability disclosure, is instead of
[01:06.560 --> 01:13.800]  trying to be coercive, trying to actually work with people that want to submit issues.
[01:14.600 --> 01:22.480]  And it is a much, much improved over that. And it does a much better job of making sure that
[01:23.400 --> 01:30.040]  every party to a vulnerability disclosure, that means vendors, customers, and hackers,
[01:30.560 --> 01:41.160]  have their needs met. So that's what I think coordinated vulnerability disclosure is for me.
[01:41.680 --> 01:47.420]  Awesome. Daniel, another researching perspective. What do you think about coordinated
[01:47.420 --> 01:54.340]  vulnerability disclosure? So we are interested in understanding how things work. And sometimes
[01:54.340 --> 02:00.980]  that involves understanding how things work that are not supposed to happen, which we then call
[02:00.980 --> 02:07.760]  vulnerability. And of course, then we want to figure out the truth behind it. And that involves
[02:07.760 --> 02:18.100]  talking to the vendor. Very often, they tell us that we got something slightly wrong. And that
[02:18.100 --> 02:24.580]  is an opportunity to correct these mistakes before we submit the paper or publish the paper.
[02:24.580 --> 02:32.760]  But also, from my perspective, it's the only ethical thing to do to protect the customers,
[02:32.760 --> 02:40.320]  to protect people who are using these products. And for me, I'm not in this game for
[02:40.320 --> 02:49.860]  finding vulnerabilities as a motivation by itself. My main motivation is finding the truth,
[02:49.860 --> 02:58.920]  finding a better understanding of how things work. That's awesome. Katie, you've had a lot
[02:58.920 --> 03:06.480]  of experience monkeying around with CVD. What does a CVD mean to you? Hold on a minute.
[03:06.680 --> 03:18.740]  All right. So CVD means to me... So when I think about it, I think of it from a risk-based
[03:18.740 --> 03:25.380]  perspective. And so what Probe is alluding to here is that my background is government in nature. So
[03:25.380 --> 03:31.580]  I spent about 15 years in the U.S. government, 12 years of that in the U.S. Air Force. And then
[03:32.260 --> 03:37.820]  several other years at the Department of Homeland Security, where I ran the vulnerability disclosure
[03:37.820 --> 03:42.600]  profiles programs that most people are familiar with. So like the NIST NVD program and the
[03:42.600 --> 03:51.100]  Carnegie Mellon CERT CC program. And the MITRE CVD program. So I was a sponsor for those programs. So
[03:51.100 --> 03:57.440]  in a single year, 2017, we coordinated and disclosed 14,800 vulnerabilities for public
[03:57.440 --> 04:05.220]  disclosure. So that's 14,000 IT vulnerabilities and about 800 ICS vulnerabilities. In the following
[04:05.220 --> 04:10.360]  two years, so 18 and 19, we coordinated and disclosed over 20,000 cybersecurity vulnerabilities.
[04:10.620 --> 04:16.900]  So I've kind of seen things from lots of different perspectives. And when I think about CVD,
[04:16.900 --> 04:24.760]  I think about balancing risk, right? So CVD is really a process. And every time you,
[04:25.180 --> 04:28.860]  when you're going through this process, understand like every organization is going to treat it
[04:28.860 --> 04:33.100]  differently, because there isn't a standardized, like one size fits all kind of thing here.
[04:34.060 --> 04:38.740]  There's differences that are going to happen across the coordination stack. So if you're
[04:38.740 --> 04:41.440]  looking at digital services, it's going to be different from software, which is going to be
[04:41.440 --> 04:45.100]  different from open source, which is going to be different from hardware or ICS, like all the
[04:45.100 --> 04:50.420]  differences are going to come into play there. But the overarching sort of thing, the big takeaway
[04:50.420 --> 04:59.660]  here is that CVD is balancing risk. It's all about making sure that there is an opportunity for the
[05:00.220 --> 05:05.860]  product vendor to fix the problem before an adversary has the opportunity to take advantage
[05:05.860 --> 05:10.720]  of that. So like, it's all about protecting the end user. And I think everybody that I've ever met
[05:10.720 --> 05:16.480]  in the entire ecosystem is all focused on that. Like we may be speaking different languages,
[05:16.480 --> 05:20.960]  we may talk past each other, but I think that everyone is trying to protect the end user.
[05:20.960 --> 05:26.880]  So to me, that's what that's what CVD is about. That's awesome. And remind you, remind me to
[05:26.880 --> 05:31.580]  circle back to you later in our talk, talk a little about some of the psychology behind some of this.
[05:31.580 --> 05:40.000]  I would love to hear my dearest friend Lisa's thoughts on kind of coordinating
[05:40.000 --> 05:44.820]  vulnerability disclosure, and kind of what some of your motivations that you've seen,
[05:44.820 --> 05:52.860]  or kind of what's behind your modus operandi? Yeah, so I think where I've seen a sort of an
[05:52.860 --> 06:01.220]  evolution of what we do now, where I think Heartbleed was sort of the big start of where
[06:01.220 --> 06:06.940]  we paid attention, realized we all need to come together a little bit. And then Spectre Meltdown
[06:06.940 --> 06:13.620]  sort of brought us even closer. So my thought is, is that not only do we work with the researchers,
[06:13.620 --> 06:21.020]  but we work across the industry to, you know, make sure that we're all doing our best to protect our
[06:21.020 --> 06:26.860]  customers. So like Katie was talking about, it's really our end users that are our focus,
[06:26.860 --> 06:31.980]  and making sure that we have the right people involved to be able to solve the issue,
[06:31.980 --> 06:36.060]  and then provide a security update so that our customers could get it and be best protected.
[06:45.180 --> 06:52.000]  So Omar, you've been doing this for a while, and you work for a very large company,
[06:52.000 --> 06:56.200]  so I'm sure you get the opportunity to see a lot of different vulnerabilities. So what does
[06:56.200 --> 07:01.060]  the whole process mean, and what are some of the motivations you've seen? Yeah, I think that
[07:01.060 --> 07:06.720]  pretty much everybody has summarized it very well, is protecting the end consumer at the end of the day.
[07:06.940 --> 07:12.800]  But I see it as a very, very complex ecosystem, right, that we're trying to actually solve.
[07:13.220 --> 07:19.860]  And Katie, you know, mentioned in one side hardware, another side software, then if you
[07:19.860 --> 07:24.540]  decompose that, then you have open source, which we're going to probably talk about a lot in here.
[07:24.820 --> 07:31.260]  And you have things that, you know, perhaps you cannot even control. I have had, I guess,
[07:31.260 --> 07:35.860]  the pleasure or not the pleasure, in some cases, where we're looking at vulnerabilities and even
[07:36.940 --> 07:41.560]  in some cases, whenever we try to actually solve the issues, the companies don't even exist anymore.
[07:42.380 --> 07:46.920]  That's another predicament, you know, that in some cases, we actually have to take into
[07:46.920 --> 07:53.300]  consideration. And it's a fairly complex ecosystem. And at the end of the day, what we have to
[07:53.300 --> 07:59.840]  actually put our heads together is how can we modernize our practice into the way that not only,
[07:59.840 --> 08:04.860]  yes, we deal with a single vulnerability, we reproduce it, we actually fix it, we find the
[08:04.860 --> 08:11.020]  patch, but how can we accelerate that process, that one, everybody's talking somewhat the
[08:11.580 --> 08:17.500]  correct vocabulary, or at least they can understand each other. And then we also understand the
[08:17.500 --> 08:24.140]  overall risk, right? And how to prioritize things and so on. So it's fairly complex, so we can
[08:24.140 --> 08:28.760]  actually talk about it for quite some time. And that's what we're here for. But those are some of
[08:28.760 --> 08:35.580]  the initial perspectives that I want to share. Thank you, I appreciate that. I want to talk
[08:35.580 --> 08:42.420]  about something we touched on in one of our prep calls. We have Anders and Daniel that come at
[08:42.420 --> 08:49.700]  this research angle from slightly different angles. So maybe you two gentlemen can figure
[08:49.700 --> 08:57.120]  out who wants to start first. But thinking about the perspective of academia versus a professional
[08:57.680 --> 09:04.760]  bug hunter, security researcher, maybe you can describe some of the particulars in
[09:05.360 --> 09:08.900]  either area. And maybe what are some of the differences you might see between those two
[09:08.900 --> 09:21.160]  different types of research? So from the industry, or rather the bug hunter, be professional or not,
[09:22.260 --> 09:29.940]  there is very often an element of good old-fashioned fun. I started out hacking
[09:30.660 --> 09:36.480]  things not because I wanted to achieve anything with it, just because it was great fun.
[09:37.800 --> 09:46.080]  When you become pro with it, there is different motivations. So if you work for a company and you
[09:46.080 --> 09:53.400]  work on their products, obviously your motivation is to make those products secure. And sometimes
[09:53.400 --> 10:01.340]  my past job, I did some research and that was sponsored by that company, but essentially me
[10:01.340 --> 10:10.900]  doing what I like to do. Their end of it was attention to that company's competences. And
[10:10.900 --> 10:18.300]  my end, of course, was having a bit of fun with it. So that is probably pretty typical of what
[10:18.300 --> 10:29.080]  hackers get out of hacking. Yeah, from the more academic side, it's more of advancing the field,
[10:29.080 --> 10:33.860]  the knowledge in the field, and there you try to figure out how things work and what the
[10:33.860 --> 10:40.220]  implications there are, what the security implications of certain understandings are.
[10:40.220 --> 10:47.720]  For instance, understanding when you can execute a certain piece of code and that does something
[10:47.720 --> 10:53.540]  that is not intended, what are the implications? And this is interesting from a security perspective
[10:53.540 --> 10:58.820]  than if it enables someone to do something that they shouldn't be allowed to do.
[10:59.700 --> 11:06.920]  Yes, often this involves disclosing a vulnerability to a vendor. But I would say that,
[11:07.520 --> 11:15.060]  for instance, bug bounties that might be very relevant for people from the industry.
[11:15.620 --> 11:26.920]  I think this plays a smaller role in academia, because you have to participate in CVD anyway,
[11:26.920 --> 11:33.740]  if you wouldn't, then you would get a lot of problems in the academic community. There's
[11:33.860 --> 11:39.040]  a lot of peer pressure that you participate in this, because that's the only ethical approach
[11:39.040 --> 11:47.600]  to handling vulnerabilities. At the same time, if you keep them secret for too long,
[11:47.600 --> 11:52.080]  then there's also the question on why did you keep them secret for so long?
[11:52.610 --> 12:00.040]  Maybe also some perspective that I can share. In academia, you see more and more often the
[12:00.040 --> 12:06.690]  effect that academic publications are easier to publish if you have a CVE.
[12:07.360 --> 12:13.260]  I don't think that's a good thing, because I think that a CVE does not necessarily
[12:14.500 --> 12:25.320]  describe that a research result really brings you new insights. A CVE is an identifier for
[12:25.900 --> 12:34.060]  a vulnerability, not an identifier that says this is something with a new insight.
[12:34.450 --> 12:40.140]  And also, if you participated in CVD, and you mentioned this in the paper,
[12:40.140 --> 12:48.460]  this also gives you bonus points, at least that's my impression, to get the paper published.
[12:50.180 --> 12:58.020]  Of course, it's good if you participate in CVD, but my feeling is also that this is going a bit
[12:58.020 --> 13:05.640]  too strong. And yes, now overemphasized in our community a bit.
[13:05.640 --> 13:09.160]  What about, you know, having an icon or a fun name?
[13:11.580 --> 13:14.680]  So, there are, of course, there are.
[13:16.740 --> 13:22.980]  I'm really a big fan of logos and names and anything that is fun.
[13:23.620 --> 13:29.160]  But of course, there are multiple layers here. For instance, we had this paper,
[13:29.160 --> 13:34.040]  Hello from the Other Side, which had this name not just by coincidence.
[13:34.740 --> 13:40.630]  And it was about a covert channel in the cloud where we send data through the cache from one
[13:41.040 --> 13:48.340]  virtual machine to the other. And we really went crazy on that. So, we built an entire TCP
[13:48.340 --> 13:53.640]  stack on top of that and tunneled an SSH session through the cache covert channel
[13:54.540 --> 14:01.600]  for whatever reason. And we then sent a music video through this cache covert channel.
[14:01.600 --> 14:06.340]  And of course, we couldn't just take any music video. So, we had to make our own parody of
[14:06.340 --> 14:11.340]  Hello from the Other Side. And this is just fun. I mean, we are just a bunch of people and we like
[14:11.340 --> 14:19.880]  to have fun. And basically, once you finish your project, it's really nice to close this up with
[14:19.880 --> 14:27.920]  something nice and funny. And logos on the other side also help about...
[14:28.700 --> 14:36.700]  So, they help you communicating about the issue. I realized that every time I create slides,
[14:36.700 --> 14:43.400]  my favorite slides are the slides that don't have any words on them, just pictures, logos,
[14:43.400 --> 14:50.400]  and icons. Because then I can follow the speaker. I don't have to read because I'm very bad at
[14:50.400 --> 14:55.220]  reading. I can't read and listen at the same time. And I bet many people can't do that. So,
[14:55.220 --> 15:02.500]  there's only icons and symbols and logos. I can follow what the person is saying and look at the
[15:02.500 --> 15:08.640]  images at the same time. And I'm always annoyed if I have to speak about the vulnerability and I
[15:08.640 --> 15:12.740]  don't have a logo for it because then I have to put text on the slide.
[15:13.740 --> 15:20.720]  So, since Daniel brought it up, let me ask the other panelists. How do you all feel about
[15:20.720 --> 15:26.700]  branded vulnerabilities? Do you like logos? Do you find that helpful to you and your constituents?
[15:26.980 --> 15:36.860]  Oh, I'm sorry, Daniel. No. When there's a logo or PR or everything, it grabs the media's attention
[15:36.860 --> 15:46.240]  so quickly. And sometimes the issue is not that severe or very hard to be able to
[15:46.240 --> 15:55.360]  breach. And it gives a little bit of extra fear to our customers and also encourages us to
[15:56.000 --> 16:04.680]  maybe reprioritize other issues first, which are more severe to our customers. So, although I
[16:04.680 --> 16:11.080]  think I get the point, it's easy to talk about it. They grab the media's attention really quickly.
[16:12.160 --> 16:19.880]  But if you put a caveat about the real severity of the issue in there,
[16:19.880 --> 16:27.420]  maybe that would help a little bit. Yeah. So, in our team, we often have this
[16:27.420 --> 16:33.520]  discussion. Should we have a logo? Should we have a website or shouldn't we? And we have...
[16:33.520 --> 16:39.440]  So, for most of our papers and most of the vulnerabilities that we discovered, we don't
[16:39.440 --> 16:45.300]  because we just say they are not significant enough. And I think that's also the responsibility
[16:45.300 --> 16:51.460]  of the researcher to first assess, is this something that significant that I need this
[16:51.460 --> 16:58.600]  additional PR or is it not? Right. Well, then you're doing it the right way. So, yay. Cheers.
[16:58.880 --> 17:05.920]  Maybe. But it's also... I mean, I will have a different view on what is really important than
[17:05.920 --> 17:10.260]  you have, right? Everyone will have a different view on that. So, something that I think is
[17:10.260 --> 17:16.320]  really, really important, you might say, well, it's not really exploitable in our use case.
[17:16.320 --> 17:19.920]  Right. We don't ever want to downplay a researcher's work. I mean, it's important,
[17:19.920 --> 17:26.360]  whatever they do. And we appreciate especially doing CBD with them and not being zero-day.
[17:27.860 --> 17:34.480]  But yeah, it's a more matter of, especially in a bigger industry when you're trying to
[17:34.480 --> 17:37.900]  protect your customers best, how do you approach it?
[17:38.220 --> 17:46.060]  There is a fight for customer attention. And for customers to make the right security decisions,
[17:46.060 --> 17:53.980]  they need to be attentive about the right things. And logos and hype in the press
[17:53.980 --> 18:01.920]  wants that. And we have bad security outcomes for our customers due to that.
[18:01.920 --> 18:08.040]  Of course, there's also the thing that even fixed vulnerabilities sometimes cause suffering
[18:08.040 --> 18:12.620]  for system administrators running around the basement and patching systems and all the like.
[18:14.180 --> 18:23.140]  And to some extent, I sometimes find overhyped bugs disrespectful to those people.
[18:25.980 --> 18:30.800]  Yeah. So, in my case, I don't have an issue with logos or anything,
[18:30.800 --> 18:37.840]  even though in a previous call and a couple of exchanges, Daniel, I alluded into that.
[18:37.980 --> 18:43.420]  Whether it's a logo, as a matter of fact, even for Cisco, we had the first vulnerability with
[18:43.420 --> 18:50.120]  emojis. So, we have emojis, logos, and everything else. So, at the end of the day, if it helps
[18:50.680 --> 18:57.360]  to do an alias for a vulnerability and bring some awareness, I'm perfectly okay with that.
[18:57.360 --> 19:02.860]  The only challenge that I'm seeing or opportunity, right, is that in some cases,
[19:02.860 --> 19:09.040]  whenever we write information, and even vendors, in my case, we have a research
[19:09.040 --> 19:13.140]  institution at Cisco called Talos, and we find vulnerabilities in other people's products, right?
[19:13.360 --> 19:19.980]  And yes, it is. So, in that case, yes, we also have logos, and we put them up there,
[19:19.980 --> 19:26.700]  and we put up blog posts and so on, right? But what we need to do collectively is to make sure
[19:26.700 --> 19:33.500]  that the media is not either downplaying it or over-proportioned. So, we have to have a balance,
[19:33.500 --> 19:38.700]  right? And we all say the media, the media, the media, right? But it's our responsibility, right,
[19:38.700 --> 19:46.440]  on how we create our security advisories. And it's both ways. Even though we're vendors here
[19:46.440 --> 19:51.640]  and everything, but we have to point fingers to us. How clear is our advisories? Can we
[19:51.640 --> 19:57.340]  have some collateral that the end consumer actually knows about, you know, what the
[19:57.340 --> 20:02.820]  implications are? And we work with the researcher to make sure that we all understand, you know,
[20:02.820 --> 20:07.560]  what the problem is. And in a previous conversation, I also mentioned that
[20:08.120 --> 20:16.620]  the biggest nerd fights in history is whenever it comes to CVSS scoring, right? And, you know,
[20:16.620 --> 20:20.740]  whenever it comes to risk. And at the end of the day, you know, whether it's CVSS, whenever we
[20:20.740 --> 20:25.300]  come up with some new ways and everything, we have to have some type of way of saying,
[20:25.300 --> 20:31.440]  yeah, you have to jump right now and fix this vulnerability versus the other 100 criticals
[20:31.440 --> 20:36.780]  that we're also, you know, dependent on. And especially nowadays, since it's not so much
[20:36.780 --> 20:41.260]  of a vulnerability coming from like a Cisco or an Intel or Red Hat or anything else, it's
[20:41.260 --> 20:46.600]  open source, right? By the time that actually I'm boring you to death in here, probably three more
[20:46.600 --> 20:52.700]  CVEs that are super important has been disclosed and we don't know about, right? So that's the type
[20:52.700 --> 20:58.540]  of balance and the type of, I guess, for a corny way of saying orchestration that needs to take
[20:58.540 --> 21:05.760]  place, right, in this CBD. I want to add another point here
[21:06.800 --> 21:16.340]  regarding the overhyping. Having a logo, having a website for something, for a vulnerability,
[21:16.340 --> 21:22.320]  for a research result does not necessarily mean that you overhype it. We had experiences where we
[21:22.320 --> 21:32.100]  just put papers on archive without any website, without any logo and anything, and media picked
[21:32.100 --> 21:46.320]  it up and media reported about it and not necessarily correct in all cases. And we had this,
[21:46.320 --> 21:53.060]  on AMD CPUs, and their media picked it up. We intentionally didn't make any website
[21:53.060 --> 22:01.780]  or logo, but media picked it up and the reporting, it was not entirely wrong,
[22:01.780 --> 22:10.760]  but it was definitely, I would say, definitely overhyped. And we discussed this also in our team
[22:10.760 --> 22:17.340]  and the conclusion that we had was that in some cases we should have a website, even if it's not
[22:18.000 --> 22:23.940]  a vulnerability that we want to hype, because it's not that significant, but just to have a very
[22:23.940 --> 22:30.980]  clear message which says how relevant is it. And I think that's the important part that we also
[22:30.980 --> 22:39.120]  always had on our websites, a short three-sentence summary like what is it, who has to care about this
[22:39.120 --> 22:45.840]  and what can an attacker do. Yeah, I think that's great. That's great you do that. And it's good
[22:45.840 --> 22:53.660]  advice. I think that I would love to see it be more adopted in the researcher world,
[22:53.660 --> 23:01.420]  for sure. I think it would help out. So before I move on to my next question, I'm just going to say
[23:01.420 --> 23:07.400]  when I retire, all of you in vendor world are doomed, because I'm going to offer my services
[23:07.400 --> 23:14.600]  to LREG for free writing up crazy headlines. But to touch on a point that Daniel made, he
[23:14.600 --> 23:21.180]  invoked the Katie Masuris clause. Let's talk about bug bounties and CVD.
[23:22.400 --> 23:28.900]  Oh, Katie, maybe you should take that one. Because our friend Katie Noble Trimble here
[23:28.900 --> 23:33.320]  actually has a lot to do with that in her organization. Could you maybe talk about how
[23:33.320 --> 23:38.660]  coordinated disclosure interacts both good and bad with bug bounty?
[23:39.620 --> 23:46.480]  Yeah, so I love Katie Mo. I think that first off, let me just say that it is an honor to
[23:46.480 --> 23:53.500]  be confused with Katie Mo. My hair color is like, you know, getting close to hers too now. I don't
[23:53.500 --> 23:59.460]  know. I think I have been accepted to conferences because they thought that I was she. When I showed
[23:59.460 --> 24:06.520]  up, I don't think they were nearly as excited to see me instead of her. So, yeah, bug bounties.
[24:06.840 --> 24:13.340]  So, bug bounties are a tool. I'm trying to be serious here about bug bounties.
[24:13.980 --> 24:19.460]  So, bug bounties are a tool, right? So, they're a tool in a toolkit. And they are there to
[24:19.980 --> 24:28.380]  incentivize. But they're a part of a well-structured product security portfolio. They're
[24:28.380 --> 24:34.040]  not the whole thing. And there are different motivations that will help people from different
[24:34.040 --> 24:39.640]  perspectives. So, in the academic world, for instance, bug bounties may not weigh as much
[24:39.640 --> 24:45.920]  because a lot of academic institutions don't allow an individual to accept that cold hard cash,
[24:45.920 --> 24:51.000]  right? If you are a professional bug hunter, that might be how you pay your mortgage.
[24:51.000 --> 24:56.620]  And so, there's a lot more tied to that. But the problem becomes that bug bounties are
[24:56.800 --> 25:02.480]  a wonderful tool. And they're great. But you have to have a good program in place already
[25:02.480 --> 25:08.240]  to accept that information, to be able to execute on that information, and figure out, like, how do
[25:08.240 --> 25:15.580]  you actually... there are so many questions about it. How do you award? How do you manage it? Like,
[25:15.580 --> 25:20.960]  are we gonna tie the payments to CVSS scores? Are we gonna tie the payments to a well thought
[25:20.960 --> 25:24.900]  out proof of concept? There's so many pieces that go into it. And so, I would say, like,
[25:24.900 --> 25:31.280]  bug bounty is not a one size fits all. It's gonna vary from organization to organization.
[25:31.420 --> 25:37.240]  And some organizations are gonna have different timelines, different pieces. It's... yeah,
[25:37.240 --> 25:42.600]  it's complex. So, bug bounty is not the end of the world. Bug bounty is a tool. It's a tool in
[25:42.600 --> 25:48.000]  your toolkit. So, it's a great tool. I love the tool because I'm the director of it at Intel.
[25:48.000 --> 25:55.140]  But it's not... it's not everything. So, vulnerability disclosure is more than bug
[25:55.140 --> 26:00.140]  bounty alone. Yeah, I think when you think about it, too, like, there's a lot of smaller
[26:00.140 --> 26:06.600]  companies that probably, you know, don't have that or don't have the funding for it. But that
[26:06.600 --> 26:11.900]  doesn't mean that they're not wanting to work with researchers. They just can't get the budget to do
[26:11.900 --> 26:17.860]  it. You know, so... so, you know, do your best to try to figure out how to reach out to those
[26:17.860 --> 26:24.060]  companies. Hopefully, they have a web page or email address, you know, secure at p-cert at
[26:24.060 --> 26:31.600]  security at. I know we struggle a little bit. But, you know, there's plenty of us around that
[26:31.600 --> 26:39.280]  can help find the right contacts. Because we're, you know, sort of come together from all over the
[26:39.280 --> 26:47.180]  place. And we're sort of an expanding group here. I like it. Bug bounties are one of the wonderful
[26:47.180 --> 26:54.100]  signs of how the industry has changed. Back in the 90s, researchers were often rewarded with...
[26:55.380 --> 27:01.500]  their disclosure was rewarded with lawsuits. And now people in the industry are working with
[27:01.500 --> 27:08.940]  the people and have started actually paying the researchers for their efforts. So I'm
[27:09.280 --> 27:15.800]  very much thumbs up on bug bounties, especially because it shows how the industry has changed
[27:15.800 --> 27:21.700]  on how they view hackers. Hackers are nowadays helpers and not the enemy. Yeah, I thought you
[27:21.700 --> 27:28.440]  were going to say t-shirts for a minute there in the early days. I like it. It just won the t-shirt.
[27:31.400 --> 27:39.260]  Oh, earlier than that, it was lawsuits. Then it became t-shirts and then maybe stickers.
[27:39.280 --> 27:45.620]  And then, yeah, and now financial rewards. Awesome stickers.
[27:48.160 --> 27:52.880]  So let's... we've touched on it a little bit. Let's see if we can spend some time now that we're
[27:52.880 --> 27:57.820]  past the top of the hour here. Let's talk about coordinated vulnerability disclosure
[27:57.820 --> 28:06.840]  inside an open source context. So I'll go last, but let our esteemed panel here,
[28:06.840 --> 28:10.800]  whether it's our researchers or our vendor friends, kind of describe
[28:10.800 --> 28:18.120]  what works out really well and what are some challenges for you within the open source world,
[28:18.120 --> 28:25.380]  which makes up about 90% of all software now. Hey, Jerry, open source won.
[28:37.400 --> 28:47.920]  I guess I start. So basically, that's top of mind for me. And it sounds corny whenever people say,
[28:47.920 --> 28:51.000]  hey, what keeps you up at night and everything else? It's actually open source right now.
[28:51.200 --> 28:57.920]  And it's not so much of the predicament of using open source. We have to use and embrace and
[28:57.920 --> 29:04.440]  contribute to open source. I mean, I'm a super big fan of that. The challenge when it comes to
[29:04.440 --> 29:10.040]  open source is that it can be anything, right? It's like IoT can be anything, right? And it's
[29:10.040 --> 29:19.680]  critical infrastructure for a lot of things. So to give you somewhat of a real life, I guess,
[29:19.680 --> 29:25.100]  realization that we had a while back, probably about four or five years ago, whenever Heartbleed
[29:25.100 --> 29:30.420]  came, we were looking in the industry, right? Not only Cisco, but a whole bunch of other companies,
[29:30.420 --> 29:37.840]  right? So what to prioritize, as far as actually giving funding, probably doing research,
[29:37.840 --> 29:43.260]  and so on. So we said, okay, so open SSL, and things like that, which are actually
[29:44.180 --> 29:50.100]  super important for us. There's a lot of people looking at it right now. So can we look at things
[29:50.100 --> 29:54.840]  that perhaps are actually critical infrastructure, that nobody actually has probably taken a look at
[29:54.840 --> 30:01.140]  it, right? We're going down the list. And as a matter of fact, that number two is perfectly
[30:01.140 --> 30:06.380]  fine. Because there are two guys that work on open SSL that get paid to do it. Yes, indeed.
[30:06.560 --> 30:13.920]  And the example that I was going to go is NTP, the Network Time Protocol, so NTPD specifically.
[30:14.120 --> 30:19.960]  And there's also two guys that don't get paid, which is bad, right? They're not even the
[30:19.960 --> 30:24.660]  full-time job, right? Well, I guess now, you fast-forward this year, it's a little bit better
[30:24.660 --> 30:30.940]  now, right? So it's a lot better. And it's not that the problem is the poor guys that actually
[30:30.940 --> 30:34.920]  are contributing to the code. It's a matter of scalability. It's a matter of actually even
[30:36.100 --> 30:40.980]  running static analysis into the thing that didn't even exist five years ago for these
[30:40.980 --> 30:46.020]  components. Now, of course, we're modernizing our ways. Even if you submit things on GitHub,
[30:46.080 --> 30:50.980]  a lot of things are actually happening. We're getting better, for sure. It's just that it's
[30:50.980 --> 30:57.140]  getting way more complicated, right? And more people actually are not only, of course,
[30:57.140 --> 31:02.940]  contributing, but using it more than contributing, which is the other predicament. And in some cases,
[31:02.940 --> 31:10.960]  we were talking about bug bounties. In some cases, actually, amazing. I'm a big fan of
[31:10.960 --> 31:15.700]  bug bounties, right? The challenge is that in some cases, we also don't think about,
[31:15.700 --> 31:22.360]  does this affect other vendors? Am I actually finding some type of vulnerabilities that perhaps,
[31:22.360 --> 31:27.560]  yes, it's a SQL injection, cross-site scripting, which is actually pretty common, but I'm doing
[31:27.560 --> 31:32.960]  some fuzzing and I crash an application. What is the underlying issue, right? And in some cases,
[31:32.960 --> 31:37.400]  it's actually kind of a commodity, even though it goes back and forth and it hasn't been shared.
[31:37.460 --> 31:43.840]  And then two or three months down the road, you say, oh, but this CVE was not shared with
[31:43.840 --> 31:48.940]  these vendors that they're also affected. So we go back and in some cases, you actually see
[31:48.940 --> 31:53.880]  people trying to find the same or found the same CVEs and they reported a different way, but
[31:54.420 --> 32:00.340]  there were no coordination. So those are the things that are crucial for us to be successful.
[32:00.340 --> 32:05.200]  And not only among vendors, but actually downstreams and upstreams as well, right? So
[32:05.980 --> 32:11.860]  especially whenever it comes to IoT is number one. Yeah. So Omar, you brought up a few points.
[32:11.860 --> 32:19.940]  One is who do you bring in, right? Ahead of time. And I think that we're seeing that even if you're
[32:21.200 --> 32:29.180]  the competition, we still want to work together in the background here. Our whole goal is to
[32:29.180 --> 32:34.580]  protect our customers and we often have the same customers, even if we're competitors.
[32:34.580 --> 32:40.840]  So we like to make sure that we're all working together and ideas is when do you bring someone
[32:40.840 --> 32:46.460]  in and it's not only the researchers. Okay, Crowe, you got your hand raised. Go ahead.
[32:46.500 --> 32:55.160]  Can I tell you a secret? Yes. Open source has worked with our competitors for 25 plus years.
[32:56.200 --> 33:04.740]  Yeah, yeah. Well, I would have to say with some of those other vendors in the industry,
[33:04.740 --> 33:14.720]  it wasn't always so nice. We're getting better. But I think not only the researchers should be
[33:14.720 --> 33:21.220]  thinking who else is affected, but when we are the receiving end, we should be thinking who else is
[33:21.220 --> 33:29.000]  affected and bring them in. I recently worked on an issue, a very recent one, where we are in
[33:29.480 --> 33:35.520]  and the researchers were in right with the industry people who were fixing the issue.
[33:35.520 --> 33:42.920]  And there was great coordination amongst the group there. It was pretty awesome to see
[33:42.920 --> 33:49.180]  of how much we've evolved. How do you do that, though, with like open source, though, when
[33:49.180 --> 33:53.240]  everything's shown? It's super easy.
[33:54.680 --> 33:59.380]  Well, and I guess I'll chime in before we let other folks.
[33:59.380 --> 34:05.540]  I found it very interesting how some vulnerabilities have been patched wide in the open
[34:06.080 --> 34:15.020]  under the cover of some other patch. I don't know what you're talking about at all.
[34:17.980 --> 34:21.180]  So, in general,
[34:23.240 --> 34:29.280]  open source, there is no single definition of what open source is. You could have two young
[34:29.280 --> 34:34.660]  ladies in Bangladesh that have an amazing idea and they're just trying to get this creativity
[34:34.660 --> 34:39.360]  out there to share with the world. You could have large corporations like a lot of the folks
[34:39.360 --> 34:44.080]  represented here. You could have academics. So, open source is a lot of different things. And
[34:44.080 --> 34:50.020]  there's really no one definition or moniker that fits, that works with every community. But if
[34:50.020 --> 34:56.540]  you're thinking about the types of open source that might make up a product or some kind of
[34:56.540 --> 35:04.120]  cloud offering, you're going to have the high end of the spectrum are things like the kernel group,
[35:04.120 --> 35:10.400]  which is very mature and organized, Kubernetes. And then you start moving down to Apache
[35:10.400 --> 35:15.780]  Foundation, these other large communities. And then you get down to something where it's
[35:16.420 --> 35:21.760]  a single person, a couple people that are just playing around. And it's hard to put any kind of
[35:22.500 --> 35:26.720]  structure or process to all these different models because a lot of people that code for
[35:26.720 --> 35:33.520]  open source are doing it for free, for no remuneration from large companies that make a
[35:33.520 --> 35:39.260]  lot of money off of it. But they do it for free because they love kind of adding value and
[35:39.260 --> 35:45.660]  expressing their creativity. And as Daniel mentioned, open source is very creative. And
[35:45.660 --> 35:52.300]  some of the larger projects, we do have ways to privately take in data. So we take in a private
[35:52.300 --> 35:57.660]  bug report that's well established within the community for a long time. And when you're
[35:57.660 --> 36:05.200]  looking at a less mature community or package or library, they might not have that capability. And
[36:05.200 --> 36:10.740]  that's where larger kind of big brothers and big sisters, like a Red Hat or a SUSE or Canonical,
[36:10.740 --> 36:15.380]  kind of step in and try to help mentor these smaller projects to help them get these good
[36:15.380 --> 36:21.640]  practices set up so they can take in a private note. Because sometimes a lot of, you know,
[36:21.640 --> 36:30.020]  my team, we track between 3,000 and 5,000 vulnerabilities a year out of 450,000 packages.
[36:30.020 --> 36:35.580]  That's a lot. Not all of them really kind of bubble up to the level of they need the
[36:35.580 --> 36:42.020]  attention of like a Spectre meltdown or a Heartbleed or a, you know, Blueborn, all these
[36:42.020 --> 36:46.680]  kind of the other big name nonsense things. But, you know, they still need to get fixed. And what
[36:46.680 --> 36:53.080]  open source is really good at is you identify a problem, the team attacks it collaboratively,
[36:53.080 --> 36:58.220]  they develop a solution and release that update very quickly. And they don't like to spend a lot
[36:58.220 --> 37:03.840]  of time making a big deal out of it because they've already moved on to the next feature,
[37:03.840 --> 37:10.460]  the next big thing. But, yeah, it can be challenging. But there are definitely ways to
[37:10.460 --> 37:15.340]  do it. There are methods to do it. There are groups that allow this. I hear a lot of bleeping
[37:15.340 --> 37:19.020]  in the background. What's going on, folks? I was wondering when you were going to change
[37:19.020 --> 37:25.600]  your hat again. Sorry. There we go.
[37:28.420 --> 37:38.960]  All right. Let me get this panel back under order. When you're thinking about doing a CVD,
[37:41.910 --> 37:45.890]  what are some things you might want to prepare for? What can you do to help make that
[37:45.890 --> 37:52.630]  coordination very successful? And let me start off with... we'll start off with Lisa because
[37:52.630 --> 37:58.170]  she has a lot of ideas. So, from your perspective, what will make you successful when you're getting
[37:58.170 --> 38:05.270]  ready to do this multiparty coordination? So, I guess it depends on how familiar you
[38:05.270 --> 38:10.210]  are with it or not. Like, if I was a researcher and I wasn't too familiar, I would potentially
[38:10.210 --> 38:17.250]  utilize CERT or something like that if I felt like it was going to cross over more than one
[38:17.250 --> 38:25.410]  company in the industry just to have them help. Because I think it could be overbearing of
[38:25.410 --> 38:33.770]  figuring out how if one, you know, if industry, you know, company A wants this date to get it
[38:33.770 --> 38:39.730]  fixed and company B wants another date and company C wants another date. So, I think that's one way
[38:40.210 --> 38:48.930]  that a researcher could use. But I think the idea is to figure out when you start is what are your
[38:48.930 --> 38:56.130]  rules? What's your embargo date? How many days do you actually want to, you know, give the company
[38:56.130 --> 39:04.290]  or vendor to fix the issue? Typically, it's 90 days. I would prefer that as a company. I think
[39:04.290 --> 39:10.110]  it's respectful to do that, especially when things get more difficult. So, think about your rules and
[39:10.210 --> 39:15.610]  what you want to do and how you want to approach it. But I'll pause there because I know we're
[39:15.610 --> 39:20.170]  running out of time to get other people in. Anders, what are your thoughts about how you
[39:20.170 --> 39:29.330]  could make CVD successful? Three things. Use it. It's a tool not only for vendors but also for
[39:30.230 --> 39:39.650]  hackers. Listen to what vendors are saying or what hackers are saying
[39:41.470 --> 39:46.550]  and try to make the best out of it and be aware that there's
[39:47.890 --> 39:53.230]  often a lot of complexities involved with it. Right. Be patient, right? Yeah.
[39:55.030 --> 39:59.030]  Omar, what are your thoughts on how you can make CVD successful?
[39:59.330 --> 40:03.710]  I think that I'm going to capitalize on something that Daniel mentioned in a previous
[40:04.790 --> 40:12.030]  call. And it's in some cases, even if you provide some data to even a vendor or whoever,
[40:12.030 --> 40:18.130]  whenever you're coordinating, you have to make it first easy to understand. But at the same time,
[40:18.130 --> 40:24.610]  in some cases, getting the right people at the right place at the right time to make sure that
[40:24.610 --> 40:28.630]  you understand the technical implications of a given vulnerability instead of running around.
[40:28.630 --> 40:33.810]  Even if you have the 90 days or 60 days or 100 days, getting that streamlined and modernizing
[40:33.810 --> 40:40.470]  the way that we actually exchange information among all the affected parties, that is number
[40:40.470 --> 40:46.870]  one. And Art Mannion, which is actually a good friend of mine, he leads... We all love Art.
[40:47.850 --> 40:54.350]  He's the person that will tell you we cannot scale, right? This is a big ecosystem.
[40:54.350 --> 41:01.050]  And just having a Rolodex of all the vendors that actually use a component is foolish, right? We
[41:01.050 --> 41:05.910]  will never, ever, ever be able to actually have a complete thing. But what we have to think about
[41:05.910 --> 41:12.670]  is what Lisa mentioned is that in some cases, we are working with competitors, right? Even more
[41:12.670 --> 41:18.930]  than our companies. For example, you know, I work with Lisa a lot. I work with Juniper even more,
[41:18.930 --> 41:23.250]  in some cases more than I worked with Cisco, whenever it comes to fixing a vulnerability,
[41:23.250 --> 41:28.970]  saying BGP, OSPF, or whatever the case might be. So that type of thing of, oh, I'm going to talk
[41:28.970 --> 41:32.970]  to this competitor, is there a problem and everything else is not, you know, that's 20
[41:32.970 --> 41:41.990]  years ago, a fallacy. And then the last one is that at the end of the day, I have the reality
[41:41.990 --> 41:47.490]  and I tell my guys at Cisco is that whenever I push a button, I publish a CV out there,
[41:47.490 --> 41:52.810]  the number ones that actually are reading that CV are the bad guys, right? And probably
[41:53.250 --> 41:57.630]  at the same time that you're committing an open source code, you know, probably that's going to be
[41:57.630 --> 42:04.610]  the case too, right? So what we have to do is think about how can we exchange information to
[42:04.610 --> 42:12.270]  the consumer, to downstream and upstream providers in a more modern way. I know that sounds corny,
[42:12.270 --> 42:17.890]  but you know, can we actually make it more and we're trying to do that machine readable, right?
[42:17.890 --> 42:22.430]  So if in the case that we actually have that Rolodex of people and everything else, you know,
[42:22.430 --> 42:28.050]  get some tool and that's what Art and, you know, the guys in CERT are also doing, you know,
[42:28.050 --> 42:33.470]  we're creating tools that allow us to do that. And even if you have, I don't know, even two years ago
[42:33.470 --> 42:38.010]  that didn't exist and it wasn't even a thought, right? So we have to move faster. That's the number
[42:38.010 --> 42:45.930]  one thing that I want to say. So as I close, I'll say Art Mannion has the best facial hair
[42:45.930 --> 42:55.150]  in the industry. But I want to thank our panel here. This was a great hour together. I really
[42:55.150 --> 43:00.230]  appreciate your time and expertise. We're going to be hanging out for a little bit on this awesome
[43:00.230 --> 43:05.530]  Discord channel that my kids made fun of me all week about because I've had it up. Thank you
[43:05.530 --> 43:10.250]  everybody for coming and your attention here, panelists, and thank you to the audience for
[43:10.250 --> 43:16.010]  having us here at the IoT Village. Enjoy your day and enjoy the rest of the con.
[43:16.010 --> 43:18.330]  We've got some great stuff lined up the rest of the weekend.
[43:25.450 --> 43:26.490]  We're out.