Thanks for joining Return-Oriented Policymaking here at the Policy Village at DEF CON. We're going to talk about open source and supply chain security. And I say we, really it's going to be this illustrious panel you see sitting on either side of me. I'm going to quickly introduce them. We're going to go down the line, let everybody offer some thoughts on this issue. The framing for the discussion. Oh, okay, got it. Is there a way we can tamp down the, I just don't want to kill people. Got it. Okay, the framing for this discussion is we've got some really interesting pressing challenges in the supply chain security challenge of open source. How do we make choices about dependencies, pass those dependencies upstream, make choices about products we're building for users, how do users manage risk for those products, looking at open source code. The return-oriented policymaking framing for this is what can we do with policy that's already on the books. So rather than write a new law, come up with a brand new construct, how do we use existing? Organizations, existing budgets, existing authorities more creatively to try and address some of these challenges. So I would encourage folks to come to the table with challenges they see as well as interesting applications of law, of policy that we have kind of already in play. Try to make this as close to a problem solving session as we can. This is going to be Chatham House, so you are free to take notes. Just please do not attribute anything that you hear here. We're going to have some notes and hopefully write those notes up in some kind of fun, graphical form. Once we're all done and tweet at DEF CON as everyone leaves the CON and is very sad and we'll want reminders of what we did here. Before we get into this, let me introduce our panelists. First, our, our, every, almost everybody's from the government and here to help. Harry Mortos, here from the Office of the National Cyber Director. Eric Mill, with the Office of Management and Budget, technically the CIO office, correct? Excellent. Ava Black, who wears many hats. They are here in their personal capacity, however, and although all the hats are exciting, none of them will be represented directly here. Which I think is accurate. All right. Brian Bellendorf, who is Executive Director with the OpenSSF, Open Source Software Security Foundation. And then Mr. Jack Cable, the man from Congress. With that, let's go back down the line, get a couple of thoughts quickly from folks, then we'll open it up for more general discussion. All right. Hey, good afternoon, everyone. I'm Harry Mortos with the Office of the National Cyber Director. Thank you for coming to listen to our roundtable discussion. Um. Just to open it up, I do want to take a moment and just take a step back from the open source and talk more, just for a moment, if you'll indulge me, about policy and cybersecurity and how thrilled I am that DEF CON has a policy track this year. Uh, it's, it's a very good development in my view. I think that oftentimes we view sort of the cybersecurity community with this barrier between the technical and the policy. And it's in all of our best interest to sort of break down that barrier wherever we see it in order to, one, encourage those with technical expertise to engage in good policymaking, uh, that expertise is extremely valuable to help influence decisions and, and, and how we think about these problems. And two, to, as, as ambassadors of the community, we are all, whether we're conscious of it or not, ambassadors of cybersecurity as, as a whole. When we interact with those who are not members of our community. So, allowing on-ramps for our policymakers to be able to, you know, engage and make the technical aspects of what we think are critical on a scientist's level. do a little bit more accessible wherever we can um is in all of our best interest so i just wanted to take a moment and thank defcon for creating this track i'm really excited to see you know what's going to come of this thanks uh hi yeah so uh eric mill with omb with the federal stereo office um so uh i think it is important to zoom out like harry talked about and at the top here is one of the things that i think rides through a whole lot of the certainly the policy track here at defcon are some of the consequences of how we sort of other technology and cyber security in american society and global society where it is just something that is happening over there that we don't need to understand until there's a crisis at which point we want to understand everything and i think open source security is a really good example of something that is uh accessible and understandable to just a very small uh almost like a little bit more accessible and understandable to just a very small almost like a little bit more a little priesthood in in society that um all of a sudden when something bad happens that involves open source security uh you know people kind of come out of the woodwork uh all of a sudden paying attention like a laser focus to this thing and they hear about it how it works and they're like well that can't be right and that that can't that can't possibly be uh the the foundation on which all society is built these days but it is since it is that way um this is one of those issues where it's really important for us all to to i think be bridging the gap between sort of you know lar what large risk averse organizations and institutions that are really important need and then just the reality on the ground of like how of how software moves around and is built in the modern day um that's certainly our goal uh to to try to be smart about it and to represent um those issues and bridge them uh and really looking forward to what folks uh bring to the discussion um so as trey said i'm eva and i do wear a lot of hats and work at microsoft i'm on the technical advisory committee for the open ssf and on the board of directors for the open source initiative that's the neutral uh body that sort of defines what is open source what is an open source software license um and uh but yeah i'm here talking more as an independent right now i would like to suggest to everyone especially policy makers and folks in the room that any solution we come up with has to balance an individual's right to freedom and freedom of expression expression through publishing source code and that's grounded in the bernstein case from the 90s has to balance that against the responsibility of corporate consumers of open source or anybody who's integrating open source into a product that that is where the responsibility should lie to ensure the safety of the product not upstream on someone who's expressing their their free speech um to that i would suggest we think about trusting the process not the information of the service so i think this is a really important point in this uh we have to think person right not the individual but how that open source project is being built managed audited secured looked at there are some examples out there and there's two paradigms here the four freedoms folks may have heard about in open source the freedom to share to use to inspect to modify and the four opens which is a much more recent innovation in open source communities it describes how to build open source software collaboratively transparently and accountably um the challenges really that we're facing here are inseparable from the challenges also facing privacy rights free expression and queer and trans people in society and reproductive justice so like ask everyone to think about any second order effects of any policy that we might be creating not just the return but what are the second order effects on civil society and free expression and civil rights from anything we're doing and so that everyone can participate equally in this incredible acceleration to society to science or technology that's happened through open source in the past 25 years thank you hey everyone um brian bellendorf as trade mentioned i lead the open source security foundation at the linux foundation uh i think uh if you're in this game you have to wear lots of hats uh so i'm also on the board of the electronic frontier foundation and the mozilla foundation and a few other things so um i you know i think trade framing is an interesting one because so much of the work in open source has proceeded without the direct implicit explicit engagement of government uh i i you know if we like to cite you know the history of the internet and the like as like plausibly coming from research that was funded by by the government and certainly it's been supportive from a policy point of view but open source software is largely been a phenomenon of the private sector it's had lots of volunteerism lots of input from universities of course but you know there's it's also been somewhat of a short path to to solving problems rather than necessarily thinking long term i i the the folks involved in it in open source software have tended to be about solving into scratching your own itch right solving your immediate problem we have foundations in this domain that are chartered with thinking long term and chartered with trying to set up just enough processes and and and alike to allow these projects to live beyond their original creators um i i to solve the so-called hit by a bus problem or as we like to call it now going to fiji um but i i but security is on only recently been a priority for for most of those uh and so at the open source security foundation what we're seeing uh is a mix of companies uh and individual uh open source developers and and now foundations coming together and saying are there ways in which we've the way that we do the supply chain the way that developers choose what dependencies to include uh and and the build processes that they adopt are there systemic weaknesses in this that um outside of solving scratching our own edge or solving an individual problem we can work to together to address what's the role for government in that well government is a major consumer of software right just like if you were to count them as a as an organization they'd be larger than perhaps the the top three four ten companies combined i somebody should do the math um but uh but they're also now a producer of open source code so they have a role commensurate to that to that size um but secondly they do have this kind of role as the guarantor of last resort uh in so many industries from the banking sector to transportation to uh to insurance it's like you know we kind of depend upon them to to help the uh keep the foundations of society running and i think the log for shell debacle by the way i want to declare moratorium on using the word log for j because that's a project with real developers and people behind it it was the log for shell all right all right so i think the log for shell uh thing kind of demonstrated to us how systematic this code and pervasive this code is and the other thing that i want to add is i think what we've seen in the past you know protecting everything from banking websites to to security doors that that were compromisable with an rfid tag with just the right uh kind of uh uh you know bad content so um this requires an olive society response and i think government can play a role as a convener in that but i think doing it in a way that that is a peer to industry and is also inherently global is important as well we do have to think of kind of the government's role in helping secure open source software from as many different lenses as possible there's the consumer end where u.s government is probably one of the largest if not the largest consumers of open source software in the world and if we look at kind of how much it uses versus how much it puts out or how much it contributes back i think we'd find that we've probably been under contributing to this community so needing to step up both there either through kind of natively used software natively developed software or software used through contracts and thinking through how we can make sure that kind of this is feeding back into the open source ecosystem and allowing kind of the contributions that happen government not just to stay in government but benefit everyone using open source software and then i think there's the perspective of say critical infrastructure where open source really does underpin pretty much every critical infrastructure their sector there is today so it's just a fact that open source if it underpins this it is an essential layer that the u.s government as its responsibility in helping protect critical infrastructure should view open source software as a fundamental component of that so being able to help say ensure that we have this long-term security not just kind of thinking in the short term but what can we do to make sure that throughout the long-term open source ecosystem is as secure as it can be and looking forward to kind of getting into some of those areas where we can help ensure security down the road that's awesome appreciate that from all so i'm going to take moderator's prerogative for a minute uh and ask one question but then we'll welcome folks in the group and if you can as we as we go around just be friendly to pass the mics down so everybody can be heard um as i listen to the to everyone what i heard were at least three kind of common themes of where government can play a role what it does already well it is able to convene right it brings folks together creates some kind of trusted ground for negotiation where it's potentially not an interested party although i think as the point was made by several of you it is an interested party it is a consumer in this case as much as it is a regulator the second is to fund things and consider the power of the public purse to support this sort of infrastructure which to me is a really interesting framing because i think to the to the point that brian makes about log for shell so often the policy framing for open source is something's on fire put it out as opposed to how do we make this healthier how do we grow these communities how do we make this a stronger ecosystem how do we think about protections beyond just the security of an actual product or a piece of code so these are absolutely core government functions so thinking about that that allocation of resources the last though and i'm curious for folks and we'll go down the line again here in a second if we have the convener and we have the funder we also have this this big lens this public good view this national security or security of the country much broader frame than a single product a single vendor a single user base and so i wonder if we could sort of reverse the flow start with jack and come down where are there where is there an existing program or an opportunity for an activity that that really leverages one of these three or another if you've got a thought one of these three constructs of where government really does play a role in the technology ecosystem in public policy yeah i would i mean i think that at least one area i think the government can help contribute is kind of understanding just at a base level how and where open source software is used and what is in need of the most support and there's been some great work already like at opensf the scorecards for instance to evaluate the security of a given open source component but just the nature of the government we have hundreds of agencies all using open source software there's tons and tons of information out there around what packages are used what types of products are used what are the where they're used looking even further say across critical infrastructure government might be able to kind of do some coalescing around there kind of with input from organizations in those sectors so kind of taking the government's picture of kind of this both across the government but then also across other sectors and being able to say this is the state of play and this is where we need to do better um i think is at least a starting point because if we don't understand what's what's most in need then it's going to be difficult to address that to bill and jack's point um i you all are familiar with the acronym ospo open source program office this is a type of organization many large companies have set up and increasingly smaller companies what do you lead microsoft's hospital or you're okay you're not welcome okay um but but almost every every large company has them now you even folks like walmart and starbucks and nike and those kinds of things and this is an office chartered with understanding how a company is using open source code uh and what policies like what what licenses are allowed and then as their developers are feeding back whether it's reporting a bug but hopefully submitting patches or releasing new code what are the policies and the right ways to do that then an ospo is a is a really useful uh leverage point to understand not only internally where is this being used and to do some of that espon style kind of inventory taking um but also outbound like how do we up level our own contributors so that they're taking security classes they're not they're not they're not inadvertently uh going to release something that causes a lot of problems or maybe you pair up so that a bad actor on your staff doesn't end up creating a reputation problem for you every government agency should have an ospo too as a component of the cio's office or the cso's office made arguably that it again is chartered with helping encourage that agency to consume our open source but then also start feeding that back upstream as well whether it's directly or indirectly so i think that's that's perhaps one useful thing that governments could think about but secondly i'm really inspired by the thought of open source software as a public good if anyone's familiar with the economist marianne matsukado she's been really one of the leading kind of thinkers in the domain of public goods that actually serve as accelerators for economic growth at the same time right and the role that that government has historically played in helping that i mean you could even consider the highway system as like a canonical example of that government today there are organizations like the national science foundation i mean it's arguably kind of quasi-independent but there's still a ton of government funding in nsf right there's nasa i mean the orange juice from the moon program kind of thing is a little bit of a silly example but uh i was tang i'm sorry tang from from the moon program um but uh but i think there's these parts of the federal government that have played this role in more basic r d basic kinds of funding that don't necessarily have direct kind of implementation as like their prerogative that are useful to think about perhaps directing some of that funding towards open source software development but also specifically towards hardening oh we're going to go are we okay okay so um i'll build a little bit on brian's suggestion we love the the oslo and government suggestion um i also want to highlight that one of the the factors that i think has led to the lack of security state that we're in right now and the the sort of rapid increase in major vulnerabilities we've seen in the past 10 years we didn't have as much adoption but also velocity and amount of code came from the vc cycle uh starting about 2008 we saw a huge amount of funding flow in through the sort of whether it's big company or startup um rapidly pumping money into building an open source company and air quotes um that produced an open source project maybe a whole bunch of projects and then walked away from them and that creates these pockets where there are few maintainers who still know how to uh continue to update and secure and fix some of these critical projects or products that have been forgotten about very deep in the the substratum of our infrastructure that um and ultimately develop our burnout right we have a sustainability problem in open source that's leading to a lot of these issues where the companies that are funding the work today are funding it on a growth model not on a sustainability model and i think there's a real opportunity for government to come in whether it's to fund research and studies into that into developer burnout into velocity and what are the incentives being applied that are not leading to good security practices and to take a public good view on how do we address sustainability of open source maintainership for all projects um one of the key things that's really important is that you don't want to focus on single project projects the only way to do it is to speak up or to speak up and to say that um you know today we're working on a project that we're working on that's committed to a sustainable development of open source but this is not a project that's carefully thought out it's a project that's very important because it's going to do a lot of work of maintainers and their desire to keep working in that role and not get burned out. So how do we measure that and address that as a public good? Well, I'll try to be brief to get the questions. Also, because they, in fact, covered a bunch of the things that I might say. So it's definitely, you know, the points made here about contributing back and about and being able to take the long-term view, those are things that really can play to the government's strengths when the government knows that it should. So it should be incredibly normal for people in federal agencies to, you know, as they invest in the development of their own software, of their custom software through their contractors, it should be considered not a waste and a sort of fun routine act to also be contributing back to some of the libraries that make up the things that they use and benefit from. And, you know, the intuition to do that is stronger for things that aren't software inside federal agencies. It's, you know, just sort of more this is just something that only a much smaller slice of a federal agency understands, and that understanding tends to go less high up the chain. Long-term thinking and the ability to just fund tooling is, you know, is absolutely, that's a core governmental mission, and something where it's sort of thereby designed to be a counterweight to some of the more growth-oriented incentives on the private sector. One of the things that I know Brian has talked about, and maybe we'll talk about here a little bit, that is one of the places that I know we want to see more investment in software security generally, and especially open-source software, is memory on safety. And that is an excellent example of a decades-long effort that, you know, we, you know, both the government and private sector could be doing work to shave years off of that timeline. You know, and to turn something that could, you know, could without much effort be a 30-year, 40-year problem into something more like a 15- or 20-year problem. And those are, that's how the government can think of things when it wants to, and it's something we'd certainly like to see. So, Trey, I think, summed it up very well when he talked about the three roles that the government can play, right? The convening authority, the funding, and the public good. From the funding perspective and the resourcing perspective, again, just echoing what Eric had said, the government is a producer as well as a consumer of open-source software. In a past life, I ran the President's Cup cybersecurity competition. We developed a platform to run a hands-on cybersecurity competition. It was critical to us that we open-source everything that we developed because we felt that it could contribute back to that public good, and now there are cybersecurity competitions all over the country that are using the code and the challenges that we developed. So we're really, really happy with that, and we really understand the value that providing these resources as open-source and allowing people to use them at their will has really paid dividends. One of the things that Eric said that I would also like to piggyback on on the memory safety piece, and it ties a little bit back to the convening function that we have. We had a one of the other hats that I wear at ONCD is for workforce education and training. We had a summit about a month and a half ago where we convened a large body of CEOs, government officials, to talk about the cyber workforce to talk about the cyber workforce crisis and the shortage of skilled professionals that we have out there. And in that discussion, memory safety specifically came up when we were talking about how we train software developers, and there was an individual who worked very closely with colleges and universities who asked one of his professors, why are we not emphasizing memory safe languages? And he says, we don't have the curriculum. This is what the book says we teach, so that's what we teach. So there's a lot of roles that the government can play that may seem like ancillary or secondary order of effects that can really have a big impact, and when we talk about memory safety in the workforce, you know, understanding how we can influence developers who have been coding in a particular language for 15, 20 years, and they may not have as strong skills in some of these newer memory safe languages. How do we make them more comfortable and get them up to speed to be able to contribute to some of these projects? I'll keep my remarks brief. I know there are some questions floating around, so I'll turn it over. Thank you all for that. One linking theme which is interesting here is government shaping the market, right? Shaping the ability for others to have transactions in a way that accumulate in a positive fashion, build curricula, change incentives, right? Change the way that we approach code. Sir, let's come to you for the first question, and then we'll come back down to you. Hey, greetings, folks. I'm Darius. I'm a UX UI designer at Microsoft. Opinions are my own, however. Nice to be here. So this is a bit of an esoteric question, maybe one that we should have asked in the very beginning of this. What do we mean when we say open source software? And how important is it that we have a shared understanding of what that is in this context, and especially as we try to scale these things? I'll grab that as a board member on the organization that literally defines what is open source software. Any software published under a license approved by the OSI is open source software. Did you mean, was there more to your question? I mean, this is a bit of a controversial topic, right? Like, free software, open source software, GPL, MIT, like, are they all open source? There's a list of licenses on the OSI site. And it is a superset, I believe. Yeah. It's a superset of the FSF's list. So do you think the minutia between these two licenses is important moving forward? I do not think it is relevant to this. But if you think it is, I'd love to understand how you think it's relevant. Oh, I could talk all day about that. Let's go offline. So to frame that differently, I think what you're asking is should there be a policy decision around whether to use strong copy left, or permissive licenses? Is that what you're asking? Exactly. Not particularly, but I will anyway. Yeah, I mean, I'll just say, for what we were talking about today, at least everything that we've gone over, I think, you know, just speaking on how we engage on it, what's the distinction that's sort of a practical matter is whether it's something that you're going and developing a private contract with an organization to produce for you or doing in-house versus something where your levers of control and review and production are much more indirect. Or something that's just sort of outside the traditional procurement and commissioning process of a thing that you're doing. So that is, you know, just as a practical construct for how we're looking at it. There's actually one other nuance that I'll add to the way you're framing it. If the government is incentivizing the creation of software, the license has a direct influence on who retains IP control, patents, and trademarks of the resulting work. And that, I think, is an interesting question. Yeah, several hats ago, I worked for the Sunlight Foundation, which is a government transparency organization, and it did quite a bit of advocacy to the government on the licensing it should use. And then at GSA, we did a whole thing around basically having all of our stuff come out of CC0 without disturbing sort of local U.S. government public domain stature. So definitely, there's a lot we could talk about about that. It is the sort of thing that could totally derail this conversation if we let it. But really, I think just from like a down to the security point of it, I know one of the things that really folks are looking at from a systemic point of view is how do you generate better security outcomes with less direct, the traditional kind of control that organizations are accustomed to having to affect those. So I think if we have time, let's come back to this, because the question around I think the control, the IP, the piece that you mentioned as well, do have some implications for both the economic model downstream, as well as security when we talk about ownership and relative authority. So that is, it's not a trivial question, which I think is a good one, but it is interesting where it has been fought over in some really significant ways already. Sir, I know you've waited patiently. Let's come to you. So I'm just going to start with I'm here in my personal capacity as a long-time member of the DEF CON community, but my comments are going to be government inside baseball. So one of the things that I can say as someone inside of agencies is that actually the licensing model is one that fundamentally derails the ability to support open source software. You cannot get past acquisitions. Very similarly, the ability to provide a grant or a contract. Open source developers are often not interested in standing up a business to be able to continue doing what they love. And that is actually a fundamental impairment of the ability to do that. And so just sort of in terms of policy and trying to sort of make it a common question to look at is just sort of trying to find ways to deal with the fact that government workers' work is public domain. The government has the ability to buy code scanning tools, but what is the implication of taking a commercial product and then applying it against an open source project and have the output of the work potentially be under a public domain or CC0 kind of license? And how does that integrate in if the project is GPL? And yet, all of this comes back to fundamentally what we're talking about is this is critical infrastructure software and the ability of the government to contribute to those projects for federal staff, for contract staff, etc. Those are big questions that actually often the inability to answer that question will derail that entire conversation and the ability to actually contribute back. Thank you. Appreciate that. Licenses become a proxy for what is actually a debate about agency. And I don't mean government agency, I mean kind of who has control or authority or even influence over a given piece of software, right? I think we've tended to think of software... When I use software, sometimes I choose package to use based a little bit more on how much I trust the developers behind it than necessarily what the features are in it, right? And I think a lot of open source developers make these very uneasy kind of judgments too out of these kind of weak social signals, right? And there's some groups like Chaos that have tried to kind of define community metrics to try to describe, here's a healthy open source project that's more likely to generate code that is likely to be higher quality by virtue of having multiple people signing off on releases and that kind of thing. And within OpenSSF, we have a couple things like the best practices badge and scorecards and other things that are varying degrees of automated ways to try to evaluate is this project more likely to lead to better quality software? I do think there's more that can and should be done down that path to help developers make decisions about what dependencies to incorporate, what platforms to build on top of, potentially as well for other actors to say, what's my total risk in my enterprise from the use of this code, right? And where in this collection of thousands of components I'm using might there be that Lib Nebraska, right? Lib Nebraska, sorry, Lib Nebraska that is the problematic thing that's going to end up, because it's one developer behind it who has gone to Fiji or worse has sold their credentials on GitHub to a black hat and it's going to cause some problems, right? So I that's I think a better path to getting to answer some of these questions is how do you, and for government trying to decide when they're using software, who am I really building a relationship with? Am I building a relationship with a vendor who's going to buffer some of that wildness for me and if they improve it they'll depend on them to upstream it? Or how much are my developers in-house trying to engage directly? And I'm encouraged by what I see as signs in the federal government and other governments to more directly allow their own state employees to use software and make those kinds of decisions, contribute upstream, whether it's USDS, 18F, those kinds of organizations, right? So yeah, I'd rather like detach it a bit from some of the more political and emotional kinds of things that I think have kind of kept us from succeeding as I think you've noted. Yeah, and I'll add that I think the licensing as you said is a proxy for control. It's a proxy also for who can profit from it. Right? And who controls the IP that comes out of the research of building it, right? We see this in the medical field where gov incentivizes research into things, vaccines, etc., and then companies retain that IP. If, well, different license models will result in different patterns there for industry and education and government collaboration. I think Stallman invented the GPL somewhat as a reaction to the Bayh-Dole Act. Somebody who's a historian probably can describe this better than I, but federal funded research used to generally be available to anybody who wanted to commercialize it. The Bayh-Dole Act in 84, 86, something like that, steered policy towards exclusive licensing in the interest of commercialization as a faster path to market than a lower IP regime. In some respects, GPL was about trying to, I don't want to say keep people from profiting, because you can profit tons off the Linux kernel. Absolutely. But from locking things back down. To keep things open in the public good. In getting a snowball effect going. Yeah. Okay. Can you hear me? Not licensing or non-propriety related question. Just an area that I thought the observations of this wonderful panel of experts would be wonderful, not a specific question for me. Definitely, I think on the open source security questions we have seen certainly globally a number of approaches. We obviously have the federal procurement and the executive order underlying that and of course NDAA as well, looking into measures of creating incentives for that vehicle in the United States, but also broader conversations around critical infrastructure based on security measures and best practices. So focusing in addition to federal procurement on the critical infrastructure sector, but we've also seen, especially in Europe and NAS2 and the CRA, so it's the Cyber Resilience Act, an approach that is looking into, based on security measures for connected devices, that is putting a much more bigger focus on open source transparency and SBOM and even additional requirements for developers as well as potential expansion in the NAS2 to additional sectors. And I was just wondering to get this panel perspective on those different segments as vehicles for incentives whether it's through federal procurement or much broader kind of regulation on critical infrastructure providers or through a specific sector that is identified as like connected products or through approaches that are creating kind of more market adoption mechanisms like labeling. What are your views in terms of the different options that government policy makers globally have to advance open source security and just kind of what are the considerations when you're thinking about these different alternatives. I think the other comment but I'm not sure that we will have time is I'm very curious, one of my observations is that open source security is also and the issues that we have identified there kind of underlying the intersection of most of the policy priorities we are tackling right now whether it's supply chain security workforce development, vulnerability disclosure and incident response so how do you see all these conversations coming together and specifically interacting with the unique challenges that are on the table for open source and specifically the OSSF mobilization plan I think that touches and kind of informs all those different conversations. Thanks. Anyone want to start? Sure. And will you want to say anything too Harry? No. Alright great. So I mean you named a couple of spaces I think I'll just talk about one a little which is there is an effort right now to really and I think you know you named federal procurement through the executive order. There is absolutely an effort to increase incentives around secure software development. There's a bunch of policy work being done in that area I really would the federal government has a nasty habit of asking for so much you know for putting vetting and review and demanding that it get the most reviewed and tight form of things and then ending up with the slowest and worst form of things and so there's a lot of well intentioned government things that can end up having some significant second order effects. And I think everybody wants this to not be an example of that and I really would encourage folks to engage like I think NIST is going to do a workshop later this summer around software development and how to apply the secure software development framework with an eye towards open source software any sort of major action the government takes will almost invariably have a public comment process associated with it. I really would encourage folks to get involved and you know take advantage of the political will that exists to create some good incentives and have that be done in a way that is in connection with the reality of how things turn out. Just as a high level thing would really just encourage folks to engage. Yeah if I can follow up on that to just sort of back stop Eric's point there is this broader political will that exists right now there's a lot of attention and movement and it really feels like we're moving in the right direction and we have the capacity to do some good things in this space so engaging now is it's a good time to be engaged in this effort you did touch on one thing that I'm going to echo in regards to sector specific guidance which I'm glad that you used the word guidance and not the dreaded R word that often is thrown around in this space but it is important to know that the government has a unique responsibility when it comes to ensuring the safety and security of Americans and there is a nexus and intersection there with open source software and cyber security practices more broadly in the sectors and she did throw out the R word here oh she did just to add on to that so yeah I agree I think we need to be thinking about this both at like an ecosystem wide scale of how are we going to kind of lift the tide across the open source ecosystem but also kind of in these individual instances whether it's consumer products or other areas where open source software is inevitably tied in and I think like even as we address the ecosystem open source software is going to be around the security of open source software is going to be something we deal with and there's just as there's unique benefits of open source software for security there's also unique challenges and I think that needs to be fed into the conversation kind of for anything that could involve open source software so it is something that is kind of top of mind and not just something we can say okay we're kind of throwing some money at it so we can stop thinking about open source software security and then on the kind of ecosystem wide scale and this ties into some of what Eric and Harry were getting at with memory safety I think we do need to be thinking about how can the government most effectively use its resources to advance the state of security with open source software as in what are the essentially how can we get the biggest bang for our buck wherever that is what are these long term improvements that aren't just say patching individual vulnerabilities in a single project but kind of leading to say for the example of memory safety something that once you've done it you're done and you've eliminated entire classes of vulnerabilities so kind of continuing to identify those areas where we can get a solid return on investment that can then ensure this long term security To speak specifically about SBOMs first I think it was incredibly helpful to have a clear signal from the White House and from the government in Executive Order 14-028 last year May that said SBOMs are important I mean even before Log4Shell like SBOMs are important we need to be able to take inventory there isn't anything else out there that does it it's like buying a bottle of ketchup you expect the ingredients on that list right and it was a wake up call to industry which kind of dragged its feet in talking about standards talking about implementations and the open source community that had done nothing really on SBOMs I mean we had some libraries some tooling Cyclone DX SPDX but even to this point the vast majority I'd love to get numbers on this but I believe the vast majority of open source software does not contain a bill of materials in every package when they ship out so a mandate like that can have one of two effects it can be something that's satisfied by the end of the supply chain where the vendors think of it like Ubuntu Red Hat or IBM or whoever is distributing the software the last mile you know go through tremendous effort to inventory everything in that package by hand and then you know create a beautiful SBOM at tremendous cost which then they will be incentivized and tempted to lock down or we get SBOM generation and consumption permeated upstream as far as we can right and that means less cost to the tail end less likelihood that becomes part of the proprietary value add of the last mile so at the OpenSSF part of this mobilization plan that Amit mentioned this is something we've put together we released in May 10 different streams of ways that we would like to improve the state of software security open source software security in the supply chain meeting kind of top down with resources and plans the bottoms up stuff going on inside the OpenSSF on a number of different domains on SBOMs the whole premise is how do we get channel just the right degree of funding into tooling into the default billing systems the default ways that software is distributed through the chain so that it's almost zero lift for the developers or as minimal lift as possible to get that as far upstream and make that pervasive make it ubiquitous right make it something that is as standard as a readme dot txt in the root directory and so that's a place where government nudging and government even requirements at one point are useful kind of like stick to get us to think about things but there's still a lot for the private sector to do to figure out how do we get shape of that to be as as equitable and easy for all of us to consume as possible and that's why we're trying to organize funding to get the libraries to be great to think about interop between cyclone DX and SPDX and to get it into every language ecosystem out there into the default build tools it's not a small project but it's one that I think we've got some really good starts on I can talk about the other nine streams at some point they include memory safety they include education but it's a pretty fun and ambitious plan so building on the topic of SBOMs I have a little bit of a different perspective and Brian and I often go back and forth on this the last mile cannot solve this effectively it is an incredible cost and decreasing accuracy the further down the supply chain you try to do that inventory in one of the biggest problems today is even those companies and projects that are producing an SBOM are only doing it at a depth of maybe one or two if they're lucky the minimum elements as defined by NTIA are inadequate today and I chat with Alan about this all the time it's a starting point it's a good starting point but it is not sufficient for what we need as an industry it's a tuple of three items that are not unique they're not canonical they're not immutable and the minimum depth is only one which really doesn't tell you if log4j is in your product it doesn't go down all the way to do that to Brian's point we have to move the generation of SBOMs up that supply chain and at zero cost to the developers to the maintainers of these small projects if it costs them time money or attention and skill they won't do it because a lot of them are doing it as a hobby just for fun or if not incentivized right now in some way for the non-commercial production of open source software to address it to add it there I think the focus area should be have it built have something maybe not an SBOM but some sort of an artifact dependency graph automatically generated by every compiler, linker, build tool and not just the CI systems that are doing the last mile of testing but actually like your GCC, LLVM LD the actual compiler itself all the way up through the one time compiled languages like javascript and python if we could instrument that across the industry then it becomes transparent and zero cost and zero friction for all developers whether they're working for fun or for profit and then at the last mile you have that dependency graph all the way back to which you can correlate all the extra metadata you need I think that solves these two parts and then to go back to one of the questions you raised about are there verticals smart devices I think is a particularly interesting one to look at as a fulcrum for a lot of this because if we do not have product labeling just in the sense of what's the ingredient list in this ketchup but was there a salmonella outbreak in the factory for one of the ingredients of this ketchup we have that capability for food in the food industry for medical devices in cars if there's a recall on brake pads I don't need an ingredient inventory list of what brake pad is in my car but my dealer is going to call me if there's a recall we need a comparable capacity for smart home devices IOT smart vehicles all of that so that when there is a critical issue that cannot be easily remediated you can notify the customers to say hey come in get this thing patched in person or replaced or whatever I guess just a small thing to jump way back to like do you think levers are needed my experience as a working software engineer as a web developer for about a decade was that I did not care about security at all and you know I ran pip install with root privileges on my laptop like it was going out of style and eventually once I got into the security space more broadly and just developed a little sense of adversarial thinking and just became a little bit more familiar with what was going on and applied that thinking to my own practices I changed and incorporated that as one of the things I cared about the most but most people don't and shouldn't have to go become into security in order to you know care and understand about those sorts of things it's not very scalable and it's not reasonable I also I spent a lot of time working on just encrypting the web generally there was you know especially in like 2013 to 2019 it was probably like a peak period where a ton of organizations on the internet web browsers governments individuals people were working very hard to you know change the default for how the web worked and it was interesting to me that one of the places where the backlash was most significant was whenever Firefox would tighten the reins on something and they would get a huge backlash from software engineers and web developers about how their lives were being made harder for it it was very similar to when automatic updates came up as a concept in the web browser community about a decade ago where it was the software engineer community that pushed back on it for a sense of self control and it took a while and other organizations to get in the mix before that eventually became the norm and that is you know the best form of all this is for the community to look upon itself and you know prioritize security and think about it in the way that they think about performance and scalability and all of the other things that feel more visceral to the individuals involved but it has been the norm that there have had to be at least some kind of external pokes on the system to get security to be taken seriously and that has just been my experience as a professional for a while so go for it if I can just real quick one of the interesting things about this discussion when we are talking about especially using the car example and the brake pads I think that is a really good one the car manufacturer will call the dealership to call you because the car manufacturer has substantial liability in this case and I think it is a good question to ask I am not going to propose any broad proposals or solutions and Chatham House rules just as a reminder but who owns the risk of open source software I mean is it the responsibility of the hobbyist developer who pulled together a package to solve a personal problem or is it the responsibility of the organization that then took that package and put it in software that they sold to critical infrastructure providers to healthcare manufacturers device manufacturers etc to me the answer is pretty simple it is the second one go ahead I know some of the agencies are working very hard on this and getting even below way below more than one or two levels right now so I am confident that is there any effort in the government to take that list to SBOMs and to give bounties to fix them in the open source software and also if you look at all these especially the smaller guys they have no idea how to add something to the report a CV to the database to democratize that CV a little process or something like that of this whole list of SBOMs that we have in the government so if any of those are updated instead of that is disclosed right no problem so the bounty and disclosure I want to throw two things here and open this up to the panel before we hit some other questions I think one Ava mentioned recall and this second one around lateral awareness is a really really important distinction it is not part of the SBOM discussion necessarily right now it is an interesting role for government so is CISA in a position potentially to come out and say we understand patterns in vulnerability because we have this view of SBOMs in the federal government can we start to mirror those out or share that information with large corporations with others in the ecosystem I think the recall piece comes back ironically back to the licensing issue because it is a question of control and what the relationship is going to be between developer and the users of those code am I pushing fixes or am I responsible for trying to pull back and have an active ongoing relationship with that product and so if it is something like a piece of hardware maybe that we are okay with the recall with the pull if it is a piece of software I think historically it has mostly been push and it is up to you to figure out how to either mitigate or pull that patch down so I am curious to come back to that but one other note I think just on the SBOM piece Heather Adkins just for informative purposes Heather Adkins works at Google is one of the co-chairs of the Cyber Safety Review Board which is a DHS chartered entity after action reviews on major cybersecurity incidents and she and the other co-chair Rob Silver were briefing out their findings on Log4Shell at Black Hat just a few days ago and one of the very specific things that she called out as a limiting factor in their recommendations was the difficulty of using SBOMs for deep dependency chain analysis and it was something that came up as a question from a couple of people and so I think coming back to where Eric and Harry were talking through there is a bit of a tendentious relationship right now over how responsible do I want to be as a developer for how my code gets used when it's well passed when it's left my hands and I think I agree with Ava with respect when I'm packaging and selling I bear a lot of responsibility but it's a very, to my ears a very reasonable question for someone who solved one problem wrote one tool to say I'm not signing up to support critical infrastructure forever I think that's okay and I think this is one of the things we're trying to manage here the power of government is substantial and the power to crush that kind of relationship is a serious risk here and every time you write a piece of code you're responsible for how everybody else uses it in perpetuity and so I want you to use MFA I don't want you to feel like you're obligated to protect that thing forever and the SBOM starts to reveal some of these relationships in a way that's more detailed and potentially more troubling than we've had before I think almost everybody agrees with something I said so I'll shut up and let people chat Is there any effort of the government to give the carrot you know I mean you could have with a billion dollars which is whatever the amount is right for all the critical programs in that SBOM as we have across the government we'll have thousands whatever you know you fix anything in one of these thousands of open source free open source projects we'll give you XYZ Just so I'm understanding your question so is basically what you're proposing kind of funneling out money to just develop developers of open source projects Anybody can find the Yeah so I agree and I think there's oh so are you getting a kind of like a bounty model for yeah so I think and there's some I think and there's been things like internet bug bounty trying this out for open source projects I think there is we do have to consider kind of the both kind of the finder of whoever finds the vulnerability but then it's also someone's responsibility to go and patch that vulnerability and if we had say a bounty like model that was only for discovering vulnerabilities I think that that could lead to some areas where we are so focused on kind of identifying vulnerabilities and getting those out that we kind of lose the longer term view of how can we kind of drive down the root cause of these in the first place so I think there can be helpful ways to do it but we need to do so in a way that kind of doesn't lose sight of the end goal of having vulnerabilities be less common in the first place rather than just kind of playing whack-a-mole and kind of knocking out vulnerabilities continuously I'm reminded of the Dilbert cartoon where hey you know I'm announcing a bug bounty and Dilbert goes off and says I'm going to code myself up a new minivan right I think there is maybe the word moral hazard is overstating the case but I think some degree of danger of trying to I mean certainly on the software that the government depends upon if it knows that there's weaknesses through some objectively measurable differentiation between this kind of project and that kind of project channeling money to try to make improvements and some underlying piece or whatever makes a lot of sense I'm generally not a fan of the idea of saying here's a piece I depend upon let's just write a check and walk away it's if there are there's so many different motives for why people write open source code but making money on the side like a gig economy like a Lyft driver is generally not one of them generally speaking people are working on open source code because they're solving bigger problems or they see a bug in something and they want to improve it there's these extrinsic these intrinsic motivators that are generally separate from I want to earn another 100 bucks this week right and we have found actually when there have been problematic projects just writing a check to the maintainers tends not to get those problems fixed but coming in and saying there's some procedures there's some policies there's some specific things we want to do there's a person that we want to bring on to that project is generally a much better way to try to up level a project than going oh you seem under resourced here's 500 dollars or something like that right so I think direct funding of open source projects can be made to work but I think it needs to be attached to something concrete something measurable as an end result of it maybe I want to clear you have two halves one the person is developing it so you got like you know 50 developers and 8 testers of these open source projects they might not have 8 testers right I mean that's the weak point right it's the testing you also might have one person just kind of doing it in their spare time so I think there's a real risk with bug bounties for open source vulnerabilities there's several they all sort of fall under it's unsustainable and if it were scaled up generalized we could easily overwhelm open source projects and contributors with trivial noise that you know people saying this thing's bad this thing's bad and you've got a hobbyist who's like I don't care it doesn't affect me who's doing the work to fix it to maintain it it doesn't lead to sustainable processes to secure these products over time I'll just be the one to say it too that in that general unconstrained setting with hobbyists doing that there's becomes a significant incentive for them to introduce vulnerabilities in the first place well there's another aspect of this and another weak point here that we talk about much less because we have less control over it and that's once a patch is written and pushed out to the community people actually applying the patch and updating updating their assets appropriately we talk about it a lot less because I think in some aspects we have a little bit it's a lot easier for us to focus energy and resources to developing a new patch to fix a critical vulnerability it's a lot harder for us to create incentives around sad to say this basic cyber hygiene of keeping your servers patched there's a real risk to especially system integrators downstream from those once that patch or vulnerability is disclosed what's the time until it's actually integrated into the products out in the wild for the larger projects and vendors that work together there are vulnerability disclosure processes to do all that under embargo that's usually restricted to the big integrators and the big projects and having perhaps more help with that coordination that might be an interesting lever I don't think the bug bounty is the lever I think the coordination might be an interesting lever to explore I just wanted to add one thing so one of the 10 streams in the OpenSSF mobilization plan is focused on funding third party audits on the goal ultimately is the top 200 most critical open source projects now a third party audit on an average sized open source project and they're all over the map but we ballparked $50,000 to $100,000 but it's not enough just to show up to an open source project and say here's where your pants are down here's a problem you have to also show up with resources at least on offer to help with the fix if not perhaps the remediation itself so we actually budgeted $200,000 per fix do that for 200 projects and that's for $40 million but that $40 million will pay out in value creation I think well beyond that each year so that's where in each one of these streams we come up with anything that created an obligation or created a thing that we would require developers to do we felt it was important to also resource that activity to fund it so that we weren't seen as showing up to open source projects with a bunch of new demands or a bunch of noise or a bunch of things that would embarrass them it's about helping uplift all of that I do want to just real quick pop back to a comment I heard over here about SBOMs being kind of unwieldy large one study I saw on building an SBOM at full depth for the Linux kernel came out around 700 megabytes of text right pick a large product build the full dependency graph to infinite depth it's going to be gigantic and a real burden to do analysis on and parse it even and keep it up to date what I'm proposing to folks to do instead of all of that metadata that is in an SBOM that doesn't need to be there for dependency resolution is to use a compact binary format one of the ones we're prototyping it comes out for the comparable size is 1000 times less right 700 kilobytes for the same product I don't know if it was first this might be an over simplification but my general question is we know I think it's on now thank you so we know that we're here because whatever model we have is not working but in terms of newbies to open source development and our ideal of where this goes in the future we used to have schoolhouse rock for bills so I'm just a bill on Capitol Hill so if we were to over simplify what is a very complex issue and have an ideal version of what we want to happen in the future not what happens to open source coders but when you have a software library or a piece of software code what is the life cycle that we wish would happen is it that it starts in a hobbyists basement or I don't code in the basement I code in my living room there's lots of light does it start there do they then join Apache or open source software foundation or wherever it goes how do the vulnerabilities then come in what does that look like the ideal version that we're headed towards and are we all unified on what that ideal vision of the future looks like great question I think there's a bifurcation we have today that I don't think is bad between people who are writing open source software to scratch their own itch a hobbyist or just someone who's like I'm going to tinker with this thing for fun share it with friends versus folks who are paid professionally to work on a piece of software where either the entirety or some component of it is also open source those are often but not always part of the foundation and I think a large foundation like Apache Eclipse CNCF open infra etc they do serve a really valuable function to enable collaboration common IP pooling for companies to work together or to establish norms and standards of processes right Apache and Eclipse are much heavier on the process norms for how to build software together as hobbyists more like CNCF and Linux foundation more so for corporations that's not entirely true I'm totally generalizing here but I do think there's a natural bifurcation there I don't think that's going to change that I think a big chunk of this is the traceability when someone is choosing to integrate an open source component into a service a product a device whether it's government or industry the traceability back that chain to not who wrote it but each piece along the way and how it was modified and what environment it was compiled in and was that environment compromised that traceability is missing today and if I have my deal one thing to change with a magic wand it'd be that is the traceability of where the code came from how it was modified how it was built all the way down to that endpoint when it's integrated into a product or a system I think that enables so much more of what we want to do so on that note I'd like to add something maybe going back to this SBOM software bill of material there seems to be a big problem of well you know well liability and trust here are a big issue right because we talked about it a little bit I mean you guys talked about it a little bit in the context of vulnerabilities right how do we track things back whose responsibility is it and so on but I find it difficult to have this discussion without really talking about the SBOM itself like who generates that who's responsible for having a a trustable SBOM like what about an SBOM that might be malicious or not exactly reflect what's going on and how can we scale this up I think I heard somebody mention audits as a possible way to enforce that but it doesn't sound very scalable so it might be appropriate for like very critical pieces of the software but how do we make it more of a you know how do we generalize this to the large scale without involving too much manual effort and while maintaining trust and you know and having a liability model that integrates everything together sounds like a very difficult question but also something we need to move forward I think we're still a small digit number of years away from the tooling the default build tooling to be for developers to generate these SBOMs you know the ideal is that it's a natural byproduct of the build process or it's a single flag that you add or something like that but the second question is how do you verify them I think but your question is really how can you trust them right well for open source software it's going to be reproducible right like if I build from the same sources I should be able to arrive at the same SBOM and that might allow us to be able to go well if it's not correct in the package that the organization around this package or the developers around this package have released what's the motivation are they was it a bug in which case I'll file a bug about it was it some attempted fraud or some attempted like cutting corners or something like that well then maybe what we need is a leaderboard to try to like you know name and shame the bad ones or something like that but I don't know I hope I hope we won't have to go that far I hope that the automation built into the tools will make it so that you know there's not it's not for reasons of laziness that people create incomplete SBOMs or it might be because they don't want to include 700 megabytes of text or something I don't I hope it's never that burden but I mean what's the scenario you're worried about I guess I guess so the you know the definition of what an SBOM is may also matter here because there's what we currently consider as the you know sort of like dependency graph of what is you know piece of software contains which we can automate with you know tooling and compilers and so on but what is the future of the SBOM should it also include things that are a bit more how can I say this that maybe give you a little more information about what your software your piece of software should be doing like maybe something like you know a manifest of the functional aspects of your code like you know does this code I don't know talk to the network does this code handle this kind of data does this code you know do this or that proposals and this is some of what's been floating on the security specific work about using SBOMs to trace capabilities right should this component be expected to talk to the network no and if it does then it's probably been compromised right there's I think some non-trivial risk of SBOMs being overloaded for all sorts of other use cases that are probably without bounds right as a generalized metadata carrier you know and I fear that that probably would slow us down in fact I think some of the slowness in the standardization work around SPDX has been because SPDX started as a licensing conformance tool right as a way to know that everything in my package is licensed under compatible open source licenses and what the end result of that should be and it's from authorized sources and that kind of thing and we are now adding a bunch of security use cases to it for inventory should it also add capability capabilities as a use case I would hesitate to have that slow down progress on you know answering the issue where am I running for J right answering that kind of question feels first order important now okay no that's a fair answer well if I could maybe take that up just out of the SBOM conversation specifically you know you're raising the concept of basically capabilities based permission systems around software which is an interesting concept whether it's open source software or not and also like bears I think some similarities to some of the major security innovations in other places where sort of third party code is expected to run so sometimes when we talk about like why we would like agencies to use the commercial cloud more is not just because you know maybe it's cheaper maybe it's faster depending on what they're doing it's also from a security perspective the availability of like what cloud providers have done with permissioning like what IAM from Amazon Web Services did and represents as sort of carved out this space of dynamic permissioning is a sea change I think in being able to create complex systems whose privileges you can manage similarly I think the way maybe a more relatable example for regular people is how their mobile phones work right and the differences between you know what permissions an app has on your phone versus what it has on a conventional sort of laptop those are considered to be foundational and dramatic security improvements and thinking about ways to apply that kind of model to software generally whether it has any intersection with SBOM standards or whatever level of coupling is appropriate like is certainly intriguing and something it's the way we should be thinking about things I know we're coming up on time so we're going to take two more from the audience we'll have one more quick question before we wrap I know sir you've been waiting and ma'am I see your hand down here so I just wanted to actually circle back to what this gentleman had suggested because what I had actually heard from him was not actually necessarily a vulnerability project but more of an intent to find a way to patch known pieces of software that came out of the software bill and materials so critical things but just to tail into that what I found interesting was one of the things you had opened with was that we don't know how to fund open source software but then you mentioned that you know sometimes writing the blank check to the developer isn't a good way to do that and then you've got audits and you know coming back in with additional funding to help resource fixing it but it still sounds like we have kind of an uncertainty in terms of you know is there a way for the government to meaningfully come in identify these are critical software projects that we're going to proactively attempt to help improve the general state of being and how to do so and so I'm just curious in your comments um um yeah go ahead my last one uh just a quick comment on the last one which you're touching on a little bit was um uh capability detection capability labeling of components that might make sense in some spaces but it certainly is not generalizable across everything an s-bomb would apply to think about the Linux kernel as a whole right if you're labeling sub-component capabilities at the level of the kernel's SBOM, those sub-component capabilities are meaningless. So generalizing this doesn't work as well. But for product labeling or component labeling, I think a security model totally makes sense. And we do see this in some areas. You see it on mobile phones and some operating systems. You see it in cloud-native computing. There are whole frameworks for labeling capabilities and then flagging if something is accessing the network that shouldn't. So that's great, but I think it's very separate from SBOMs. Could you say a little more about why it doesn't scale in the Linux kernel? SBOMs roll up. You start from the small components. You combine many small components to make bigger and bigger things. You get a tree form. And the capability of sub-component components may modulate as it gets combined with other components. So it doesn't make sense to say all components and pieces of software must be labeled with capabilities as part of the SBOM, because when you roll up the SBOM, it's irrelevant now. You need to relabel that at each step and purge that data from the... It doesn't work in the way SBOMs are used by industry. So maybe the... Okay. Maybe the solution here is not maybe exactly the SBOM. Maybe something's the other way around. I see a difference, but... Yes, there... It is valid, and there are things that do that today. They're just not SBOMs. Okay, I understand. Ma'am, over here. Yeah, thank you. I'm going to take us in a little bit of a different direction, maybe. So you said, you know, one of the three things was funding, and I heard various discussions about how to fund people to build better stuff. But I've also heard some pretty aspirational ideas here about automating development of SBOMs, automating finding the... I mean, there's a lot of stuff, and at very, very, very large scale. And so then I'll look particularly to people that are talking to NSF or DARPA or the different agencies, right? It seems to me that there needs to be significant investment in the research so that five years from now, we're not having the same conversation over and over, because almost everything talked about for the last hour has been pretty manual, pretty much, you know, people just doing. And, yeah, we've got billions of people with open source, but instead, we need to build more sophisticated approaches to managing this. And that's going to take major investment of research funding. And I would love to just add one call to build on that, and sustainable work, right? To connect your two points, could somewhere like the NSF or anywhere else fund the sustaining work that doesn't get funded through bug bounty programs, through just a one-time code audit? Where can we locate, is it in industry, academia, or other where, the long-term work to maintain these projects? Well, one of the, we've been doing some work, and it's been great working with Eric on the open source software security initiative that we've been co-leading from the White House. But one of the things that we've been very focused on as we move forward with this effort is the overall software ecosystem. When we talk about the resources that the government needs, we talk about the resources that the government needs. We talk about the resources that the government brings to bear, and how we play a role. It goes beyond just grant funding that's provided, which typically is for, you know, specific R&D. There are a couple grant programs out there that are very unique because they're looking at the ecosystem, they're looking at governance structures. There's one out of the NSF, the POSE grant program, that exclusively focuses on the governance of open source software. And we think that this is a really good first step. More resources are certainly needed in this area, but it's a great start, and we're very excited to be working with our partners. We're excited to be working with our NSF colleagues on that. Yeah, A-plus, totally agree. So, really appreciate, I think, everybody coming in, and especially at DEF CON, this is one of these conversations, it's very hard to get this kind of diversity of folks in the room. I will say, so this session is one of a couple that's occurring as a result of a partnership between the Atlanta Council, a think tank in D.C., and the DEF CON Policy Village. We, together with a number of the folks that you see here, are working to build out an open source policy network of folks like yourselves who are interested in this, doing security research, contributing back to open source projects, and want to be part of addressing some of these questions about open source infrastructure and governance over the next year and a half and beyond. If that sounds interesting to you and you'd like to be part of the next couple of these conversations, please find us on the web or come up and talk to me. I'm happy to give you email. We'd love to get everybody in here engaged, I think, as we go. This is obviously not a small set of problems. I'd also just ask very quickly if you could join me in thanking the panelists who are here for making it out to the desert surviving so far.