1 00:00:00,570 --> 00:00:03,000 - So, this is Zachary Minneker. 2 00:00:03,000 --> 00:00:08,000 He is going to talk about how to get MUMPS 30 years later 3 00:00:09,840 --> 00:00:13,773 with some more stuff added on that you can read. 4 00:00:15,120 --> 00:00:16,534 - Great, good to go. 5 00:00:16,534 --> 00:00:18,870 (people cheering and clapping) 6 00:00:18,870 --> 00:00:23,490 All right, let me just set my timer here real quick 7 00:00:23,490 --> 00:00:24,840 and then we will get going. 8 00:00:25,920 --> 00:00:27,211 Okay. - What's up? 9 00:00:27,211 --> 00:00:28,850 - What's up? - Great job. 10 00:00:30,099 --> 00:00:33,478 - I think I already do. I recognize you from the office. 11 00:00:33,478 --> 00:00:34,311 - My bad. 12 00:00:35,250 --> 00:00:36,690 - All right, cool. 13 00:00:36,690 --> 00:00:37,890 Thanks for coming to everybody. 14 00:00:37,890 --> 00:00:40,470 This is how to get MUMPS 30 years later. 15 00:00:40,470 --> 00:00:42,093 My name's Zach Minneker. 16 00:00:43,740 --> 00:00:46,470 So to just start, I'm gonna just give you a little bit 17 00:00:46,470 --> 00:00:48,630 of an agenda of what we're gonna talk about here. 18 00:00:48,630 --> 00:00:51,090 First, I'm just gonna talk about the history 19 00:00:51,090 --> 00:00:53,220 of the thing we're talking about here. 20 00:00:53,220 --> 00:00:55,590 I'm gonna talk about how to break MUMPS, 21 00:00:55,590 --> 00:00:57,960 how to break an EMR that's called VistA, 22 00:00:57,960 --> 00:01:00,660 and then I'm gonna talk about the future 23 00:01:00,660 --> 00:01:02,880 and the past of these things. 24 00:01:02,880 --> 00:01:04,500 Before I start, I just wanna say, 25 00:01:04,500 --> 00:01:06,123 just get some definitions going. 26 00:01:07,500 --> 00:01:09,660 First, I'm gonna say EMR throughout 27 00:01:09,660 --> 00:01:12,090 by which I mean electronic medical records 28 00:01:12,090 --> 00:01:13,890 which is just software that is used 29 00:01:13,890 --> 00:01:16,530 to maintain patient records, stuff like that. 30 00:01:16,530 --> 00:01:18,360 I'm also gonna say VA throughout 31 00:01:18,360 --> 00:01:21,390 by which I mean the Department of Veterans Affairs 32 00:01:21,390 --> 00:01:25,800 which handles concerns for veterans in the United States, 33 00:01:25,800 --> 00:01:28,650 chiefly for what we care about their medical care 34 00:01:28,650 --> 00:01:30,900 after they're out of the military, 35 00:01:30,900 --> 00:01:33,510 and then I'm also gonna talk about FOIA 36 00:01:33,510 --> 00:01:35,520 which is the Freedom of Information Act, 37 00:01:35,520 --> 00:01:38,400 which basically you can submit a form to the government 38 00:01:38,400 --> 00:01:41,790 that says, "I want information about this specific thing," 39 00:01:41,790 --> 00:01:44,460 and as long as it's not long as they don't have a reason 40 00:01:44,460 --> 00:01:47,040 to say no, they give it to you basically. 41 00:01:47,040 --> 00:01:48,900 Yeah, cool, so, who am I? 42 00:01:48,900 --> 00:01:50,430 Like I said, my name's Zachary Minneker. 43 00:01:50,430 --> 00:01:52,830 I work for a company called Security Innovation. 44 00:01:53,880 --> 00:01:55,803 I don't work for the government. 45 00:01:57,360 --> 00:01:58,860 I break stuff for a living. 46 00:01:58,860 --> 00:02:01,770 I also used to work in healthcare, kind of sorta, 47 00:02:01,770 --> 00:02:04,560 and now I just will work on breaking healthcare stuff 48 00:02:04,560 --> 00:02:06,900 for fun, and I care a lot about the history 49 00:02:06,900 --> 00:02:09,060 of a lot of the software that we use. 50 00:02:09,060 --> 00:02:12,573 I'm a real big fan of the PDP 11, for example. 51 00:02:14,190 --> 00:02:17,070 Also, I'm speaking on behalf of myself 52 00:02:17,070 --> 00:02:18,990 and not of my employer, 53 00:02:18,990 --> 00:02:20,400 and also I made these slides 54 00:02:20,400 --> 00:02:23,160 in an antsy tech editor called Moebius, 55 00:02:23,160 --> 00:02:25,890 and they should be available on the media server 56 00:02:25,890 --> 00:02:28,620 under the name, the title of the talk.info. 57 00:02:28,620 --> 00:02:29,550 They're not up there right now, 58 00:02:29,550 --> 00:02:31,050 but I will get those up there. 59 00:02:31,920 --> 00:02:34,833 So anyway, so what's this talk about? 60 00:02:36,140 --> 00:02:38,310 In healthcare, there's a language called MUMPS 61 00:02:38,310 --> 00:02:41,580 that effectively undercards 62 00:02:41,580 --> 00:02:46,230 a ton of modern healthcare infrastructure. 63 00:02:46,230 --> 00:02:48,900 You've probably seen this XKCD, 64 00:02:48,900 --> 00:02:50,670 you've got on the left there, 65 00:02:50,670 --> 00:02:53,460 these towers that say all of digital infrastructure, 66 00:02:53,460 --> 00:02:54,630 and then there's a little block 67 00:02:54,630 --> 00:02:57,090 that says, "Random project maintained 68 00:02:57,090 --> 00:03:00,780 by somebody in Nebraska since 2003." 69 00:03:00,780 --> 00:03:03,360 This talk is sort of about hitting that block with a hammer. 70 00:03:03,360 --> 00:03:06,060 It's basically what we're trying to do here. 71 00:03:06,060 --> 00:03:09,000 So, like I said, we're gonna talk a little bit 72 00:03:09,000 --> 00:03:12,270 about the history of the things involved here. 73 00:03:12,270 --> 00:03:15,543 So, where the hell did MUMPS come from? 74 00:03:16,470 --> 00:03:19,230 In 1966, there was a group of engineers 75 00:03:19,230 --> 00:03:21,610 that were working for a Dr. Octo Barnett 76 00:03:22,860 --> 00:03:25,740 at Mass general hospital in Boston 77 00:03:25,740 --> 00:03:27,693 including Neil Papparlardo 78 00:03:27,693 --> 00:03:29,670 that they started working on this new language 79 00:03:29,670 --> 00:03:32,130 that they called the Massachusetts General Hospital 80 00:03:32,130 --> 00:03:35,070 Utility Multi-Programming System 81 00:03:35,070 --> 00:03:37,293 which of course shortens to MUMPS. 82 00:03:38,700 --> 00:03:41,430 There is some movement now to call it the M language, 83 00:03:41,430 --> 00:03:42,930 I'm gonna refer to it that MUMPS throughout, 84 00:03:42,930 --> 00:03:44,880 because I think that's a more fun name. 85 00:03:45,990 --> 00:03:49,710 So, MUMPS was specifically imagined 86 00:03:49,710 --> 00:03:52,710 as a language for healthcare environments. 87 00:03:52,710 --> 00:03:55,680 It was originally written allegedly for PDP-9s 88 00:03:55,680 --> 00:03:58,140 and then Digital Equipment Corporation grabbed it, 89 00:03:58,140 --> 00:04:02,613 and they turned it into a standalone OS for PDP-11s 90 00:04:02,613 --> 00:04:04,233 that they called MUMPS-11. 91 00:04:06,081 --> 00:04:06,914 Some records say 92 00:04:06,914 --> 00:04:09,843 that originally it was deployed on PDP-7s. 93 00:04:10,950 --> 00:04:14,070 There's also it has a lot of influence 94 00:04:14,070 --> 00:04:19,070 from a language called BBN TELECOMP which I say mostly, 95 00:04:19,140 --> 00:04:23,910 just to say that this is a pre-Unix language, 96 00:04:23,910 --> 00:04:26,433 this is a pre see language. 97 00:04:27,511 --> 00:04:31,230 This is before the standard concept 98 00:04:31,230 --> 00:04:34,710 of how programming works as you and I know it, right? 99 00:04:34,710 --> 00:04:36,930 And so originally it wasn't like a language, 100 00:04:36,930 --> 00:04:37,950 it was an environment, 101 00:04:37,950 --> 00:04:40,620 it was everything you needed to make software 102 00:04:40,620 --> 00:04:42,753 for healthcare uses. 103 00:04:43,710 --> 00:04:47,340 So, as just an example of one of the little oddities here, 104 00:04:47,340 --> 00:04:49,590 there wasn't an Unix epoch, right? 105 00:04:49,590 --> 00:04:52,350 There was no number of minutes since, 106 00:04:52,350 --> 00:04:54,690 or number of seconds since 1970. 107 00:04:54,690 --> 00:04:57,540 So instead they have this variable that's called HOROLOG 108 00:04:57,540 --> 00:05:00,960 that has two numbers separated by a comma, 109 00:05:00,960 --> 00:05:03,120 and the first number is the number of days 110 00:05:03,120 --> 00:05:06,420 since January 1st, 1841. 111 00:05:06,420 --> 00:05:08,250 And the whole reason for that is 112 00:05:08,250 --> 00:05:12,870 because their assumption was that the oldest living veteran 113 00:05:12,870 --> 00:05:14,520 that they would have to give healthcare 114 00:05:14,520 --> 00:05:18,060 to fought in the civil war, 115 00:05:18,060 --> 00:05:21,390 and that was their measure that they went for so yeah, 116 00:05:21,390 --> 00:05:25,083 also the maximum data can support is December 31st, 9999, 117 00:05:26,458 --> 00:05:27,360 so Y2K compliant 118 00:05:27,360 --> 00:05:29,673 which good job really thinking ahead there. 119 00:05:31,200 --> 00:05:33,480 So originally, MUMPS wasn't super well-standardized. 120 00:05:33,480 --> 00:05:35,190 It was this idea that they were playing around with, 121 00:05:35,190 --> 00:05:37,950 it showed up in PDPs, it showed up all over the place, 122 00:05:37,950 --> 00:05:41,670 but eventually the VA hired two engineers and said, 123 00:05:41,670 --> 00:05:42,990 Hey, there's this new thing. 124 00:05:42,990 --> 00:05:45,783 Let's look into doing something with this, 125 00:05:46,680 --> 00:05:48,960 and eventually they started working on this suite 126 00:05:48,960 --> 00:05:51,870 of utilities over at the VA 127 00:05:51,870 --> 00:05:56,870 that eventually sort of coalesced into this single total EMR 128 00:05:57,480 --> 00:06:00,570 that they called VistA, capital V, capital A, 129 00:06:00,570 --> 00:06:01,413 real cute name, 130 00:06:02,430 --> 00:06:04,440 and the people who were working on this was this group 131 00:06:04,440 --> 00:06:06,300 that was called The Hardhats, 132 00:06:06,300 --> 00:06:09,750 and just as an example of the people we're talking about, 133 00:06:09,750 --> 00:06:14,373 they're still around, and this is their website as of 2021. 134 00:06:15,660 --> 00:06:17,253 Absolutely incredible. 135 00:06:18,180 --> 00:06:21,093 These folks are really going for it. 136 00:06:23,340 --> 00:06:27,780 So, VistA, this EMR that they were working on VistA, 137 00:06:27,780 --> 00:06:28,950 it sort of grew throughout the years. 138 00:06:28,950 --> 00:06:30,990 It got bigger and bigger and bigger, 139 00:06:30,990 --> 00:06:33,510 and it became really well-loved and well-respected. 140 00:06:33,510 --> 00:06:34,593 On the one hand, 141 00:06:35,670 --> 00:06:39,240 this is you have people that have a need for software 142 00:06:39,240 --> 00:06:40,800 and they're making it themselves. 143 00:06:40,800 --> 00:06:44,400 This is four doctors by doctors effectively, right? 144 00:06:44,400 --> 00:06:45,360 On the other hand, 145 00:06:45,360 --> 00:06:50,360 it's effectively shadow IT as a development strategy. 146 00:06:50,580 --> 00:06:54,750 So there was a lot of working that needed 147 00:06:54,750 --> 00:06:58,341 to be done to get stuff, to fit together, basically. 148 00:06:58,341 --> 00:07:03,341 MUMPS the language was and is extremely fast. 149 00:07:05,700 --> 00:07:10,380 It's no SQL partially because it beat SQL to market. 150 00:07:10,380 --> 00:07:14,130 Also, partially because it fits every definition of no-SQL, 151 00:07:14,130 --> 00:07:16,920 and it's like, it's perfect for anytime you've got data 152 00:07:16,920 --> 00:07:20,523 that needs a lot of rights, banks, sciences, hospitals. 153 00:07:23,550 --> 00:07:26,280 Nowadays, VistA is still widely deployed 154 00:07:26,280 --> 00:07:29,070 at basically every VA hospital. 155 00:07:29,070 --> 00:07:30,270 Doctors really love it. 156 00:07:30,270 --> 00:07:32,490 People who have interacted with it, love it. 157 00:07:32,490 --> 00:07:35,190 Hospitals outside of the VA system, 158 00:07:35,190 --> 00:07:37,443 use VistA for certain things. 159 00:07:38,400 --> 00:07:39,780 There's inside of the VA, 160 00:07:39,780 --> 00:07:44,220 there's this effort to modernize their EMR 161 00:07:44,220 --> 00:07:46,170 which means that they're trying to move, 162 00:07:46,170 --> 00:07:48,270 they're trying to get rid of VistA, 163 00:07:48,270 --> 00:07:51,240 but it's still deployed all over the place, 164 00:07:51,240 --> 00:07:56,240 and MUMPS is still widely used even outside of healthcare. 165 00:07:56,520 --> 00:07:59,940 There's some of the biggest EMRs in the world use it. 166 00:07:59,940 --> 00:08:02,520 Core banking systems use it. 167 00:08:02,520 --> 00:08:05,910 The European space agency has deployed it 168 00:08:05,910 --> 00:08:08,670 within the decade I believe, or within the last decade 169 00:08:08,670 --> 00:08:12,660 which people are still finding uses for it. 170 00:08:12,660 --> 00:08:14,820 If you wanna join me on this adventure, 171 00:08:14,820 --> 00:08:19,230 you can install MUMPS by running in a Ubuntu or Debian, 172 00:08:19,230 --> 00:08:22,830 sudo apt install -y fis-gtm 173 00:08:22,830 --> 00:08:26,070 which will install FIS' GT.M. 174 00:08:26,070 --> 00:08:27,180 You need that -y 175 00:08:27,180 --> 00:08:31,650 because if you even think about installing MUMPS, you must, 176 00:08:31,650 --> 00:08:35,760 so you have to have apt install it for you 177 00:08:35,760 --> 00:08:37,500 and VistA, you can actually just get, 178 00:08:37,500 --> 00:08:42,150 at some point, it got FOIAD is my understanding. 179 00:08:42,150 --> 00:08:47,150 This appears to have started somewhere in September of 2004, 180 00:08:47,640 --> 00:08:50,700 but just somebody at the VA just uploads it 181 00:08:50,700 --> 00:08:52,770 to an FTP server every month, 182 00:08:52,770 --> 00:08:55,413 and whatever the most modern version of VistA is, 183 00:08:55,413 --> 00:08:57,390 it's just every year come 184 00:08:57,390 --> 00:08:59,573 or every month you can just grab a new version. 185 00:09:00,660 --> 00:09:03,570 So we're gonna talk about a couple of different, 186 00:09:03,570 --> 00:09:05,130 we're gonna talk about MUMPS a little bit. 187 00:09:05,130 --> 00:09:07,230 I'm not gonna talk a whole lot about how VistA works, 188 00:09:07,230 --> 00:09:08,493 but I just wanted to ask, 189 00:09:09,390 --> 00:09:12,780 has anyone in here written any MUMPS? 190 00:09:12,780 --> 00:09:14,070 I know there's at least one person. 191 00:09:14,070 --> 00:09:16,683 Okay, so we've got two people, three people. 192 00:09:17,550 --> 00:09:22,080 All right, oh, wait, okay so we've got less than 10, 193 00:09:22,080 --> 00:09:23,670 makes sense, all right. 194 00:09:23,670 --> 00:09:25,440 Look at MUMPS as a cool language. 195 00:09:25,440 --> 00:09:28,530 I'm gonna demonstrate to you that it's a cool language. 196 00:09:28,530 --> 00:09:32,283 First off in MUMPS, three plus six times two is 18. 197 00:09:33,660 --> 00:09:36,150 So if we think about our order of operations here, 198 00:09:36,150 --> 00:09:39,180 six times two is 12 plus three is 15, 199 00:09:39,180 --> 00:09:40,920 so that doesn't make a whole lot of sense. 200 00:09:40,920 --> 00:09:42,450 The reason why this is happening is 201 00:09:42,450 --> 00:09:46,323 because all math is strictly evaluated from left to right? 202 00:09:48,570 --> 00:09:50,133 It gets weirder from here. 203 00:09:52,071 --> 00:09:53,010 We're gonna keep going in general. 204 00:09:53,010 --> 00:09:55,260 I find it to be a pretty readable language, 205 00:09:55,260 --> 00:09:59,883 but it's from a time where size for computers was 206 00:09:59,883 --> 00:10:01,830 at a really high premium. 207 00:10:01,830 --> 00:10:03,780 So a lot of code isn't commented. 208 00:10:03,780 --> 00:10:05,940 It just isn't 'cause they didn't wanna have to store it. 209 00:10:05,940 --> 00:10:07,890 There's on some implementations. 210 00:10:07,890 --> 00:10:10,050 There's a performance cost to actually having comments 211 00:10:10,050 --> 00:10:11,070 in the code, 212 00:10:11,070 --> 00:10:13,890 and then also a lot of the keywords in the language 213 00:10:13,890 --> 00:10:17,490 can be shortened down to single characters, 214 00:10:17,490 --> 00:10:19,800 which gets kind of wild, 215 00:10:19,800 --> 00:10:23,580 so here's just an example of some MUMPS. 216 00:10:23,580 --> 00:10:26,583 I don't believe this runs, but it looks okay, 217 00:10:28,320 --> 00:10:30,330 and if you notice, I just wanna point out at the bottom, 218 00:10:30,330 --> 00:10:31,710 the third to last line there, 219 00:10:31,710 --> 00:10:36,643 you have . Else Do ; for the comment. 220 00:10:40,800 --> 00:10:43,560 Those two spaces after the Else are important 221 00:10:43,560 --> 00:10:46,443 because white space is significant in the language. 222 00:10:47,940 --> 00:10:50,850 You can do some stuff with it, but it's significant, 223 00:10:50,850 --> 00:10:51,683 but like I said, 224 00:10:51,683 --> 00:10:54,360 you can shorten a lot of keywords down to single characters, 225 00:10:54,360 --> 00:10:57,660 so we can go from that to that, 226 00:11:00,330 --> 00:11:03,720 extremely, honestly, still kind of readable 227 00:11:03,720 --> 00:11:07,410 as long as everything stays just sort of short per line. 228 00:11:07,410 --> 00:11:10,263 Everything's cool, but we can go smaller. 229 00:11:11,695 --> 00:11:14,760 There is no reason to stop at only, 230 00:11:14,760 --> 00:11:17,910 if you think about how code is written, 231 00:11:17,910 --> 00:11:20,463 usually it flows from top to bottom. 232 00:11:22,680 --> 00:11:25,440 This is an invention that does not need to exist. 233 00:11:25,440 --> 00:11:28,290 What if instead of writing your code vertically, 234 00:11:28,290 --> 00:11:30,093 you wrote it horizontally. 235 00:11:31,290 --> 00:11:34,863 So, that same code can be turned into that. 236 00:11:36,801 --> 00:11:40,350 This is basically code golf. This is enterprise code golf. 237 00:11:40,350 --> 00:11:43,410 And in fact, if you go on some code golf forums, 238 00:11:43,410 --> 00:11:46,290 people are using this language to do code golf. 239 00:11:46,290 --> 00:11:48,450 It's a great language for it. 240 00:11:48,450 --> 00:11:51,300 And I'm not like I am not cherry picking here. 241 00:11:51,300 --> 00:11:53,980 This is actual source from VistA 242 00:11:55,170 --> 00:11:56,700 and this is how readable it is. 243 00:11:56,700 --> 00:11:58,140 If you look at that first line, 244 00:11:58,140 --> 00:12:00,990 you have N space and then a couple of variable names 245 00:12:00,990 --> 00:12:04,530 and then the line below it, you have set D sub equal zero. 246 00:12:04,530 --> 00:12:07,563 That's just setting a variable to a certain value. 247 00:12:08,820 --> 00:12:13,623 4 (DSUB=) and then a bunch of, 248 00:12:16,260 --> 00:12:19,200 that is how this code gets written is entire four loops 249 00:12:19,200 --> 00:12:20,133 on one line. 250 00:12:21,120 --> 00:12:23,490 Here's another example and for some reason 251 00:12:23,490 --> 00:12:25,380 in the VistA source code, 252 00:12:25,380 --> 00:12:28,260 there doesn't seem to be a strong coding style 253 00:12:28,260 --> 00:12:29,093 that was enforced. 254 00:12:29,093 --> 00:12:31,500 There's no linters for this language. 255 00:12:31,500 --> 00:12:36,253 So in this case you have if for written literally as IF, 256 00:12:37,110 --> 00:12:38,370 they're literally using if, 257 00:12:38,370 --> 00:12:41,193 but then new and set are just single characters, right? 258 00:12:42,390 --> 00:12:46,410 It's like this language gets rough to look at, 259 00:12:46,410 --> 00:12:48,600 but on modern implementations, 260 00:12:48,600 --> 00:12:50,853 now that we've talked about writing the code, 261 00:12:51,810 --> 00:12:53,400 we need to run the code. 262 00:12:53,400 --> 00:12:57,300 Generally, MUMPS is described as both an interpreted 263 00:12:57,300 --> 00:13:00,270 and also a compiled language, 264 00:13:00,270 --> 00:13:04,470 so, on the one hand, you can write your code 265 00:13:04,470 --> 00:13:06,870 and then tell MUMPS, "Hey, run this code," 266 00:13:06,870 --> 00:13:09,090 and what it does is compile it, 267 00:13:09,090 --> 00:13:10,950 store that as a shared object 268 00:13:10,950 --> 00:13:14,730 or at least GT.M and MUMPS due, I should be clear. 269 00:13:14,730 --> 00:13:17,670 It compiles it, stores that as a shared object 270 00:13:17,670 --> 00:13:20,640 and then it loads that shared object into its memory space 271 00:13:20,640 --> 00:13:23,003 and then jumps into that code that you just wrote. 272 00:13:23,880 --> 00:13:27,510 And so what that means is you can deploy VistA code 273 00:13:27,510 --> 00:13:28,590 as just the source code, 274 00:13:28,590 --> 00:13:31,470 which is kind of small and then compile on site, 275 00:13:31,470 --> 00:13:35,043 which is a pretty useful feature to have, 276 00:13:36,510 --> 00:13:41,510 and it's like this is how it works in the modern era. 277 00:13:41,880 --> 00:13:43,770 So like I said earlier, 278 00:13:43,770 --> 00:13:46,320 just to be real clear about what we're talking about. 279 00:13:46,320 --> 00:13:48,240 That's the language MUMPS. 280 00:13:48,240 --> 00:13:51,660 VistA is written entirely in MUMPS. 281 00:13:51,660 --> 00:13:54,450 The way that I got it for this research is 282 00:13:54,450 --> 00:13:57,330 just by downloading it from that FTP server I mentioned, 283 00:13:57,330 --> 00:14:00,720 and a lot of this is based on a certain flavor 284 00:14:00,720 --> 00:14:02,370 of the FOIA version of MUMPS. 285 00:14:02,370 --> 00:14:06,060 There's some modifications that get done to VistA sort of 286 00:14:06,060 --> 00:14:09,330 after the fact that get packaged 287 00:14:09,330 --> 00:14:12,303 into different distributions for different uses and stuff. 288 00:14:15,055 --> 00:14:17,417 If you are using a version of VistA 289 00:14:17,417 --> 00:14:21,270 that is deployed using GT.M for example, 290 00:14:21,270 --> 00:14:24,780 usually you're storing your routines in a folder 291 00:14:24,780 --> 00:14:27,150 that's traditionally just called r/, 292 00:14:27,150 --> 00:14:30,270 so, if you follow me on this and you use VistA 293 00:14:30,270 --> 00:14:33,690 that is deployed with GT.M using Docker or something, 294 00:14:33,690 --> 00:14:35,250 look for that routines folder 295 00:14:35,250 --> 00:14:37,650 'cause it's gonna have all of your source code in it. 296 00:14:37,650 --> 00:14:40,320 All right so, that's all of our history, 297 00:14:40,320 --> 00:14:41,313 so then I show up. 298 00:14:42,420 --> 00:14:44,100 So, how did I actually get involved in this? 299 00:14:44,100 --> 00:14:46,260 So at Security Innovation, the company I work at, 300 00:14:46,260 --> 00:14:49,380 we have research time where we look at interesting stuff, 301 00:14:49,380 --> 00:14:51,380 learn how to break new things, whatever, 302 00:14:52,320 --> 00:14:54,810 and I have been using mine to systematically go 303 00:14:54,810 --> 00:14:56,880 through a bunch of different healthcare protocols, 304 00:14:56,880 --> 00:14:58,890 look at different EMRs, kind of do whatever, 305 00:14:58,890 --> 00:15:03,600 and I had heard about VistA maybe five or six years ago, 306 00:15:03,600 --> 00:15:06,330 and I didn't realize it was MUMPS. 307 00:15:06,330 --> 00:15:10,530 I didn't realize how foundational MUMPS was to all of, 308 00:15:10,530 --> 00:15:11,850 a lot of healthcare stuff, 309 00:15:11,850 --> 00:15:14,050 talk more about where it's being used later, 310 00:15:15,120 --> 00:15:16,410 and places I had worked at 311 00:15:16,410 --> 00:15:20,370 in hospitals had always used Java-based EMRs, 312 00:15:20,370 --> 00:15:25,140 so I just never got a whole lot of exposure to MUMPS, 313 00:15:25,140 --> 00:15:28,533 and on top of that, I desperately want to be cool, 314 00:15:29,430 --> 00:15:31,980 and I think hacking weird code is cool, 315 00:15:31,980 --> 00:15:34,380 and I think I've demonstrated that VistA is weird 316 00:15:34,380 --> 00:15:38,883 and MUMPS is weird, so thus, we can play. 317 00:15:40,290 --> 00:15:42,930 So, let me just talk about what a deployment 318 00:15:42,930 --> 00:15:44,493 of VistA sort of looks like. 319 00:15:45,330 --> 00:15:50,040 It's basically this, so you have some hardware, some X86, 320 00:15:50,040 --> 00:15:51,630 probably machine that you're running it on, 321 00:15:51,630 --> 00:15:54,000 that you have an operating system that's running on, right? 322 00:15:54,000 --> 00:15:56,220 On top of that, you're running some sort 323 00:15:56,220 --> 00:15:57,570 of MUMPS implementation. 324 00:15:57,570 --> 00:16:01,350 For my use, this is either GT.M or MUMPS. 325 00:16:01,350 --> 00:16:04,350 There's also a Windows implementation that's called cache 326 00:16:04,350 --> 00:16:05,880 that's pretty common, 327 00:16:05,880 --> 00:16:08,250 and then on top of that is VistA, 328 00:16:08,250 --> 00:16:11,340 so in VistA, when you make a new string or whatever, 329 00:16:11,340 --> 00:16:14,040 you're interacting, you're asking the MUMPS implementation 330 00:16:14,040 --> 00:16:16,890 to give you memory to use as a string, 331 00:16:16,890 --> 00:16:19,640 and it goes to the operating system to get that memory. 332 00:16:20,670 --> 00:16:25,670 So, the way that in a actual hospital VistA gets used is 333 00:16:25,710 --> 00:16:29,310 that you have clients that talk to it using this RPC method 334 00:16:29,310 --> 00:16:31,380 that's called XWB. 335 00:16:31,380 --> 00:16:35,370 A really common client is CPRS 336 00:16:35,370 --> 00:16:36,960 which I'll talk about in a sec here, 337 00:16:36,960 --> 00:16:40,293 but that's our general map of what this thing looks like. 338 00:16:41,250 --> 00:16:46,250 So, I go out and say, I wanna attack this as an attacker, 339 00:16:46,410 --> 00:16:48,030 or I wanna attack this as a client. 340 00:16:48,030 --> 00:16:50,730 I wanna just be able to show up at a VA hospital, 341 00:16:50,730 --> 00:16:52,650 plug into a wall and go, 342 00:16:52,650 --> 00:16:55,950 so I want to start with their client 343 00:16:55,950 --> 00:16:59,790 and then start exploring what I can do here. 344 00:16:59,790 --> 00:17:02,880 So I go and grab the most common VistA client 345 00:17:02,880 --> 00:17:06,270 which is called CPRS. CPRS is really widely available. 346 00:17:06,270 --> 00:17:07,560 I think it's up on GitHub now. 347 00:17:07,560 --> 00:17:10,410 It's written in Delphi, so it's more readable than MUMPS. 348 00:17:11,520 --> 00:17:12,960 So I install CPRS, 349 00:17:12,960 --> 00:17:15,780 I run a version of VistA that's deployed without TLS 350 00:17:15,780 --> 00:17:17,103 which isn't hard. 351 00:17:18,150 --> 00:17:19,383 That'll come back later, 352 00:17:20,220 --> 00:17:21,810 and then I start capturing packets, 353 00:17:21,810 --> 00:17:25,210 and so, I get a lot of RPC traffic that looks like this 354 00:17:26,730 --> 00:17:31,730 and ignoring the normal TCP stuff at the bottom, 355 00:17:32,850 --> 00:17:35,490 we've got all this askie at the bottom 356 00:17:35,490 --> 00:17:39,270 that I just don't really understand. 357 00:17:39,270 --> 00:17:40,710 If you look on the third line, 358 00:17:40,710 --> 00:17:44,520 you can make out that there's one 27.0.01 that's on there, 359 00:17:44,520 --> 00:17:46,530 that makes sense, I'm running the server on local host, 360 00:17:46,530 --> 00:17:48,930 but I have no idea what's going on, 361 00:17:48,930 --> 00:17:51,240 and I can't really turn to the source code 362 00:17:51,240 --> 00:17:55,200 because at this point, this is a year into this research 363 00:17:55,200 --> 00:17:57,270 and I don't know, MUMPS. 364 00:17:57,270 --> 00:18:00,150 So I'm gonna do this in the dumbest way possible 365 00:18:00,150 --> 00:18:02,900 and just start looking for keywords in the source code. 366 00:18:04,080 --> 00:18:06,210 And so when I'm dragging through the source at some point, 367 00:18:06,210 --> 00:18:11,190 I start finding this code and if you look here, 368 00:18:12,090 --> 00:18:14,990 you'll notice there's a line that says, TYPE=(XR="[XWB]"), 369 00:18:20,744 --> 00:18:24,660 that's our code, that is consuming this RPC traffic, 370 00:18:24,660 --> 00:18:26,313 so we've got a way in. 371 00:18:27,750 --> 00:18:30,750 So, like I said, don't wanna learn MUMPS, 372 00:18:30,750 --> 00:18:34,263 and so, I turned to old reliable here, which is BooFuzz. 373 00:18:35,130 --> 00:18:36,990 So BooFuzz, if you haven't used, 374 00:18:36,990 --> 00:18:41,370 it is a Python library for basically making network fuzzes 375 00:18:41,370 --> 00:18:43,980 where you don't really have to, you can just say, 376 00:18:43,980 --> 00:18:45,690 here's what the network traffic looks like, 377 00:18:45,690 --> 00:18:48,450 go fuzz this thing, here's the address. 378 00:18:48,450 --> 00:18:50,370 But to do that, I need to capture a lot of traffic 379 00:18:50,370 --> 00:18:54,240 and then turn that traffic into this BooFuzz script, 380 00:18:54,240 --> 00:18:58,830 and if you use VistA for, with CPRS for, I don't know, 381 00:18:58,830 --> 00:19:03,830 15 minutes or so, you'll generate hundreds of RPC calls, 382 00:19:03,930 --> 00:19:08,760 so, I start writing these notating these by hand 383 00:19:08,760 --> 00:19:11,280 into a BooFuzz script and they get through about 20 before. 384 00:19:11,280 --> 00:19:13,380 I'm like, this is dumb and I write a script 385 00:19:13,380 --> 00:19:15,990 that'll just create the BooFuzz script for me, 386 00:19:15,990 --> 00:19:19,260 and then I end up with an 18,000 line BooFuzz script 387 00:19:19,260 --> 00:19:20,700 that gets me nothing. 388 00:19:20,700 --> 00:19:23,433 I ran that for a couple of MUMPS, absolutely nothing, 389 00:19:24,330 --> 00:19:27,270 so I switched tech, I think to myself, 390 00:19:27,270 --> 00:19:30,060 I don't necessarily, the network is slow. 391 00:19:30,060 --> 00:19:31,800 Let's see if we can cut the network out. 392 00:19:31,800 --> 00:19:35,790 So, I learn enough MUMPS to write a harness 393 00:19:35,790 --> 00:19:38,490 that will take input from standard in instead of 394 00:19:38,490 --> 00:19:42,030 from that instead of from a socket 395 00:19:42,030 --> 00:19:44,820 and it will still hit that RPC code. 396 00:19:44,820 --> 00:19:46,350 So, after doing that, 397 00:19:46,350 --> 00:19:49,620 I can now use AFL++ in dumb mode, 398 00:19:49,620 --> 00:19:51,690 not have to worry about instrumentation, 399 00:19:51,690 --> 00:19:54,903 just feed input into this thing and see if it dies, 400 00:19:56,310 --> 00:19:59,640 and that also doesn't seem to be working, 401 00:19:59,640 --> 00:20:00,690 so I think to myself, 402 00:20:01,650 --> 00:20:04,350 what if I just instrument it and then I can see 403 00:20:04,350 --> 00:20:06,873 if code is actually getting hit. 404 00:20:08,820 --> 00:20:10,380 And so, we need to talk a little bit 405 00:20:10,380 --> 00:20:13,680 about instrumenting some MUMPS implementations here. 406 00:20:13,680 --> 00:20:15,420 So there's two MUMPS implementations 407 00:20:15,420 --> 00:20:17,370 that I kind of care about for this research. 408 00:20:17,370 --> 00:20:20,640 The first one is GT.M. The other one is MUMPS. 409 00:20:20,640 --> 00:20:22,200 Both of them are open source. 410 00:20:22,200 --> 00:20:24,630 MUMPS is based on GT.M 411 00:20:24,630 --> 00:20:26,880 because of some historical reasons. 412 00:20:26,880 --> 00:20:30,570 The VistA deployment that I was working on was based on GT.M 413 00:20:30,570 --> 00:20:34,050 so, I have already have a stood up GT.M instance, 414 00:20:34,050 --> 00:20:37,560 MUMPS is very easy to get going if you wanna get it going, 415 00:20:37,560 --> 00:20:41,310 and both MUMPS and GT.M are written by C wizards 416 00:20:41,310 --> 00:20:43,563 who are way cooler than me, 417 00:20:44,640 --> 00:20:47,520 and they do everything they possibly can 418 00:20:47,520 --> 00:20:51,810 to make C even faster, big parts of it are written 419 00:20:51,810 --> 00:20:54,630 in assembly which is of GT.M and MUMPS 420 00:20:54,630 --> 00:20:55,713 which is fascinating, 421 00:20:56,700 --> 00:20:59,640 but all I have to do is make three changes to the code 422 00:20:59,640 --> 00:21:02,823 to get AFL to work which is good for me, 423 00:21:04,140 --> 00:21:05,847 and so, since I'm down here anyway 424 00:21:05,847 --> 00:21:08,910 and instrumenting this underlying MUMPS implementation, 425 00:21:08,910 --> 00:21:11,160 I figured I might as well 426 00:21:11,160 --> 00:21:15,750 just fuzz the MUMPS implementation anyway 427 00:21:15,750 --> 00:21:18,843 like fuzz how it handles source code input, 428 00:21:19,920 --> 00:21:22,320 and so to do that, MUMPS has all 429 00:21:22,320 --> 00:21:25,860 of these test-driven development source code examples 430 00:21:25,860 --> 00:21:29,670 that they all explore weird states, 431 00:21:29,670 --> 00:21:33,210 and so that's a really perfect corpus for this, 432 00:21:33,210 --> 00:21:34,680 and in general, that's my advice. 433 00:21:34,680 --> 00:21:36,900 If you're fuzzing something that you don't understand, 434 00:21:36,900 --> 00:21:40,170 but they have code tests, just steal their code tests, 435 00:21:40,170 --> 00:21:42,840 just steal their test inputs and just use those. 436 00:21:42,840 --> 00:21:45,090 It works a lot, 437 00:21:45,090 --> 00:21:47,610 and then at this point, I've written enough MUMPS 438 00:21:47,610 --> 00:21:50,280 that I can finally read MUMPS. 439 00:21:50,280 --> 00:21:54,240 So now I can actually go through and read the source code 440 00:21:54,240 --> 00:21:56,370 and make some sense of it, 441 00:21:56,370 --> 00:21:57,990 so, I start looking through the authentication. 442 00:21:57,990 --> 00:21:59,310 I start looking at the input handling. 443 00:21:59,310 --> 00:22:00,600 I start looking at how it interacts 444 00:22:00,600 --> 00:22:01,920 with the underlying system, 445 00:22:01,920 --> 00:22:05,100 mostly just looking for quick wins and stuff. 446 00:22:05,100 --> 00:22:08,400 So, we've got three pathways of attack here. 447 00:22:08,400 --> 00:22:11,280 First, we're fuzzing the VistA RPC mechanisms 448 00:22:11,280 --> 00:22:13,680 using a MUMPS harness and AFL++. 449 00:22:13,680 --> 00:22:16,080 Second, I'm just fuzzing how MUMPS 450 00:22:16,080 --> 00:22:21,080 and GT.M handle source code input using MUMPS's tests, 451 00:22:22,530 --> 00:22:25,890 and then third, I'm just looking through the code by hand, 452 00:22:25,890 --> 00:22:28,143 looking for anything weird that I can see. 453 00:22:29,550 --> 00:22:31,230 So, what'd we find? 454 00:22:31,230 --> 00:22:35,340 So, first off, the RPC fuzzing got us just nothing. 455 00:22:35,340 --> 00:22:37,260 There's a really boring technical reason for this 456 00:22:37,260 --> 00:22:40,410 that I'm not gonna get into, but just absolutely nothing. 457 00:22:40,410 --> 00:22:45,153 Fuzzing YottoDP and GT.M got us 30 CVEs, 458 00:22:46,110 --> 00:22:48,480 all of those are memory corruption bugs. 459 00:22:48,480 --> 00:22:51,480 It's everything from buffer overflows to use after freeze 460 00:22:51,480 --> 00:22:52,650 to null pointer derereferences 461 00:22:52,650 --> 00:22:54,810 to everything you can possibly imagine, 462 00:22:54,810 --> 00:22:57,270 and I wanna be really clear about what the attack surface 463 00:22:57,270 --> 00:22:58,890 for those looks like this. 464 00:22:58,890 --> 00:23:01,350 I'm talking about modifying source code 465 00:23:01,350 --> 00:23:03,390 that gets fed to the interpreter, 466 00:23:03,390 --> 00:23:05,880 so you have to be in a really specific spot 467 00:23:05,880 --> 00:23:07,320 to exploit these. 468 00:23:07,320 --> 00:23:10,530 I don't think it's easier than you would expect 469 00:23:10,530 --> 00:23:12,870 to get there, but yeah, so cool. 470 00:23:12,870 --> 00:23:15,997 These CVEs are CVE-2021-4481 to 44510 471 00:23:19,260 --> 00:23:21,600 and these bugs are weird. 472 00:23:21,600 --> 00:23:24,660 Like I said, this was written by sea graybeards 473 00:23:24,660 --> 00:23:29,310 using every possible trick you can imagine, 474 00:23:29,310 --> 00:23:31,110 and so there's all of these weird states 475 00:23:31,110 --> 00:23:34,500 that ended up getting explored doing all of this. 476 00:23:34,500 --> 00:23:36,990 So let's like take one of those bugs and talk about it. 477 00:23:36,990 --> 00:23:39,990 So we're gonna look at 4486 478 00:23:39,990 --> 00:23:41,940 So what I'm gonna do is I'm gonna show you the input. 479 00:23:41,940 --> 00:23:43,680 I'm gonna show you the crash. 480 00:23:43,680 --> 00:23:45,990 We're gonna talk about why this crash is happening, 481 00:23:45,990 --> 00:23:47,340 and then I'm gonna show you the crash again 482 00:23:47,340 --> 00:23:48,840 from a different angle and show you 483 00:23:48,840 --> 00:23:52,920 what actually is causing the memory corruption here. 484 00:23:52,920 --> 00:23:53,753 One sec, 485 00:23:59,400 --> 00:24:00,650 all right, so here we go. 486 00:24:01,890 --> 00:24:06,690 So, first I'm just gonna open that input 487 00:24:06,690 --> 00:24:07,920 and just kind of show this to you. 488 00:24:07,920 --> 00:24:10,710 This is the input that is gonna cause the crash. 489 00:24:10,710 --> 00:24:15,060 This is just a non minified input that the fuzzer found. 490 00:24:15,060 --> 00:24:17,730 If you look at this line here, 491 00:24:17,730 --> 00:24:19,410 you can see this right command 492 00:24:19,410 --> 00:24:22,173 which is actually what causes the crash to happen. 493 00:24:23,700 --> 00:24:25,230 So if I bail out of vim real quick, 494 00:24:25,230 --> 00:24:28,530 I'm just gonna run YottoDB in GDB. 495 00:24:28,530 --> 00:24:30,840 It is configured to just read that input 496 00:24:30,840 --> 00:24:34,110 and try to create a source code or a shared object 497 00:24:34,110 --> 00:24:36,453 from that and we get this Seg fault, right. 498 00:24:37,680 --> 00:24:41,193 If we take a look at the state of the registers, 499 00:24:43,020 --> 00:24:45,840 what we will see here is that rip is at this, 500 00:24:45,840 --> 00:24:50,550 the instruction pointer is at 555c6950s, 501 00:24:50,550 --> 00:24:54,630 and so, if we look at the instructions around that location, 502 00:24:54,630 --> 00:24:56,820 there's just sort a bunch of garbage there. 503 00:24:56,820 --> 00:24:58,800 There's that instruction at the bottom 504 00:24:58,800 --> 00:25:01,083 that GDP can't really make sense of, 505 00:25:02,280 --> 00:25:03,300 and I'm gonna talk about this later, 506 00:25:03,300 --> 00:25:04,680 but that's somewhere in the heap, 507 00:25:04,680 --> 00:25:07,410 just trust me that that's in the heap. 508 00:25:07,410 --> 00:25:09,210 So if we look at the line of source code 509 00:25:09,210 --> 00:25:12,930 that caused this crash, it's inside of op_write, 510 00:25:12,930 --> 00:25:14,280 and there's this call 511 00:25:14,280 --> 00:25:18,443 that uses io_curr_device.out=>disp_ptr. 512 00:25:20,250 --> 00:25:21,390 If I print that, 513 00:25:21,390 --> 00:25:24,120 you can see that there's some memory addresses in here 514 00:25:24,120 --> 00:25:26,070 that don't make a lot of sense to me, 515 00:25:26,070 --> 00:25:31,070 but there's also that write function pointer is at 555c6950. 516 00:25:33,510 --> 00:25:37,680 So, what's actually happening here. 517 00:25:37,680 --> 00:25:41,490 There's a specific order of strings being created 518 00:25:41,490 --> 00:25:44,580 and attempts to compile the code 519 00:25:44,580 --> 00:25:47,670 that is corrupting some data structure in memory 520 00:25:47,670 --> 00:25:49,590 that contains a function pointer. 521 00:25:49,590 --> 00:25:52,410 So then later in the source code file that's being parsed, 522 00:25:52,410 --> 00:25:54,120 there's this call to write 523 00:25:54,120 --> 00:25:56,040 that where the function pointer gets corrupted, 524 00:25:56,040 --> 00:25:59,130 and we just jump out into the middle of absolutely nowhere. 525 00:25:59,130 --> 00:26:00,780 So in this case, 526 00:26:00,780 --> 00:26:02,160 we're jumping to somewhere in the middle of the heap, 527 00:26:02,160 --> 00:26:04,773 but that's just purely chance in this case. 528 00:26:05,700 --> 00:26:07,500 So the thing that's actually being corrupted 529 00:26:07,500 --> 00:26:10,020 is this io_curr_device.out. 530 00:26:10,020 --> 00:26:14,310 io_curr_device.out. is the current input, output device. 531 00:26:14,310 --> 00:26:17,760 It handles taking input from the user and also printing 532 00:26:17,760 --> 00:26:21,240 and emitting source code and stuff like that. 533 00:26:21,240 --> 00:26:25,653 It has a dispatch table that's called disp_ptr, 534 00:26:26,760 --> 00:26:29,910 and that dispatch table is just a bunch 535 00:26:29,910 --> 00:26:33,360 of function pointers that point to different functions 536 00:26:33,360 --> 00:26:35,970 that you can rewrite on the fly if you need to change 537 00:26:35,970 --> 00:26:38,100 what the MUMPS implementation is doing, 538 00:26:38,100 --> 00:26:42,180 and then, we are trying to perform the write function, 539 00:26:42,180 --> 00:26:44,943 that's in that dispatch pointer using some input. 540 00:26:46,260 --> 00:26:47,580 So once the corruption happens, 541 00:26:47,580 --> 00:26:51,600 we end up with this where that io_curr_device.out 542 00:26:51,600 --> 00:26:53,340 just gets corrupted, 543 00:26:53,340 --> 00:26:55,410 so it's completely kind of destroy. 544 00:26:55,410 --> 00:26:57,360 That dispatch pointer, excuse me, 545 00:26:57,360 --> 00:27:01,230 points to just somewhere randomly which means that 546 00:27:01,230 --> 00:27:05,430 that write function call is completely random. 547 00:27:05,430 --> 00:27:10,430 It's just some other, some area of memory basically, 548 00:27:12,270 --> 00:27:15,960 so but why does that happen? 549 00:27:15,960 --> 00:27:19,230 And what actually is this corruption look like? 550 00:27:19,230 --> 00:27:23,130 Basically, what we end up having is these two objects 551 00:27:23,130 --> 00:27:26,910 in memory that are at the same memory location. 552 00:27:26,910 --> 00:27:28,650 We're overlapping two chunks, basically. 553 00:27:28,650 --> 00:27:31,980 I'll explain this more later, but in other contexts, 554 00:27:31,980 --> 00:27:34,740 if you're just doing normal heap exploitation things, 555 00:27:34,740 --> 00:27:38,310 you can get into a similar state using a use after free 556 00:27:38,310 --> 00:27:39,333 or a double free, 557 00:27:40,350 --> 00:27:43,170 and let me demonstrate that for you. 558 00:27:43,170 --> 00:27:45,030 I'm gonna look at that crash again, 559 00:27:45,030 --> 00:27:46,980 but we're gonna take a slightly different look and look 560 00:27:46,980 --> 00:27:50,910 at the way specifically that malloc is being called here. 561 00:27:50,910 --> 00:27:55,860 So, let me restart the program 562 00:27:55,860 --> 00:27:58,020 and then we're gonna run it again with that same input, 563 00:27:58,020 --> 00:28:03,020 just and see what happens, and if we take a look here 564 00:28:03,630 --> 00:28:08,630 at this is we are inside of op_write and now we're breaking 565 00:28:08,820 --> 00:28:13,170 at this anchor link function call. 566 00:28:13,170 --> 00:28:18,170 Before a call to malloc, dispatch pointer looks fine. 567 00:28:18,990 --> 00:28:22,890 The symbols are being like this is correct, 568 00:28:22,890 --> 00:28:26,220 and if we look at some strings around that area 569 00:28:26,220 --> 00:28:27,540 where that dispatch pointers is, 570 00:28:27,540 --> 00:28:29,820 or where curr_device sought out is, 571 00:28:29,820 --> 00:28:32,040 there's nothing really reasonable. 572 00:28:32,040 --> 00:28:37,040 After a call to malloc, there's this macro that gets called 573 00:28:37,050 --> 00:28:39,690 that uses the output that it gets from malloc, 574 00:28:39,690 --> 00:28:42,420 and if we check io_curr_device after that, 575 00:28:42,420 --> 00:28:45,060 now all of a sudden there's a string written there, 576 00:28:45,060 --> 00:28:49,470 so we're overwriting some data that's in that curr_device 577 00:28:49,470 --> 00:28:53,220 that io_curr_device and the dispatch points are now is 578 00:28:53,220 --> 00:28:55,740 just completely collaborative like it is just nonsense, 579 00:28:55,740 --> 00:28:57,990 and if you look at the rest of this, 580 00:28:57,990 --> 00:29:01,230 all of these have been just completely destroyed. 581 00:29:01,230 --> 00:29:03,240 So let me rerun that again, 582 00:29:03,240 --> 00:29:05,370 and this time we're gonna step into that call to malloc 583 00:29:05,370 --> 00:29:08,073 to figure out what the malloc is actually doing. 584 00:29:09,570 --> 00:29:11,760 So here's our completely normal call to malloc. 585 00:29:11,760 --> 00:29:16,620 We step in and we are not in malloc. This is gtm_malloc. 586 00:29:16,620 --> 00:29:18,070 They wrote their own malloc 587 00:29:19,140 --> 00:29:22,290 and replaced the system malloc with it, 588 00:29:22,290 --> 00:29:26,460 so if I break at another macro later inside 589 00:29:26,460 --> 00:29:31,460 of this custom malic, there's this call to get cued element 590 00:29:31,890 --> 00:29:36,890 that gets some piece of memory that starts around like e200, 591 00:29:37,050 --> 00:29:40,950 ignore 55555, somewhere on the heap e200, 592 00:29:40,950 --> 00:29:44,790 and if I look at where curr_device.out is, 593 00:29:44,790 --> 00:29:49,023 it is at e210, so there's 16 bites between those two. 594 00:29:49,980 --> 00:29:53,853 So, before that crashing call, 595 00:29:56,842 --> 00:29:58,860 before the crash that happens op_write, 596 00:29:58,860 --> 00:30:00,390 io_curr-Device is well formed, 597 00:30:00,390 --> 00:30:03,723 and it's at a memory address that ends with e210. 598 00:30:04,650 --> 00:30:08,820 There's this call to malloc that goes to gtm_malloc 599 00:30:08,820 --> 00:30:10,950 instead of GlibC malloc 600 00:30:10,950 --> 00:30:14,610 and eventually returns this memory address 601 00:30:14,610 --> 00:30:16,293 that ends an e200. 602 00:30:17,318 --> 00:30:20,340 The devs have made their own memory allocator inside 603 00:30:20,340 --> 00:30:22,170 of the heat that manages, 604 00:30:22,170 --> 00:30:23,700 there is the heat memory allocator, 605 00:30:23,700 --> 00:30:25,500 and then there is their memory allocator 606 00:30:25,500 --> 00:30:27,933 managing the same locations in memory. 607 00:30:29,100 --> 00:30:31,890 There's at least two memory allocators in use 608 00:30:31,890 --> 00:30:35,100 on this application which is just super wild, 609 00:30:35,100 --> 00:30:38,250 and by a little bit of some magic, 610 00:30:38,250 --> 00:30:40,890 you can get that second memory allocator 611 00:30:40,890 --> 00:30:44,130 to return overlapping chunks, basically. 612 00:30:44,130 --> 00:30:46,740 So just to do this a little bit visually 613 00:30:46,740 --> 00:30:49,860 on the far left here, we have the way 614 00:30:49,860 --> 00:30:51,960 that process memory is laid out. 615 00:30:51,960 --> 00:30:54,300 You've got the text at low addresses, you've got the memory, 616 00:30:54,300 --> 00:30:57,150 you've got the heap, and then there's like the heap, right? 617 00:30:57,150 --> 00:31:00,060 The heap is made up of chunks, 618 00:31:00,060 --> 00:31:01,560 memory that is either allocated 619 00:31:01,560 --> 00:31:03,870 in which case it's labeled chunk or it's freed 620 00:31:03,870 --> 00:31:07,743 like in the memory allocator can do whatever it wants to it. 621 00:31:08,670 --> 00:31:11,283 If we take a look at one of those chunks, 622 00:31:12,150 --> 00:31:15,180 we've just got some memory that we can use for whatever. 623 00:31:15,180 --> 00:31:20,100 During initialization, GT.M and YottoDB allocate a chunk, 624 00:31:20,100 --> 00:31:22,380 a really big chunk, and then they just say, 625 00:31:22,380 --> 00:31:24,630 this is the memory that we are going to use 626 00:31:24,630 --> 00:31:26,583 for any MUMPS program that's written. 627 00:31:27,600 --> 00:31:32,600 So then, when io_curr_device runs, 628 00:31:32,760 --> 00:31:37,140 or when gtm runs, it initializes this io_curr_device, 629 00:31:37,140 --> 00:31:39,330 somewhere in that same memory space, 630 00:31:39,330 --> 00:31:43,650 and then that gtm_malloc returns a similar look 631 00:31:43,650 --> 00:31:47,640 of memory space and they overwrite that io_curr_device, 632 00:31:47,640 --> 00:31:50,640 so you have in one part of the code, 633 00:31:50,640 --> 00:31:52,530 the code thinking that we're looking at the input 634 00:31:52,530 --> 00:31:56,100 and output device and in a different part of the code, 635 00:31:56,100 --> 00:31:59,790 they think it's a routine header which lets us get 636 00:31:59,790 --> 00:32:02,673 that overlapping chunks type confusion thing. 637 00:32:03,750 --> 00:32:06,900 So, basically we have these two mechanisms 638 00:32:06,900 --> 00:32:09,120 that are managing memory malloc and gtm_malloc, 639 00:32:09,120 --> 00:32:13,170 and then we get a type confusion bug in the way 640 00:32:13,170 --> 00:32:17,220 that gtm_malloc specifically is handling that memory. 641 00:32:17,220 --> 00:32:21,300 So, this is a heat bug inside of a memory manager 642 00:32:21,300 --> 00:32:23,250 that is managing memory managed 643 00:32:23,250 --> 00:32:25,598 by a different memory manager. 644 00:32:25,598 --> 00:32:27,240 (people cheering) So like yeah, 645 00:32:27,240 --> 00:32:30,180 the address there is not completely random, 646 00:32:30,180 --> 00:32:32,340 but it's not really in our control, 647 00:32:32,340 --> 00:32:34,710 and I really wanted to talk about this bug in particular 648 00:32:34,710 --> 00:32:38,403 'cause it's just so (beep) weird. It's so weird. 649 00:32:40,320 --> 00:32:44,220 That's what we got looking at MUMPS 650 00:32:44,220 --> 00:32:45,390 and limitations themselves. 651 00:32:45,390 --> 00:32:46,740 So what about that source review 652 00:32:46,740 --> 00:32:50,010 where we were just looking at VistA, right? 653 00:32:50,010 --> 00:32:53,340 This next slide I have to read really carefully. 654 00:32:53,340 --> 00:32:55,980 The source code review just looked at just was looking 655 00:32:55,980 --> 00:32:58,740 for quick wins and only looked at the auth mechanisms, 656 00:32:58,740 --> 00:33:00,420 input handling, by how it interacts 657 00:33:00,420 --> 00:33:02,190 with the underlying system. 658 00:33:02,190 --> 00:33:04,530 So that RPC mechanism I was talking about 659 00:33:04,530 --> 00:33:09,300 that the clients use is gated by a encryption mechanism 660 00:33:09,300 --> 00:33:12,963 that uses roll-your-own encryption from the 90s. 661 00:33:14,010 --> 00:33:18,210 So if you deploy VistA without TLS, 662 00:33:18,210 --> 00:33:21,420 creds are poorly encrypted and transmitted in a way 663 00:33:21,420 --> 00:33:24,360 that attackers can trivially decrypt them 664 00:33:24,360 --> 00:33:26,460 or simply replay the packets. 665 00:33:26,460 --> 00:33:29,340 There also appears to be hardcoded creds in the source, 666 00:33:29,340 --> 00:33:31,380 but because of some particulars, 667 00:33:31,380 --> 00:33:34,053 I'm not super sure that they can be used, 668 00:33:34,920 --> 00:33:38,070 and I would absolutely love to explain to you 669 00:33:38,070 --> 00:33:43,070 how this works, but we had some problems disclosing this. 670 00:33:43,470 --> 00:33:46,590 So let me show you my disclosure timeline real fast. 671 00:33:46,590 --> 00:33:48,090 So on January 3rd, 672 00:33:48,090 --> 00:33:49,500 we sent an email to the VA 673 00:33:49,500 --> 00:33:52,140 at following their disclosure policy. 674 00:33:52,140 --> 00:33:53,737 We received an automated email that said, 675 00:33:53,737 --> 00:33:55,620 "Somebody will email you back," 676 00:33:55,620 --> 00:33:57,360 nobody emailed us back, 677 00:33:57,360 --> 00:33:59,490 so, then on the 10th, I emailed them. 678 00:33:59,490 --> 00:34:00,547 I sent them another email that said, 679 00:34:00,547 --> 00:34:02,850 "Hey, there's some problems I really want 680 00:34:02,850 --> 00:34:04,110 to talk to you about these." 681 00:34:04,110 --> 00:34:06,840 They sent another automated email, no follow up. 682 00:34:06,840 --> 00:34:09,960 We never got another email from the VA after this, 683 00:34:09,960 --> 00:34:13,773 and then on the 10th, I sent another email and got nothing. 684 00:34:14,910 --> 00:34:18,690 So I assume that either something changed on their back end 685 00:34:18,690 --> 00:34:22,200 or they just blocked my email address, so cool. 686 00:34:22,200 --> 00:34:23,400 So then I reached out to somebody I know 687 00:34:23,400 --> 00:34:26,760 who works at the DHS. They did not respond. 688 00:34:26,760 --> 00:34:29,310 I then reached out to CISA directly. 689 00:34:29,310 --> 00:34:31,020 They also did not respond, 690 00:34:31,020 --> 00:34:33,480 and then I called CISA on the phone. 691 00:34:33,480 --> 00:34:34,860 This is a thing that you can do. 692 00:34:34,860 --> 00:34:36,870 Their phone number is on the internet. 693 00:34:36,870 --> 00:34:39,270 You can find them and call them. 694 00:34:39,270 --> 00:34:41,940 And somehow I think because of a phone tree thing, 695 00:34:41,940 --> 00:34:44,760 that call just got disconnected before I could ever speak 696 00:34:44,760 --> 00:34:47,880 to somebody and explain what the hell was going on. 697 00:34:47,880 --> 00:34:49,650 So then I called CISA again, 698 00:34:49,650 --> 00:34:53,250 and it was told that any information that I give them is 699 00:34:53,250 --> 00:34:54,780 just not gonna be provided to the VA. 700 00:34:54,780 --> 00:34:56,970 They're not gonna give it to the people 701 00:34:56,970 --> 00:34:58,080 to try to fix the bug, right? 702 00:34:58,080 --> 00:34:58,913 They said they were gonna give it 703 00:34:58,913 --> 00:35:00,900 to their threat hunting teams. 704 00:35:00,900 --> 00:35:04,770 So then I reached out to CMU CERT and I received an email 705 00:35:04,770 --> 00:35:06,720 that was like, "Hey, give me more details," 706 00:35:06,720 --> 00:35:10,803 and then they didn't respond to responses to that email. 707 00:35:11,700 --> 00:35:13,380 This says they never responded. 708 00:35:13,380 --> 00:35:16,953 Within the last week, they have started responding, 709 00:35:17,880 --> 00:35:20,880 but I don't believe they, 710 00:35:20,880 --> 00:35:22,770 I'm not super sure what's going on there. 711 00:35:22,770 --> 00:35:25,710 I still don't think the VA has been told about this problem, 712 00:35:25,710 --> 00:35:29,640 and I wanna be clear, this is an EMR that is deployed 713 00:35:29,640 --> 00:35:33,450 in like it is at VA hospitals right now. 714 00:35:33,450 --> 00:35:36,543 It is also at civilian hospitals in the United States. 715 00:35:38,130 --> 00:35:40,530 So, also we disclosed a lot of bugs 716 00:35:40,530 --> 00:35:43,470 to a bunch of MUMPS distributions. 717 00:35:43,470 --> 00:35:45,427 For YottoDB, we sent them an email that was like, 718 00:35:45,427 --> 00:35:47,370 "Hey, we found some bugs." 719 00:35:47,370 --> 00:35:48,427 They sent us an email back that was like, 720 00:35:48,427 --> 00:35:51,300 "Cool, do you mind teaching us how to do this?" 721 00:35:51,300 --> 00:35:52,980 And I said, "Hell yeah," 722 00:35:52,980 --> 00:35:54,457 and then we explained to them like, 723 00:35:54,457 --> 00:35:55,470 "Hey, here's how you fuzz. 724 00:35:55,470 --> 00:35:57,690 Here's like the changes you need to make," 725 00:35:57,690 --> 00:35:58,590 and then they started fuzzing 726 00:35:58,590 --> 00:36:01,680 and they've found tons and tons and tons of bugs, 727 00:36:01,680 --> 00:36:05,550 and then by February, we just go to them in November. 728 00:36:05,550 --> 00:36:06,840 By February, new version was out 729 00:36:06,840 --> 00:36:08,820 that had all the fixes in it. 730 00:36:08,820 --> 00:36:10,710 GT.M, we sent them in the email in December. 731 00:36:10,710 --> 00:36:13,353 By March, they had fixed all of the bugs. 732 00:36:14,341 --> 00:36:15,780 (people clapping) So yeah, 733 00:36:15,780 --> 00:36:17,180 that's our disclosure there. 734 00:36:21,960 --> 00:36:23,310 So what does this all mean? 735 00:36:24,540 --> 00:36:28,830 When I first started looking at this research, 736 00:36:28,830 --> 00:36:30,240 I've done a lot of fuzzing projects 737 00:36:30,240 --> 00:36:32,100 and never found anything. 738 00:36:32,100 --> 00:36:33,150 I always kind of figured that, 739 00:36:33,150 --> 00:36:36,300 oh, all the fun memory corruption bugs were dead, right? 740 00:36:36,300 --> 00:36:40,260 Nope, there's still big stuff out there that has real kind 741 00:36:40,260 --> 00:36:41,960 of obvious memory corruption bugs. 742 00:36:42,960 --> 00:36:44,910 MUMPS isn't really gonna go anywhere. 743 00:36:44,910 --> 00:36:48,600 I think at this point we're stuck with it. 744 00:36:48,600 --> 00:36:51,660 When MUMPS first got off the ground, 745 00:36:51,660 --> 00:36:52,650 people started using it for stuff. 746 00:36:52,650 --> 00:36:54,300 It was faster than everything. 747 00:36:54,300 --> 00:36:56,070 It was cutting edge. It was innovative. 748 00:36:56,070 --> 00:36:57,420 It was everything you want, 749 00:36:58,380 --> 00:37:00,540 and a lot of companies jumped on this bandwagon 750 00:37:00,540 --> 00:37:01,500 and are still there. 751 00:37:01,500 --> 00:37:04,890 Core banking uses MUMPS, a bunch of healthcare stuff 752 00:37:04,890 --> 00:37:07,830 uses MUMPS outside of just VistA. 753 00:37:07,830 --> 00:37:10,110 Like I said, the ESA uses MUMPS. 754 00:37:10,110 --> 00:37:12,000 Based on some numbers that I've seen, 755 00:37:12,000 --> 00:37:15,120 more than 50% of healthcare records in the United States 756 00:37:15,120 --> 00:37:17,730 pass through some application that's written 757 00:37:17,730 --> 00:37:19,890 in this language at some point, 758 00:37:19,890 --> 00:37:22,440 and there's still more weird machines 759 00:37:22,440 --> 00:37:23,340 out there to break. 760 00:37:23,340 --> 00:37:26,820 There's still more stuff to find. 761 00:37:26,820 --> 00:37:27,720 So what should you do? 762 00:37:27,720 --> 00:37:32,100 If you're working on VistA or a VistA derived product 763 00:37:32,100 --> 00:37:33,330 or something like that, 764 00:37:33,330 --> 00:37:36,783 make sure you're deploying TLS everywhere. 765 00:37:37,710 --> 00:37:42,060 This deploying TLS is not difficult 766 00:37:42,060 --> 00:37:46,020 on a lot of MUMPS implementations, but it's not trivial, 767 00:37:46,020 --> 00:37:50,700 and just make sure even internally do not trust that you're 768 00:37:50,700 --> 00:37:54,930 just behind your inside of your VLANs and everything's fine. 769 00:37:54,930 --> 00:37:57,420 Make sure you've got TLS everywhere. 770 00:37:57,420 --> 00:38:00,480 If you deploy MUMPS or a MUMPS-based product, 771 00:38:00,480 --> 00:38:01,320 you need to update. 772 00:38:01,320 --> 00:38:04,890 If you're using GT.M from apt, like I said to do earlier, 773 00:38:04,890 --> 00:38:08,070 your four versions behind and your two versions 774 00:38:08,070 --> 00:38:10,860 behind the patch that has all of the fixes to the bugs 775 00:38:10,860 --> 00:38:15,030 that were disclosed, so update basically, probably build 776 00:38:15,030 --> 00:38:17,943 from source, and if you're a hacker looking for research, 777 00:38:18,830 --> 00:38:21,333 I can't think of look at healthcare stuff. 778 00:38:22,671 --> 00:38:25,350 Healthcare stuff is still not getting the eyes 779 00:38:25,350 --> 00:38:28,470 on it that it needs like look at healthcare stuff. 780 00:38:28,470 --> 00:38:31,203 Also, if you work at the VA, send me an email. 781 00:38:33,930 --> 00:38:35,403 This shouldn't be this hard. 782 00:38:40,426 --> 00:38:43,076 I've just trashed this language and this product for, 783 00:38:43,950 --> 00:38:45,450 at this point like 40 minutes, 784 00:38:45,450 --> 00:38:47,460 and I just wanna say that everybody 785 00:38:47,460 --> 00:38:49,563 who worked on this is a hero to me. 786 00:38:51,030 --> 00:38:52,530 I am not kidding about this. 787 00:38:52,530 --> 00:38:54,360 Looking at some of this code, you see these names 788 00:38:54,360 --> 00:38:57,690 that these people that are wizards, right? 789 00:38:57,690 --> 00:39:02,690 MUMPS was this incredible idea that was just like, 790 00:39:03,270 --> 00:39:04,740 Hey, we have computers now, 791 00:39:04,740 --> 00:39:06,480 what can we do for healthcare stuff? 792 00:39:06,480 --> 00:39:10,620 And they made this incredible thing that is still in use 793 00:39:10,620 --> 00:39:13,620 that you can still play around with and still learn. 794 00:39:13,620 --> 00:39:15,810 It's not exactly an esoteric language, 795 00:39:15,810 --> 00:39:18,150 but if you're looking for a new esolang, 796 00:39:18,150 --> 00:39:20,553 look at MUMPS, it's neat. 797 00:39:21,690 --> 00:39:25,920 VistA as the EMR is super well-respected. 798 00:39:25,920 --> 00:39:27,420 It's really flexible. 799 00:39:27,420 --> 00:39:29,460 There's a story about the VA has just been 800 00:39:29,460 --> 00:39:31,923 in constant scandal forever, forever, and ever, 801 00:39:33,270 --> 00:39:38,270 and there's a story about how during some bad times, 802 00:39:39,750 --> 00:39:43,590 they basically, there was this congressional testimony 803 00:39:43,590 --> 00:39:46,890 that was like, "Oh, everything over there is broken 804 00:39:46,890 --> 00:39:48,300 except for that EMR. 805 00:39:48,300 --> 00:39:50,790 That EMR is the best EMR I have ever seen," 806 00:39:50,790 --> 00:39:53,910 and a bunch of doctors have said this about MUMPS, 807 00:39:53,910 --> 00:39:57,063 and I like, just as a fun little side thing, 808 00:39:59,490 --> 00:40:02,250 VistA got named VistA in like 1994 809 00:40:02,250 --> 00:40:04,950 which is almost 30 years ago now, 810 00:40:04,950 --> 00:40:06,600 and the reason why they called it VistA is 811 00:40:06,600 --> 00:40:09,190 because previously they were calling it DHCP 812 00:40:10,770 --> 00:40:12,720 which then led to be a problem. 813 00:40:12,720 --> 00:40:16,230 I think it stands for distributed healthcare program 814 00:40:16,230 --> 00:40:17,370 or something like that, 815 00:40:17,370 --> 00:40:20,100 and they renamed it to VistA, capital V, capital A 816 00:40:20,100 --> 00:40:21,757 and I'm sure at the time they were like, 817 00:40:21,757 --> 00:40:23,370 "This is the greatest name. 818 00:40:23,370 --> 00:40:26,280 No one will ever scoop us on this name." 819 00:40:26,280 --> 00:40:29,073 Why the hell would you call something VistA? 820 00:40:30,510 --> 00:40:32,310 And then a decade later, 821 00:40:32,310 --> 00:40:36,300 here comes Bill Gates's microsoft.com. 822 00:40:36,300 --> 00:40:39,600 So, that's slide 63 of 64. 823 00:40:39,600 --> 00:40:41,970 Here's slide 63 and a half. 824 00:40:41,970 --> 00:40:44,100 This is my great slide. 825 00:40:44,100 --> 00:40:45,300 Thank you for having me. 826 00:40:46,284 --> 00:40:47,850 This really means a lot to me, 827 00:40:47,850 --> 00:40:51,000 this community and everybody who has stood on the stage 828 00:40:51,000 --> 00:40:53,760 and all of the research that everyone who has ever gone 829 00:40:53,760 --> 00:40:57,543 to DEFCON has ever done has been a great inspiration to me. 830 00:40:58,890 --> 00:41:00,420 Thanks to everybody at that side. 831 00:41:00,420 --> 00:41:03,423 Thanks to just everybody who helped out on this. 832 00:41:06,120 --> 00:41:09,480 Thank you so much for your time. That's everything I got. 833 00:41:09,480 --> 00:41:12,840 You can yell at me at Twitter, (people clapping) 834 00:41:12,840 --> 00:41:13,673 That's it, yeah. 835 00:41:17,970 --> 00:41:19,740 - [Audience Member] You're my hero. 836 00:41:19,740 --> 00:41:21,213 - Thank you. You're my hero. 837 00:41:23,250 --> 00:41:24,330 All right, oh yeah, 838 00:41:24,330 --> 00:41:27,550 also my Twitter account is the word binaries backwards 839 00:41:28,920 --> 00:41:31,493 which it's just doesn't look like it, but it is. Cool, yeah. 840 00:41:36,074 --> 00:41:38,365 Do we have time for questions? 841 00:41:38,365 --> 00:41:40,615 - [Man In Red T-Shirt] Yeah 842 00:41:41,594 --> 00:41:42,677 - Cool, yeah. 843 00:41:44,280 --> 00:41:46,480 - Any questions? Anybody have any questions? 844 00:41:48,630 --> 00:41:52,953 All right, I have absolutely killed the crowd, hell yeah. 845 00:41:53,910 --> 00:41:56,610 All right, yeah like I said, go look at MUMPS. 846 00:41:56,610 --> 00:41:58,863 It's a fun language. It's real weird. 847 00:42:00,300 --> 00:42:02,730 There's a lot of multibillion dollar companies 848 00:42:02,730 --> 00:42:04,110 that deploy stuff that's written in MUMPS, 849 00:42:04,110 --> 00:42:07,970 so there's some fun stuff out there. What's up? 850 00:42:07,970 --> 00:42:10,590 (faintly speaking) 851 00:42:10,590 --> 00:42:11,940 There is not, the question is, 852 00:42:11,940 --> 00:42:13,920 is there a wire shark decode for the RPC? 853 00:42:13,920 --> 00:42:18,510 There is not. There is a long dead. 854 00:42:18,510 --> 00:42:20,100 I think there is a get commit. 855 00:42:20,100 --> 00:42:23,880 I can't remember what project it's on for a JS file 856 00:42:23,880 --> 00:42:27,870 that I think is called RPC Snoop which you're not, 857 00:42:27,870 --> 00:42:29,310 that's gonna be difficult to find, 858 00:42:29,310 --> 00:42:32,400 but look for RPC Snoop in relation to VistA 859 00:42:32,400 --> 00:42:33,900 and then, hopefully you'll find it. 860 00:42:33,900 --> 00:42:37,680 It's a node.js file that was floating around. 861 00:42:37,680 --> 00:42:40,023 Yeah, cool. 862 00:42:42,840 --> 00:42:43,953 Any other questions? 863 00:42:46,500 --> 00:42:48,893 All right, thank you very much. 864 00:42:48,893 --> 00:42:53,144 (people cheering and clapping) 865 00:42:53,144 --> 00:42:54,344 Yeah, thank you. - Yeah