WEBVTT 00:00.000 --> 00:08.180 So our next speaker is from NYU Tandon School of Engineering, from the Digital Future Lab. 00:09.100 --> 00:19.340 His focus for this talk will be using engineering and AI to build the next gen of smart scopes capable of diagnostic support. 00:19.700 --> 00:28.660 And fun little fact, on his spare time, he does 3D print graffiti artists in the style of green army men. 00:33.260 --> 00:41.380 So presenting their talk, microscopes are stupid, I agree, here is Lou Auguste. 00:46.700 --> 00:52.580 Hello everybody, good morning, good morning, thanks for coming to see this speech on microscopes are stupid. 00:52.900 --> 01:08.020 My name is Lou Auguste, I work at NYU Tandon School of Engineering in their future lab, so it's really, I'm an entrepreneur in residence there, they incubate my company, been running it there for three years now, and we work on solving this problem. 01:08.160 --> 01:12.600 So just a little bit before we get started, we've won some awards over the last couple of years. 01:12.600 --> 01:16.120 We won the ASME award for best hardware prototype in 2015. 01:16.540 --> 01:22.280 Last year we were nominated for science startup of the year by this competition out in Germany called Falling Walls. 01:22.400 --> 01:32.100 We won the Indian government's endowment for science research last year, and this year we're finalists for Qualcomm's prize for design in India. 01:32.420 --> 01:53.480 I published my methods in the British Medical Journal in 2015 on how to use smartphones to create whole slide images, and I did a pre-clinical validation study of that information in 2016, which I presented at USCAP, which is the United States and Canada Anatomical Pathologist Society. 01:53.940 --> 01:58.960 And this year we received IRB approval to do our clinical validation study, which I'm writing up right now. 02:00.180 --> 02:02.720 So yes, microscopes are stupid. 02:02.720 --> 02:04.460 I said it, and that's it. 02:04.560 --> 02:10.240 If you look at this microscope here, it's such a dinosaur that it's not even connected to the internet. 02:13.360 --> 02:18.840 So basically the problem is microscopes are dependent on the operator, and who's operating that microscope. 02:19.100 --> 02:29.040 And the problem that we're facing is as we look out at more and more of the healthcare landscape in the world, we find that there are things called healthcare deserts. 02:29.040 --> 02:35.200 These are places in America, the developing world, where options for healthcare don't exist. 02:35.400 --> 02:36.860 And it's not that they don't have microscopes. 02:36.860 --> 02:37.940 They have microscopes. 02:37.940 --> 02:40.860 They don't have anyone that could help them do diagnosis. 02:41.680 --> 02:49.220 So this is very true, as I said, in Kentucky, just as true as it is in Haiti, just as true as it is in India. 02:50.760 --> 02:55.220 So, before we go forward, I want to give a shout out to my boy, Anton van Leeuwenhoek. 02:56.260 --> 02:58.980 He's like the G of microscopes. 02:59.020 --> 03:07.620 So basically, he was born in the 1500s, and 40 years before he was born, they had created the compound microscope. 03:08.500 --> 03:13.360 Leeuwenhoek was a tradesman, and he was actually into glassworking. 03:13.360 --> 03:15.180 He didn't have any advanced degrees. 03:15.180 --> 03:17.120 He didn't have a PhD. 03:17.120 --> 03:19.740 So he was eliminated from a lot of the science community. 03:20.120 --> 03:27.260 But what he did was he created, with his knowledge of glassblowing or glassmaking, he created this microscope here. 03:27.360 --> 03:31.880 And this microscope had a magnification of over 200 times magnification. 03:32.200 --> 03:38.660 Up to that point, they had already invented the compound microscope, which is very similar to what we see today in any laboratory. 03:38.760 --> 03:43.120 But that only had a magnification of 20 to 30 times magnification. 03:43.640 --> 03:47.400 So van Leeuwenhoek was definitely one of the first disruptors. 03:47.460 --> 03:54.280 And he's also considered the father of microbiology, because of what he was able to see through his microscope that he created. 03:54.440 --> 04:01.120 And in fact, it wasn't really until the 1800s that they made any advancements past what he made with this microscope here. 04:02.500 --> 04:04.640 So also, humans are stupid. 04:04.680 --> 04:11.560 Because when we're given the task, humans get tired, and they miss things. 04:11.560 --> 04:14.780 And diagnosis itself depends on memory. 04:15.160 --> 04:20.480 So you probably can't see this right here, because the screen's a little bit contrasty. 04:20.620 --> 04:30.480 But if you were looking for cancer on this thing, there's actually a monkey in the upper right-hand corner of the lung that most people miss when they're looking for cancer. 04:30.540 --> 04:38.900 So I'm not sure if you could see that here, but that's usually a good example of how people miss things that are obvious when they're doing something that's repetitive. 04:39.560 --> 04:45.180 So a little history on telepathology, which is sort of what we do. 04:45.420 --> 04:47.540 So this is 1986. 04:47.560 --> 04:51.340 This was the first demonstration of a robotic microscope. 04:51.600 --> 04:59.760 And you can only imagine the satisfaction that this man felt clutching those keys and hearing those sweet clicking sounds as he moved the microscope around. 04:59.980 --> 05:06.080 And really, the history of telepathology is sort of a history of the internet in many ways as well. 05:06.080 --> 05:18.360 Because as things improved with the internet, we were able to do more and more in terms of sharing slides, and then virtually moving on to creating virtual slides and sharing those slides over the cloud. 05:18.540 --> 05:26.140 And now with things like TensorFlow, we're actually getting to the point where we can deploy AI locally on the device. 05:26.360 --> 05:31.260 And that's a big step, I think, that's happening right now in the telepathology industry. 05:32.740 --> 05:43.200 So, really, the work that I've done over the last four years has been focused on helping health outcomes in the developing world and solving the problems that they're facing. 05:43.300 --> 05:46.900 And so one of the biggest problems we look at is this really simple equation. 05:46.900 --> 05:48.940 Lost time equals lost lives. 05:49.060 --> 05:54.340 So there's a worldwide shortage of pathologists or doctors able to make diagnosis. 05:54.660 --> 05:58.240 And because of that, people can't get diagnosed. 05:58.240 --> 06:01.800 And so if people can't get diagnosed, they can't begin treatment. 06:01.860 --> 06:06.800 And the later you wait to begin treatment, the worse your health outcome is going to be. 06:08.720 --> 06:21.120 So a lot of the research we've done is creating just simple ratios when we look and see how many specialists there are in a country versus how many people there are. 06:21.120 --> 06:28.360 And so you can see that in the US, you've got a ratio of basically 1 to 17,000. 06:29.260 --> 06:35.360 But then if you look at a country like China, you've got 1 to 58,000. 06:35.420 --> 06:40.860 And then if you look at a country like Haiti, you've got 1 to 1,375,000. 06:40.920 --> 06:47.320 So you can see that there's just not enough doctors to make diagnosis for all these people. 06:47.820 --> 06:52.680 And there's also things that happen when you're in one of these countries that has a small number of doctors. 06:52.920 --> 06:58.640 You can actually fix your price as one of those doctors, and your price can be a lot higher than it would be in another country. 06:58.860 --> 07:03.960 And so not only are those people not able to get access, but they have to pay a lot more money for access to diagnosis. 07:04.840 --> 07:16.360 So the College of American Pathologists states that between 60% and 70% of all diseases need a diagnosis using some sort of cellular biology. 07:16.360 --> 07:22.300 And that includes cancer, things like malaria, a lot of diseases that we know today. 07:23.060 --> 07:28.240 Anyway, so another thing we've studied is this travel burden. 07:28.540 --> 07:31.840 And a lot of people have done studies here in America. 07:31.840 --> 07:35.960 And so we have three studies here, one from Kentucky, one from Illinois, and one from Georgia. 07:36.240 --> 07:47.120 And basically all these studies stated that people that lived at a far distance away from the primary diagnosis center, they would get staged for cancer much later. 07:47.600 --> 07:49.540 And so that's a big thing that we're facing. 07:49.640 --> 07:57.900 So like I said, if it takes you a long time to get a diagnosis, you're probably going to get diagnosed much later when it's going to be much more difficult to cure. 07:59.160 --> 08:03.840 So to give you an example of one of the patients that I saw when I was down in Haiti. 08:04.020 --> 08:07.660 So we had this woman come about three years ago and present herself. 08:07.660 --> 08:09.000 And she had a breast mask. 08:09.560 --> 08:19.400 And so we had a surgeon at the hospital who was able to do a biopsy for her, but there was no one available that could look at her specimen once we had it. 08:20.020 --> 08:26.660 So we gave her her specimen in a jar, and we gave her instructions on how to get to the nearest laboratory. 08:26.920 --> 08:31.000 And so that laboratory is about six, seven hours away, and she never made that visit. 08:31.320 --> 08:37.960 So a couple of years later, she came back in, and this time the mask had grown so large that it actually had ulcerated out of her breast. 08:37.960 --> 08:45.060 So now you see we definitely can say, okay, well, this without a doubt is going to be breast cancer right here. 08:45.340 --> 08:57.140 But when we first saw her, when she first presented herself, when she only had a small mask, we could have used diagnostic testing to prove that it was breast cancer at that point, and consequently we could have started treatment much earlier. 08:57.500 --> 09:00.340 And that's sort of an example of what happens. 09:00.760 --> 09:08.440 Also it's important to note that the cost of testing, diagnostic testing, in Haiti is $50 per test. 09:08.580 --> 09:12.800 So that's like a really important number to kind of remember. 09:13.440 --> 09:20.520 So the previous solutions were like to travel to Port-au-Prince, find a lab, wait a few months and hopefully get a diagnosis. 09:21.220 --> 09:34.020 What we were doing before, a long time ago, was collecting all the samples, flying them to America, getting a lab to process them in the United States, sending all the results back and praying that we could find all the people that we had tested while we were down there. 09:35.060 --> 09:42.620 Another chance is like the doctor operates based on symptoms, and then really what mostly ends up happening is people die of unknown diseases. 09:43.400 --> 09:46.320 So that's pretty heavy, so I'm going to chill out for a second. 09:49.360 --> 09:54.100 But we're like, we're working to solve this problem because we know we can. 09:54.100 --> 09:57.280 And we know we can use technology to solve this problem. 09:57.700 --> 10:02.360 And so what we've done is we've been focusing on building human-machine teams. 10:02.940 --> 10:10.940 So the vision really is, and it's something that the WHO sort of shares with us as well, is to create more healthcare workers. 10:10.940 --> 10:21.340 Because the paradigm of having a hospital, having a doctor that we know and well love here in America or in other developed nations, it's just not going to work in a lot of these countries. 10:21.340 --> 10:37.360 So we have to take people that are willing to go to school for maybe six months to a year , train them at a skill, that could be a skill like becoming a lab tech, and then give them the technology they need in order to share those slides so that they could make more diagnosis. 10:38.120 --> 10:39.560 And so that's something we've been working on. 10:39.560 --> 10:44.120 This is the team that I go down to Haiti with from time to time to do that. 10:45.300 --> 10:59.900 So another big thing we've been looking at is, okay, so once we can build the capacity for doctors or for healthcare workers to prepare specimens to upload those specimens to the internet, well, who are they going to share those specimens with? 10:59.900 --> 11:03.600 So normally what we did in the past was we shared them with doctors here in America. 11:03.740 --> 11:05.900 But doctors in America are really busy. 11:06.020 --> 11:13.100 They don't have enough time to look at cases from another hospital, especially if it's not helping them on the bottom line, it's not helping them make more money or do whatever. 11:13.700 --> 11:17.680 So what we've actually done is we're working with this team of doctors here in Tanzania. 11:18.240 --> 11:26.040 And so these are all doctors, these are all pathologists, the women that you're seeing here, and that's our friend Mumbek, he's also a doctor in Tanzania. 11:26.040 --> 11:35.000 And what we've been working is in this hospital, Mumbek in Dar es Salaam, we're setting them up with the technology that they need. 11:35.260 --> 11:40.840 So remember when I said it was important, remember that it costs $50 to get a slide diagnosed in Haiti. 11:40.840 --> 11:43.780 Well, in Tanzania, the cost of reading a slide is only $12. 11:43.900 --> 11:49.060 So already we're saving money for the people that are in Haiti by giving them this option. 11:49.200 --> 11:53.060 And the doctors in Tanzania are only making about $700 per month. 11:53.060 --> 11:56.540 So they are actually interested in supplementing their income. 11:57.240 --> 12:02.180 And they also have an unused capacity, because they're sitting in traffic, and most of them have drivers. 12:02.180 --> 12:08.620 So when they're sitting in traffic, there's time for them, if they're on their tablet or smartphone device, to actually be reviewing cases. 12:08.900 --> 12:20.840 So our solution is really to actually send our cases from Haiti to Tanzania to decrease the cost of testing, so the people in Haiti can afford to pay for testing, and they can get a diagnosis quickly. 12:21.340 --> 12:30.760 And then hopefully what we'll do is export the lab training program that we're developing in Haiti, export that over to Tanzania, and then create more lab techs, more health workers, that can work with what we're building. 12:31.900 --> 12:34.000 So that's sort of where we're at. 12:34.080 --> 12:39.440 And how we've been accomplishing this right now is we've been developing a system we call SmartScope. 12:39.880 --> 12:42.740 So this system basically has two components. 12:43.260 --> 12:49.600 It attaches to a regular microscope, and over the eyepiece, we have a camera that just fits on the eyepiece. 12:49.600 --> 12:52.780 And so that'll fit up to an outer diameter of 44 millimeters. 12:52.860 --> 12:55.920 So that'll cover most eyepieces that are out there. 12:56.260 --> 13:05.880 And then we built this robotic slide holder, and it just attaches with two screws to the side of the stage of the microscope. 13:06.320 --> 13:18.040 So basically with these two pretty simple pieces of technology we've developed over the last three years, we can take a standard microscope, and then we can turn it into an IoT device. 13:18.430 --> 13:24.500 And so what this is is something that we can deploy at a cost of about $1,000 to $2,000. 13:24.900 --> 13:36.520 And on the other side, the companies that were previously building whole slide imaging machines and dynamic microscopes, they were normally charging between $3,000 and $100,000 for one of these standalone devices. 13:36.880 --> 13:46.900 And if you've worked in the developing world, you'll know that any sort of standalone device that they have, there's a good likelihood that it's going to be broken by the time you get back there the next time you come down. 13:46.900 --> 13:49.360 And there's no one that's going to service these devices. 13:49.540 --> 13:58.040 So we've actually created this technology really mimicking what we saw in 3D printers, and we've created technology that you can actually repair yourself. 13:58.440 --> 14:01.180 So I think that's a big change right now. 14:01.340 --> 14:08.500 And then you can share it on your... it runs on the cloud, so you can share it on your desktop, you can share it on your tablet, or you can even look at it on your smartphone. 14:10.140 --> 14:13.920 So how we built it up to this point was we used the Intel Edison. 14:14.280 --> 14:22.720 We thought that was a really good solution for us to start with, because it gave us a lot of versatility, we could run a whole bunch of programs on it. 14:22.720 --> 14:29.460 But the one drawback of working with the Edison was that it required us to have a computer to run everything with. 14:29.620 --> 14:32.740 So we couldn't sort of do things, everything on the device. 14:32.740 --> 14:41.720 And that was sort of where we're at, and we're actually deploying these units into the field, having sort of more and more use cases for it. 14:41.720 --> 14:45.820 So we have hospitals in Indiana that are using it for immunohistochemistry stains. 14:45.880 --> 14:49.380 They're just making sure, they're just using it for Q&A, for that kind of stuff. 14:49.820 --> 14:51.260 But it's been good. 14:51.260 --> 14:54.500 And so now we're looking at the next... what's next for us? 14:54.500 --> 14:55.780 What's next in the future? 14:56.240 --> 14:59.920 And so, I mean, I think this quote from Freeman Dyson is really good. 15:00.200 --> 15:04.000 New directions in science are launched by new tools more than they are by new concepts. 15:04.140 --> 15:06.940 The effect of concept-driven revolution is to explain old things in new ways. 15:06.940 --> 15:11.420 The effect of tool-driven revolution is to discover new things that have to be explained. 15:11.420 --> 15:21.400 So I think in a lot of ways, when Leeuwenhoek went on and developed a microscope that could see at 200 times magnification, then he could explain microbiology. 15:21.420 --> 15:33.340 For us, what we see in the vision is not only are we sort of working with this concept of people that are going to be providing healthcare workers, we're also giving them the tools they need. 15:33.380 --> 15:39.540 And we think these tools are powerful enough and cheap enough that they can be deployed in every single laboratory anywhere in the world. 15:39.540 --> 15:44.940 And we can create a whole global network, something that we like to call Skype for Microscopes. 15:45.000 --> 15:50.620 And this is how we're going to be able to track new diseases as new diseases emerge into the world. 15:50.620 --> 15:53.680 We're going to need to have these capabilities. 15:53.680 --> 15:55.860 We can't just send an expert there right away. 15:55.860 --> 15:58.220 There's time that it takes to transfer someone there. 15:58.340 --> 16:05.780 But having a sort of a network that's really light, we can create people around the world that can share this information with a centralized location. 16:06.060 --> 16:08.240 And so I think that's sort of where we're at. 16:08.240 --> 16:11.580 So I'll give you a little bit of a demo of how everything works. 16:12.120 --> 16:14.660 Let me see, just press the play button there. 16:15.740 --> 16:17.340 So this is our new website. 16:17.340 --> 16:19.200 It's launching later this month. 16:20.020 --> 16:26.640 Basically, once you sign in, you can access any devices that you have permissions to access. 16:27.120 --> 16:28.920 And so the devices are there. 16:29.300 --> 16:36.500 And then these are invitations, so you can receive invitations to join a session by someone else that has a device. 16:37.060 --> 16:42.180 And then once we have everything sort of running, we have this interface here. 16:42.180 --> 16:44.100 And you'll see the slide come up in a second. 16:44.660 --> 16:48.360 And so what we're able to do is we're able to control the movement of the slide. 16:48.480 --> 16:51.040 But we can also do things like add comments. 16:51.060 --> 16:53.860 So right now we're adding a comment to this box here. 16:54.100 --> 16:57.040 That could be sort of like anything you're seeing on the slide. 16:57.600 --> 17:00.080 You can submit that comment and that's recorded. 17:00.340 --> 17:03.100 We can invite people into the session, really simple. 17:03.100 --> 17:05.180 So we're going to invite Carlos right now. 17:05.180 --> 17:06.600 He's going to come in. 17:06.780 --> 17:07.960 We're going to invite him. 17:08.100 --> 17:10.620 You can see we have all the information available there. 17:11.040 --> 17:12.420 That's running on the local host. 17:13.000 --> 17:14.840 We have all the information on the slide there. 17:14.840 --> 17:17.720 And then we can slide that away and then you can view the whole slide. 17:18.420 --> 17:24.540 And so this is just a feed from a regular microscope, just from the eyepiece of the microscope. 17:25.160 --> 17:26.900 So obviously, I mean, there's some imperfections. 17:26.900 --> 17:28.300 You're seeing dirt and stuff like that. 17:28.300 --> 17:31.620 But this is what you would be seeing if you were looking through that eyepiece anyway. 17:31.620 --> 17:36.060 So it's a lot of, I mean, it's got clinical quality. 17:36.240 --> 17:38.020 It's getting the job done. 17:38.020 --> 17:43.940 And it really requires very little IT infrastructure to work in any way. 17:44.940 --> 17:47.280 So, yeah, I mean, now you're viewing the slide. 17:47.280 --> 17:53.000 And this is sort of an example of how we can look at different parts of the slide in order to determine a diagnosis. 17:53.380 --> 17:56.860 And then if we see something we like, we can take an image of that. 17:57.060 --> 17:59.100 We can add a comment to that image. 17:59.500 --> 18:02.900 And then once we add the comment to the image, we'll save that as well. 18:04.040 --> 18:07.240 And that just downloads to your computer, so you have a copy of it. 18:07.780 --> 18:13.140 And the idea, basically, is once you close out of the session, what you'll be able to do is create a report. 18:13.420 --> 18:18.580 And that report would just act as a single lab report, very similar to any sort of lab report you get from any hospital. 18:18.920 --> 18:23.680 It would tell you who was in the session, who started the session, what hospital the session took place at. 18:23.940 --> 18:26.780 And it would tell you if any sort of diagnosis was made. 18:26.780 --> 18:28.900 And then it would give thumbnails of any images. 18:29.300 --> 18:30.180 Okay, cool. 18:30.180 --> 18:31.860 So I've got about five minutes left. 18:33.460 --> 18:35.840 So, let's finish with this guy. 18:36.940 --> 18:38.960 And then we'll move on to the next slide. 18:39.200 --> 18:45.360 So, that's sort of where we're at in terms of how we see connecting laboratories around the world together. 18:45.600 --> 18:50.120 So let me talk to you a little bit about something that's called FlockMind, which I think is pretty interesting. 18:50.240 --> 18:51.900 So this was a study that was done. 18:51.900 --> 18:57.080 The doctor is like, some people don't like her, she's like sort of one of those figures. 18:57.200 --> 19:00.000 But she did this study basically where she took pigeons. 19:00.140 --> 19:05.360 And she trained the pigeons in order to make diagnosis on breast cancer using images from slides. 19:05.740 --> 19:11.620 So the pigeons themselves had an 85% diagnostic accuracy when determining cancer. 19:12.420 --> 19:19.940 But when they took the whole flock of pigeons, and then they actually took the result of the flock, they found that it actually raised up to 98%. 19:21.120 --> 19:22.880 So it was actually pretty good. 19:22.880 --> 19:27.880 But the thing they weren't telling you when I read the article was that they only used a training set. 19:27.880 --> 19:31.660 So they trained the pigeons on these images, and they just gave them the same training images again. 19:31.660 --> 19:34.680 They never gave them a testing set, so they sort of tried to hide that data. 19:34.680 --> 19:35.920 So I'm going to tell you that one. 19:35.920 --> 19:40.280 But it's still really interesting that with a training set, they could have the pigeons make their diagnosis. 19:41.340 --> 19:45.020 So that's essentially what we're trying to do right now with AI. 19:45.020 --> 19:52.520 So we're using TensorFlow, and we're basically creating our own methods of detecting cancer. 19:52.520 --> 19:55.280 So this is one in particular for cervical cancer. 19:55.280 --> 20:00.800 And we're doing screening programs where we can determine what level of cervical cancer that people are facing. 20:01.180 --> 20:04.020 And it's actually going really well. 20:04.020 --> 20:05.100 We're really happy. 20:05.100 --> 20:10.100 And we're able to build a lot of data because we have our own slide scanners, which is a big thing that holds a lot of people in line. 20:10.860 --> 20:14.280 And right now, we're working with Qualcomm, as I mentioned before. 20:14.280 --> 20:19.760 And the reason why we're really excited about that is because we're trying to actually create a new device altogether. 20:20.040 --> 20:23.500 This would be a standalone device that wouldn't need a computer or anything like that. 20:23.500 --> 20:27.120 And we really see this as the future of microscopy in a lot of ways. 20:27.300 --> 20:30.580 We can do everything that we were talking about before on this device. 20:30.580 --> 20:31.620 We could control our motors. 20:31.620 --> 20:32.980 We could do everything we want there. 20:33.020 --> 20:35.860 We could run our CV-based image stitching programs. 20:36.040 --> 20:37.720 And we can use TensorFlow and AI. 20:37.720 --> 20:45.220 And a lot of people are getting at us right now because they have been developing AI and they've been looking for platforms to run it on that are so cheap that they can be deployed into the developing world. 20:46.040 --> 20:48.200 And on the conclusion, that's what I'm going to say. 20:48.460 --> 20:49.480 It's all going to be good. 20:49.480 --> 20:51.800 We're all going to be alright because we're working on solving this problem. 20:52.160 --> 20:53.980 Thanks very much for your time today.