1
00:00:00,000 --> 00:00:23,040
Go ahead and get started. So one of the problems that I run into a lot in OSINT is one of the most basic ones, which is essentially just finding some string, some information in a huge pile of data. It seems really simple. It seems like it shouldn't be a problem. And yet I keep running into it again and again. And I found this tool and I want to add it to all of your tool belts. It's a very simple one, but it's a very effective one for me at least. It's called Recall.

2
00:00:23,040 --> 00:00:28,980
So the short version of this talk is that there's a dude, that guy, who made a really cool thing called Recall.

3
00:00:29,060 --> 00:00:33,460
I bumped into it one day. I'll tell the story of that. And I think it's really interesting. It's really awesome.

4
00:00:33,580 --> 00:00:37,180
It was really useful for me. And I want the rest of you to be able to use it as well.

5
00:00:38,800 --> 00:00:46,160
So again, one of the big problems that I keep running into is we have all kinds of information, too much information, and I have a heck of a time digging through it.

6
00:00:46,220 --> 00:00:51,460
I ran into this quote while looking for something interesting to explain how it was and figured it was just relevant enough.

7
00:00:53,240 --> 00:01:02,580
So one of the things that happened recently was I submitted a talk to Source Boston where I was going to dig through a bunch of the NSA's leaked documents,

8
00:01:02,760 --> 00:01:10,120
look at a lot of the programs they had, and compare them to what really a normal person could do if they had a few million dollars and a lot more programmers laying around.

9
00:01:11,320 --> 00:01:15,440
Long story short, the NSA's stuff is not magic. It's an interesting talk. I had a good time.

10
00:01:15,600 --> 00:01:20,960
But I started by looking at this page by all the EFF's documents and going, oh, man.

11
00:01:21,080 --> 00:01:21,440
Now I've got to run it.

12
00:01:21,460 --> 00:01:23,420
I've got to write a talk. It's going to be a giant pain in the neck.

13
00:01:23,660 --> 00:01:32,860
And so I started by writing a little Python script that can pull down all of the EFF's released documents and ended up with this, with nearly a gigabyte of PDFs.

14
00:01:33,140 --> 00:01:35,620
Not really searchable. They weren't really OCR'd.

15
00:01:35,940 --> 00:01:39,100
And basically this is what happened to me at that point.

16
00:01:39,320 --> 00:01:49,980
Like, I've got a talk to deliver in about a couple weeks, and I'm looking at like 882 megabytes of PDFs and just blank face, fear, sweat.

17
00:01:50,140 --> 00:01:50,860
It was terrible.

18
00:01:52,800 --> 00:01:54,060
Until I bumped into this.

19
00:01:54,260 --> 00:01:57,440
And this was essentially the thing that saved my ass on that talk.

20
00:01:57,820 --> 00:01:59,980
And I think it can be really useful for you guys as well.

21
00:02:00,300 --> 00:02:02,420
It's essentially a desktop full search tool, like it says there.

22
00:02:02,540 --> 00:02:05,420
But it can do a heck of a lot more that they don't tell you right away.

23
00:02:06,360 --> 00:02:09,560
It's a huge understatement to say that all it does is full text search on the desktop.

24
00:02:10,560 --> 00:02:13,980
So to get started, the basic steps are really simple.

25
00:02:14,100 --> 00:02:14,980
You just install an open recall.

26
00:02:15,220 --> 00:02:15,940
You look at it.

27
00:02:16,000 --> 00:02:17,940
You think, wow, that is so many buttons.

28
00:02:18,200 --> 00:02:19,120
You look at the user manual.

29
00:02:19,660 --> 00:02:21,400
You realize the user manual is also huge.

30
00:02:21,460 --> 00:02:21,920
You should ignore it.

31
00:02:22,640 --> 00:02:27,540
And then realize that there are too many buttons for you to realize how it's going to work on your own.

32
00:02:27,660 --> 00:02:28,860
And end up reading the user manual.

33
00:02:29,060 --> 00:02:30,300
And I'm here so that you don't have to do that.

34
00:02:30,400 --> 00:02:32,840
Because it turns out there's only one really quick thing you have to do to get started.

35
00:02:33,060 --> 00:02:35,400
And from there, it's just all nice.

36
00:02:36,220 --> 00:02:38,120
So to get started, you actually need an index.

37
00:02:38,340 --> 00:02:39,180
They don't really tell you this.

38
00:02:39,240 --> 00:02:41,840
And it took a long time for me to figure out how the heck to get the PDFs in.

39
00:02:42,420 --> 00:02:47,300
Essentially what you've got to go do is set up a place that recall looks and builds what it calls an index.

40
00:02:47,480 --> 00:02:51,040
And the index is it gathering metadata so it can search the metadata quickly.

41
00:02:51,460 --> 00:02:57,180
And then direct you to the documents you need much faster than if it had to search through actually a gigabyte of PDFs every time.

42
00:02:58,000 --> 00:02:59,200
What this looks like is here.

43
00:02:59,320 --> 00:03:06,300
So if you go to the preferences tab and put in the top directories, it'll drop you right into a place where you can toss whatever directories you want into there.

44
00:03:06,640 --> 00:03:10,580
And the cool thing is that you can put almost anything in those directories.

45
00:03:10,680 --> 00:03:12,520
And recall will eat it up and let you search through it.

46
00:03:12,700 --> 00:03:15,260
So in my case, it was a ton of PDFs.

47
00:03:15,300 --> 00:03:18,160
But in your case, it can probably be zip files.

48
00:03:18,300 --> 00:03:19,380
It will transparently open those.

49
00:03:19,540 --> 00:03:20,140
It can be mail.

50
00:03:20,360 --> 00:03:21,440
It can be PDFs.

51
00:03:21,460 --> 00:03:21,720
It can be PDFs as before.

52
00:03:21,980 --> 00:03:25,880
And we'll even get to how you can save web pages and search through those as well later on.

53
00:03:27,100 --> 00:03:30,760
So the short version of this, if you're interested in looking at the talk, you can check that out.

54
00:03:31,040 --> 00:03:32,000
The wrap is unfortunate.

55
00:03:32,880 --> 00:03:42,000
But if you point it at about a gigabyte of PDFs and then have it search through them, recall itself, again, if you go to the preferences tab and set up the index here.

56
00:03:42,240 --> 00:03:44,320
I had it pre-indexed because it takes a couple minutes.

57
00:03:44,860 --> 00:03:46,360
So it's already set up and ready to roll.

58
00:03:46,560 --> 00:03:49,120
But the short version is you've got a gigabyte of PDFs.

59
00:03:49,140 --> 00:03:50,780
You're not sure how to search through them.

60
00:03:51,460 --> 00:03:53,940
And instead, you can just put Prism in, and here they are.

61
00:03:54,560 --> 00:03:59,940
The really interesting thing is that you can go ahead and open them, and it will have already set up the search for you in here.

62
00:04:00,140 --> 00:04:02,900
So as you poke through, it can just drop you to whichever.

63
00:04:04,040 --> 00:04:06,040
Again, this isn't like a revolutionary thing.

64
00:04:06,140 --> 00:04:08,740
This is not something that's going to ridiculously change lives.

65
00:04:08,860 --> 00:04:09,540
You're searching.

66
00:04:09,900 --> 00:04:14,420
But, boy, did it save me a lot of time when I had nearly a gigabyte of PDFs to go through.

67
00:04:14,600 --> 00:04:16,260
And that carries over to other things as well.

68
00:04:16,320 --> 00:04:20,060
If you're trying to search through mail, if you're trying to search through zip files, whatever you're looking for,

69
00:04:20,200 --> 00:04:21,440
this can basically eat it up.

70
00:04:21,460 --> 00:04:22,860
And spit out search results for you.

71
00:04:24,660 --> 00:04:27,300
So on top of this, that's sort of the basic functionality, right?

72
00:04:27,380 --> 00:04:30,040
That's the really simple, you've got some PDFs, you need to search through them.

73
00:04:30,220 --> 00:04:32,800
But there's a heck of a lot built on top of this that I think is really interesting.

74
00:04:33,400 --> 00:04:41,080
So first off, Firefox actually has an add-on built by the same person that lets you go in and save the Firefox pages that you visit.

75
00:04:41,460 --> 00:04:43,600
And then be able to search through those full text as well.

76
00:04:44,080 --> 00:04:47,740
So if you've looked at something like Hunchly, this is basically the exact same thing.

77
00:04:48,120 --> 00:04:50,580
Except it requires a lot more fiddling to work, but it's free.

78
00:04:50,680 --> 00:04:51,440
So it's sort of up to you.

79
00:04:51,460 --> 00:04:56,000
It's up to you on whether you're looking to spend money for a better experience or whether you're looking to use something open source.

80
00:04:57,000 --> 00:05:00,680
This is actually incredibly useful if you're digging through a lot of web pages.

81
00:05:00,880 --> 00:05:06,100
And then later on down the line, you're like, oh yeah, I think there was some mention of this at some point, like a few days ago.

82
00:05:06,400 --> 00:05:07,660
But maybe it's not there anymore.

83
00:05:07,780 --> 00:05:08,860
Maybe it's been taken down since.

84
00:05:09,080 --> 00:05:09,400
Who knows?

85
00:05:09,880 --> 00:05:14,220
This will save a local copy of all of them and index them automatically so you can search through them.

86
00:05:14,300 --> 00:05:14,920
It's really interesting.

87
00:05:15,380 --> 00:05:18,180
The setup for this is kind of a pain in the neck.

88
00:05:18,720 --> 00:05:19,940
It took me a while to figure it out.

89
00:05:20,000 --> 00:05:21,100
But once you figure it out, you're like, oh.

90
00:05:21,780 --> 00:05:22,500
Oh, okay.

91
00:05:22,680 --> 00:05:23,120
All right.

92
00:05:23,160 --> 00:05:23,520
That was it.

93
00:05:23,880 --> 00:05:25,660
So essentially, it's not really documented.

94
00:05:25,900 --> 00:05:28,640
But that is where the Firefox plug-in saves all its stuff to.

95
00:05:28,860 --> 00:05:34,620
And once you dig through and find that directory, from there, you basically add the index the same way as you did before.

96
00:05:34,920 --> 00:05:43,960
But then it will still not work and you'll be confused until you do the last thing where you go to preferences and then set up the web history and have it cache through the web histories there as well.

97
00:05:44,320 --> 00:05:46,540
So I'll put up a blog post with these instructions here.

98
00:05:46,660 --> 00:05:47,380
Or you can take a picture now.

99
00:05:47,500 --> 00:05:49,720
But it's kind of a pain in the neck to dig through and find this on its own.

100
00:05:50,600 --> 00:05:51,380
And you've seen.

101
00:05:51,460 --> 00:05:54,460
You've seen the index before, but this is the new thing as well.

102
00:05:54,660 --> 00:05:59,940
You've got to go over to the web history and add in that same directory there and set some space for it to store stuff.

103
00:06:03,320 --> 00:06:10,840
So then the way to actually save pages is really interesting because you can go in and you can either index pages manually, where you index this page or always index this site.

104
00:06:11,080 --> 00:06:15,240
Or you can set just a regex and say everything on this domain, I want to be saved.

105
00:06:15,380 --> 00:06:19,840
Or everything where this word or this regex appears, I want that to be saved.

106
00:06:19,840 --> 00:06:26,500
And so it's an interesting way that you can go through and only save the things that are actually relevant to whatever you're working on at the time, so you have less to search through.

107
00:06:27,180 --> 00:06:33,360
Finally, if you go to file and rebuild index, that will clear out everything you had there before and completely rebuild the index from scratch.

108
00:06:33,540 --> 00:06:35,060
If you update it, it'll just add it in.

109
00:06:35,120 --> 00:06:37,260
If you rebuild, it'll wipe it out and restart it.

110
00:06:37,480 --> 00:06:41,020
So depending on what you're looking for, that has tripped me up a couple of times.

111
00:06:41,300 --> 00:06:42,040
You've got to be careful.

112
00:06:43,320 --> 00:06:49,820
So things that are neat with this especially is that it still has the full text search, which includes through all of the code, all the stuff that wouldn't necessarily display.

113
00:06:51,040 --> 00:06:56,180
There's also a copy saved locally, which is really, really useful if you're looking at stuff that's ephemeral and has a tendency to disappear.

114
00:06:56,460 --> 00:06:57,680
And you don't really do anything.

115
00:06:57,980 --> 00:07:08,080
So if you set up a whole domain to be indexed, then all you've got to do is just browse through it, look through stuff, make sure you click all the links, and you'll end up with a full text searchable database that's useful for whatever you're looking for.

116
00:07:09,120 --> 00:07:10,700
So again, this probably reminds you of Hunchly.

117
00:07:11,040 --> 00:07:16,900
It's basically an open source version of the same thing that I just accidentally stumbled across one day that I have never heard of anyone using before.

118
00:07:17,300 --> 00:07:19,800
I don't know if this is something that is commonly known anymore.

119
00:07:19,820 --> 00:07:20,360
It was just me.

120
00:07:20,440 --> 00:07:22,200
But I figured I should introduce it to you guys.

121
00:07:22,560 --> 00:07:25,160
Again, Hunchly is almost certainly way better.

122
00:07:25,320 --> 00:07:26,280
There's kind of a pain in the neck.

123
00:07:26,340 --> 00:07:27,360
It crashes from time to time.

124
00:07:27,460 --> 00:07:28,040
But it is free.

125
00:07:29,620 --> 00:07:33,640
But the cool thing is this is still not all that Recall can do.

126
00:07:34,820 --> 00:07:35,940
Again, it'll eat anything.

127
00:07:36,100 --> 00:07:44,520
It has a ton of little features under the hood that seem not particularly interesting until one day you need to open up, like, k-word files, and then you'll be very happy.

128
00:07:45,560 --> 00:07:48,220
So again, you can throw just about anything into that index.

129
00:07:48,380 --> 00:07:49,800
It'll be able to eat it up and let you full text search it.

130
00:07:50,760 --> 00:08:03,200
There's also a web UI, which if you're sharing documents among multiple people, perhaps this is something that's interesting to you to be able to actually do that search over the network and have multiple people collaborate over the same system.

131
00:08:03,860 --> 00:08:07,040
As well, the reason the web UI works at all is that it's in Python.

132
00:08:07,380 --> 00:08:09,360
There's a Python binding underneath all of this.

133
00:08:09,500 --> 00:08:13,660
And so if you're looking to script through things, the SDK for this is actually pretty interesting.

134
00:08:13,820 --> 00:08:15,040
Not that difficult to learn.

135
00:08:15,140 --> 00:08:16,520
There's only, like, six or seven functions.

136
00:08:16,840 --> 00:08:19,800
And you can query into them and pull out whatever information you need.

137
00:08:19,820 --> 00:08:21,880
Again, full text search in Python.

138
00:08:22,240 --> 00:08:31,760
Getting recall set up and calling it from Python was actually easier and more effective than using Python's PDF libraries to open up those PDFs and try to pull information out of them.

139
00:08:32,140 --> 00:08:38,540
Like, it was actually faster to set this up than use Python's own built-in PDF stuff, which was shocking and awesome.

140
00:08:39,540 --> 00:08:42,280
So the next time you need to search, I hope that this is something that's in your tool belt.

141
00:08:42,540 --> 00:08:44,220
It's really simple to set up if you're on Linux.

142
00:08:44,360 --> 00:08:46,080
And there's a Microsoft Windows version as well.

143
00:08:46,200 --> 00:08:49,620
I haven't tried it, but it's probably even easier to set up than Linux, because of course.

144
00:08:50,780 --> 00:08:55,200
And it's something that can eat almost anything, give you a full text search, and it's worth taking a look at.

145
00:08:55,860 --> 00:09:03,540
So you guys can follow me on Twitter, and I'll put out the blog post with the instructions and slides from this talk probably on Tuesday when I have recovered.

146
00:09:04,340 --> 00:09:07,720
So yeah, long story short, I hope this is in your tool belt next time you need to search through something.

147
00:09:07,940 --> 00:09:09,580
And if you have any questions, just let me know.

148
00:09:21,260 --> 00:09:27,700
So the web UI, I think if you access it over the network, I don't think you can collaborate as far as, like, multiple people typing at the same time.

149
00:09:27,780 --> 00:09:30,800
But you could both access the web UI with the same underlying documents.

150
00:09:31,160 --> 00:09:40,200
Which, if you're trying to control documents or something, I think would be an easy way to give multiple people access without having to copy, you know, a terabyte of documents over to multiple machines.

151
00:09:45,660 --> 00:09:47,840
I believe so.

152
00:09:47,840 --> 00:09:53,180
So basically what you have through this web UI doesn't look like you can actually go through and fiddle with stuff directly.

153
00:09:53,920 --> 00:09:56,960
I'm sure, you know, once you downloaded these, you could do what you wanted with them.

154
00:09:57,100 --> 00:10:01,460
But I don't see any way in here, nor did I find one, to go in and change documents.

155
00:10:02,440 --> 00:10:02,840
Yep.

156
00:10:03,740 --> 00:10:04,140
.

157
00:10:08,480 --> 00:10:14,660
So it comes with a whole bunch of different OCR utilities, or, like, various text extraction utilities, including OCR.

158
00:10:15,960 --> 00:10:17,660
I think the way that I did the...

159
00:10:17,660 --> 00:10:17,820
.

160
00:10:17,840 --> 00:10:20,380
I think the way that I did the piping with that was through Python.

161
00:10:20,800 --> 00:10:23,880
Although I'm not sure if there's a way directly on the command line to do that.

162
00:10:24,020 --> 00:10:28,320
I know some of the things, like the indexer, can be run from the command line instead of being run from the GUI.

163
00:10:28,440 --> 00:10:33,100
I'm not sure if there's also a command line pipe argument that lets you access the stuff in it.

164
00:10:33,140 --> 00:10:34,680
Or if you'd have to just write a little Python script.

165
00:10:34,980 --> 00:10:39,880
Again, probably four or five lines of Python that you can run on the command line and pipe that out through there.

166
00:10:40,580 --> 00:10:40,940
.

167
00:10:41,480 --> 00:10:41,840
.

168
00:10:45,040 --> 00:10:45,800
That's a good question.

169
00:10:46,020 --> 00:10:47,100
I didn't try that.

170
00:10:49,380 --> 00:10:50,720
There's probably a way.

171
00:10:50,840 --> 00:10:52,600
I have no idea what it would be.

172
00:10:55,780 --> 00:10:56,300
Cool.

173
00:10:56,460 --> 00:10:57,240
Thanks, everybody.

174
00:10:58,080 --> 00:10:58,600
.

175
00:10:58,600 --> 00:10:58,980
.

176
00:11:00,840 --> 00:11:01,360
.

177
00:11:01,360 --> 00:11:01,480
.

178
00:11:01,480 --> 00:11:01,740
.

179
00:11:01,740 --> 00:11:02,040
.

180
00:11:02,040 --> 00:11:02,120
.