1
00:00:00,000 --> 00:00:04,200
We're going to get started right now, and it is my pleasure to introduce you to Ankur

2
00:00:04,200 --> 00:00:10,820
Tyagi. We're going to talk about visual network and file forensics. So, Ankur.

3
00:00:15,760 --> 00:00:21,680
Thanks a lot, everyone. Thanks for joining me. I will be talking about visual

4
00:00:21,680 --> 00:00:28,520
network and file forensics. Can you hear me? Can you hear me now? Yeah.

5
00:00:28,520 --> 00:00:35,360
All right. So the prime objective of this talk is to actually, you know, introduce you to

6
00:00:35,360 --> 00:00:40,280
visual forensics and how visualization can be very useful when you combine it with static and

7
00:00:40,280 --> 00:00:45,640
dynamic analysis of network content, which is, you know, just some binary block for you. You

8
00:00:45,640 --> 00:00:48,900
don't really know what exactly you are seeing on the network and you would like to understand

9
00:00:48,900 --> 00:00:53,000
what kind of traffic is being, you know, what kind of content is being transferred between

10
00:00:53,000 --> 00:00:57,120
client and the server. Or if you are trying to analyze a malware and you would like to

11
00:00:58,520 --> 00:01:02,700
see this malware can do, what kind of packer it has used or whether it is packed or not,

12
00:01:02,780 --> 00:01:06,140
whether you are looking at some encrypted content or whether you are looking at some

13
00:01:06,140 --> 00:01:10,600
compressed content. If you would like to do that, you can use visualization in

14
00:01:10,600 --> 00:01:14,720
combination with static and dynamic analysis. So that's the primary goal of the talk.

15
00:01:14,820 --> 00:01:21,140
We would like to, you know, see how visual analysis can complement static and dynamic

16
00:01:21,140 --> 00:01:28,200
analysis as well. All right. The tool will, so during the presentation I will

17
00:01:28,520 --> 00:01:36,060
also demo a framework, a tool that I have created. It can analyze binary blocks and it can

18
00:01:36,060 --> 00:01:41,100
right now specifically target two types of file formats. These are PCAP and PE files. So

19
00:01:41,100 --> 00:01:47,860
PCAP files are, PCAP files will, you know, help the tool to extract network content and then

20
00:01:47,860 --> 00:01:54,140
visualize it. For the PE files, which are the prime, you know, prime containers through which

21
00:01:54,140 --> 00:01:58,500
malwares are delivered on most of the Windows systems, you will see that, you know, the

22
00:01:58,520 --> 00:02:04,620
Windows file format, once you statically analyze it or, and in combination with the dynamic

23
00:02:04,620 --> 00:02:11,100
behavior report from sandboxes like Cuckoo and in case you have used Noriband, Noriband is

24
00:02:11,100 --> 00:02:16,340
again a sandbox, a minimal sandbox that you can use. Reports from those sandbox combined with

25
00:02:16,340 --> 00:02:21,760
the static analysis can complement, can be complemented with the visual analysis output that

26
00:02:21,760 --> 00:02:28,380
we will get. Okay. So primarily we will be targeting, we will be looking at the visual analysis

27
00:02:28,520 --> 00:02:32,980
in combination with structural properties. So these, these structural properties will be common

28
00:02:32,980 --> 00:02:38,660
for any type of a content. So let's say you have an example, you come across an incident wherein

29
00:02:39,180 --> 00:02:44,020
you have a PCAP file. Now you, you don't really know what, what actually happened during, you

30
00:02:44,020 --> 00:02:48,440
know, when the compromise happened on the system. All you have are some artifacts from the system

31
00:02:48,440 --> 00:02:55,700
and a PCAP of the actual event from one of the SIEMs or IDS devices. Now you would like to, you

32
00:02:55,700 --> 00:02:58,500
know, analyze these artifacts and to do that, you will need to have a PCAP file. You will need to

33
00:02:58,520 --> 00:03:04,540
have a, you know, look at the visualization techniques and use the demo. During the demo, we

34
00:03:04,540 --> 00:03:11,140
will see how exactly the tool can be useful in this. We will also look at the use cases. So

35
00:03:11,140 --> 00:03:17,240
primarily visualization in combination with static and dynamic can be a useful, you know,

36
00:03:17,240 --> 00:03:23,900
technique to classify content or to cluster it. But this has some, you know, gaps but definitely

37
00:03:23,900 --> 00:03:28,500
this can be a first level filter. And we will, you know, look at that. Okay. So we will be

38
00:03:29,800 --> 00:03:33,760
looking at a binary blob. So first of all, let's talk about, you know, the, the binary blob.

39
00:03:33,900 --> 00:03:38,980
So what, what we are looking at is a binary blob. We, we don't really know what exactly the content,

40
00:03:39,180 --> 00:03:46,720
what exactly the file contains. We will look at it from a black box standpoint. We will look at it

41
00:03:46,720 --> 00:03:51,780
in a sense that we don't, don't have any understanding of what kind of file we are looking at.

42
00:03:52,180 --> 00:03:57,760
Okay. So if we have a file about which we don't know anything, we would like to, you know, first of

43
00:03:58,520 --> 00:04:02,000
all, look at it and see what exactly the file might contain. We don't want to submit it to our

44
00:04:02,000 --> 00:04:07,700
sandbox because, you know, submitting to a sandbox and then waiting for the output to come is probably

45
00:04:07,700 --> 00:04:11,980
feasible for a few files. But you cannot do that when you get a, you know, when you get hundreds

46
00:04:11,980 --> 00:04:17,800
and thousands of files in, in a pipeline. And you get these files on a daily basis. You cannot do,

47
00:04:17,960 --> 00:04:22,120
you cannot just keep on submitting them to the sandbox and, you know, have some analysts look at

48
00:04:22,120 --> 00:04:26,700
the report of the sandbox and then decide whether these files are bad or good or, you know, whether

49
00:04:26,700 --> 00:04:28,500
they have to be submitted to other sandboxes.

50
00:04:28,520 --> 00:04:32,560
or other analysis probes. So visualization

51
00:04:32,560 --> 00:04:36,340
can be one of the techniques in which you can probably,

52
00:04:36,480 --> 00:04:40,640
you know, gather some insight on the file and then make a decision

53
00:04:40,640 --> 00:04:44,480
whether this file has to be actually submitted to a sandbox or, you know, even

54
00:04:44,480 --> 00:04:47,220
CPU intensive operation in the cycle.

55
00:04:48,880 --> 00:04:52,540
The structure of a blob can be visualized

56
00:04:52,540 --> 00:04:56,420
and this structure can be used then to classify content

57
00:04:56,420 --> 00:04:59,760
based on the file type. So you can look at the visualization

58
00:05:00,300 --> 00:05:03,960
and see whether a file has certain characteristics, certain,

59
00:05:04,100 --> 00:05:08,540
you know, standard blobs. And if these blobs

60
00:05:08,540 --> 00:05:12,300
are visually visible in multiple files, it would probably mean that

61
00:05:12,300 --> 00:05:16,340
most of those files are common. They have a common header structure or they have

62
00:05:16,340 --> 00:05:20,160
a common, you know, derivative. They are following a certain standard. All these files.

63
00:05:20,800 --> 00:05:24,300
So when this is used in combination with static and dynamic, this can

64
00:05:24,300 --> 00:05:28,180
be actually useful to classify the content quickly. You would not need

65
00:05:28,180 --> 00:05:32,340
to actually submit it to a sandbox or you would not need to, you know, actually

66
00:05:32,340 --> 00:05:36,140
ask your analyst to spend some time analyzing that file and making a decision.

67
00:05:36,360 --> 00:05:40,080
So that time can be used to do something else to, you know, actually analyze

68
00:05:40,080 --> 00:05:44,700
and reverse engineer a new malware sample that you don't know anything about.

69
00:05:44,880 --> 00:05:48,340
But to quickly classify files, you don't need to

70
00:05:48,340 --> 00:05:52,040
submit those to a malware analyst to do it manually. You can use

71
00:05:52,040 --> 00:05:56,240
visual in combination of static and dynamic to quickly classify

72
00:05:56,240 --> 00:05:59,760
the content. Okay? So

73
00:05:59,760 --> 00:06:04,020
to do all of this, basically, we will be seeing

74
00:06:04,020 --> 00:06:07,800
a framework that I created. It's called Rudra. There is again

75
00:06:07,800 --> 00:06:12,020
a tool that I created. It's called FlowInspect. So these tools can be

76
00:06:12,020 --> 00:06:15,820
useful for you when you are looking at certain file types. So for example, you have a

77
00:06:15,820 --> 00:06:20,120
PCAP file as input. FlowInspect can be useful. You can

78
00:06:20,120 --> 00:06:23,820
have your regular expressions or, you know, fuzzy strings matched

79
00:06:23,820 --> 00:06:28,080
upon the PCAP file. You can ask the tool to do that for you. You can ask

80
00:06:28,080 --> 00:06:31,960
the tool to extract network, you know, streams, reassemble them first of all,

81
00:06:32,000 --> 00:06:36,020
match the regular expression upon them, and then if you would like to save them to a

82
00:06:36,020 --> 00:06:40,080
new file, you can do that. It's basically a minimal idea that you can use

83
00:06:40,080 --> 00:06:43,960
without actually using, you know, really good tools. This is

84
00:06:43,960 --> 00:06:47,800
written in Python, so, you know, it exposes an API. It is a

85
00:06:47,800 --> 00:06:50,100
command line tool. You can, you know, use it to

86
00:06:50,120 --> 00:06:54,140
execute in your own tool chain. Rudra, similar to FlowInspect,

87
00:06:54,300 --> 00:06:58,040
has the capability to also analyze malware content. So by malware

88
00:06:58,040 --> 00:07:01,440
here I mean PE files, Windows PE file format. So

89
00:07:01,440 --> 00:07:05,840
if you have a PE file as input and you would like to understand what exactly the file has, you have

90
00:07:05,840 --> 00:07:10,060
hundreds of parsers available, right? But then Rudra can also be used

91
00:07:10,060 --> 00:07:14,120
in combination that you don't really need to use a third party

92
00:07:14,120 --> 00:07:18,080
parser and then add visualization heuristics on top of it.

93
00:07:18,080 --> 00:07:22,000
You can use Rudra's parsing capabilities and in combination with visual

94
00:07:22,000 --> 00:07:25,640
you get a combined report. So you have a tool which can do both these

95
00:07:26,920 --> 00:07:29,320
which can do both of these. So now the

96
00:07:29,920 --> 00:07:33,980
output that you get can be used for a, as an input to the

97
00:07:33,980 --> 00:07:37,600
heuristics engine. So the heuristics engine can take the output and, you know,

98
00:07:37,600 --> 00:07:42,160
come up with a scoring model and then say whether the output actually signifies

99
00:07:42,160 --> 00:07:46,000
that the file is bad or good. Okay? The tool has

100
00:07:46,000 --> 00:07:48,920
a plugin based architecture and it provides JSON reports. So now

101
00:07:49,680 --> 00:07:53,700
with the plugins what you can do is you can, based on the content or

102
00:07:54,620 --> 00:07:58,040
whatever analysis output you get, based on the output you can now write

103
00:07:58,040 --> 00:08:01,760
some rules. So the heuristics engine that you can write on your own will be

104
00:08:01,760 --> 00:08:05,680
able to make a decision whether the file is good or bad. So the tool will not do that for you.

105
00:08:05,760 --> 00:08:10,160
The tool will just tell you whether the file's content, it will extract the properties

106
00:08:10,160 --> 00:08:13,940
statistical properties, static properties based on the file type and dynamic

107
00:08:13,940 --> 00:08:17,900
behavior if it's a PE file. It will combine everything and give you a generic

108
00:08:17,900 --> 00:08:21,780
report. Now based on the report you can write your own plugins to actually

109
00:08:21,780 --> 00:08:25,780
classify the file as good or bad. In certain cases it might be the, it might

110
00:08:25,780 --> 00:08:29,960
happen that, you know, some certain combinations of these static fields

111
00:08:29,960 --> 00:08:32,820
and the dynamic behavior in combination with visualization

112
00:08:33,580 --> 00:08:37,860
is an indicator of bad activity or is a strong

113
00:08:37,860 --> 00:08:41,840
indication of a suspicious file. But this might not be the case for every file.

114
00:08:41,840 --> 00:08:45,640
It might not be the case for everyone, right? It might be a case that

115
00:08:45,640 --> 00:08:49,740
certain organizations they have a policy whether they would like to filter everything with really

116
00:08:49,740 --> 00:08:53,300
high entropy. And for someone else it might not be the case. So

117
00:08:53,300 --> 00:08:57,780
the tool will give you the feedback and then this feedback can be used to write

118
00:08:57,780 --> 00:09:01,440
your own plugins to decide whether something is good or bad. Okay?

119
00:09:04,120 --> 00:09:05,680
So the structural properties

120
00:09:05,680 --> 00:09:08,680
which are common for any type of input that a tool gets

121
00:09:08,680 --> 00:09:12,300
they are very, you know, they are like

122
00:09:12,300 --> 00:09:15,760
hashes. You get to, you know, calculate the unique and the

123
00:09:16,240 --> 00:09:20,260
common hashes. So by common I mean the similarity hashes like SSD.

124
00:09:20,600 --> 00:09:24,580
The tool will also calculate the entropy, compression ratio, min-size. So these

125
00:09:24,580 --> 00:09:27,760
are some structural properties which can be calculated for any type of

126
00:09:28,780 --> 00:09:32,640
content that is provided as input to the tool. Now this will be

127
00:09:32,640 --> 00:09:36,560
common for PKF files and PE files. When you visualize a PE file you see

128
00:09:36,560 --> 00:09:40,200
some interesting patterns and when you visualize a PKF file you see some interesting

129
00:09:40,200 --> 00:09:44,340
structures and we will have a look at them. It is very important

130
00:09:44,340 --> 00:09:47,980
to, you know, remember the fact that entropy, that the

131
00:09:48,380 --> 00:09:52,280
entropy and compression ratio, these are structural properties. So entropy is always

132
00:09:52,280 --> 00:09:56,440
indirectly proportional to compression ratio. So a file with very high entropy

133
00:09:56,440 --> 00:10:00,500
will always have a really low compression ratio. Now we will have

134
00:10:00,500 --> 00:10:04,020
a look at these properties when we see the report. Here is a

135
00:10:04,020 --> 00:10:07,560
snapshot of what exactly the file metadata looks like. So you have a file

136
00:10:07,560 --> 00:10:12,000
which is actually a PCAP of, so there is

137
00:10:12,000 --> 00:10:15,880
a Windows executable which is packed by a UPX packer and it is being transferred

138
00:10:15,880 --> 00:10:19,480
over network. Now this has been captured as part of a PCAP

139
00:10:19,820 --> 00:10:24,000
and this PCAP has a size of 160 bytes. So when I

140
00:10:24,000 --> 00:10:27,420
see this PCAP I don't really have a context of, you know, whether

141
00:10:27,420 --> 00:10:31,580
what kind of content is being transferred and what kind of packets are there

142
00:10:31,580 --> 00:10:34,000
in this particular PCAP. I would like to pass it by a wire

143
00:10:34,020 --> 00:10:37,440
shark. Wire shark will, you know, analyze the streams and let me know. If it has,

144
00:10:37,600 --> 00:10:41,960
you know, if the protocols are supported by wire shark, almost

145
00:10:41,960 --> 00:10:45,880
everything is supported by it. So it will let me know what all fields are there and it

146
00:10:45,880 --> 00:10:49,960
will tell me the details about the metadata or the headers of the packets.

147
00:10:50,260 --> 00:10:54,020
It will not be able to tell me what kind of content is there. It will not be able to

148
00:10:54,020 --> 00:10:57,960
tell me if the content is packed or not. In this case, what you are seeing

149
00:10:57,960 --> 00:11:01,860
here is a PCAP which has a really high entropy. Entropy is 7.6 and the

150
00:11:01,860 --> 00:11:04,000
compression ratio, like I said, will be very low.

151
00:11:04,020 --> 00:11:07,900
Because the entropy is high. It's 5.04. So if you have

152
00:11:07,900 --> 00:11:11,960
the best possible compression available, in this case, for this

153
00:11:11,960 --> 00:11:15,720
file you will only be able to reduce the size of the file by just 5%.

154
00:11:15,720 --> 00:11:19,840
Now this is because the file has really high entropy content. So most of the

155
00:11:19,840 --> 00:11:23,420
content is random. You cannot, you know, find patterns and minimize them.

156
00:11:23,560 --> 00:11:28,160
And that is why the min file size can only be reduced to 152 KB.

157
00:11:28,620 --> 00:11:32,000
Now these alone are not really useful facts

158
00:11:32,000 --> 00:11:36,000
but then when combined with, you know, dynamic analysis or visualization

159
00:11:36,000 --> 00:11:39,980
they can be used as part of a heuristics engine to classify

160
00:11:39,980 --> 00:11:41,800
certain content as good or bad.

161
00:11:45,020 --> 00:11:47,740
The most important fact here is to

162
00:11:47,740 --> 00:11:50,940
understand that, you know, static properties, statistical properties

163
00:11:50,940 --> 00:11:55,940
are useful when they are visualized. So there are two types of

164
00:11:55,940 --> 00:11:59,040
visualizations. So first one is basically just a histogram.

165
00:11:59,040 --> 00:12:03,060
What we do is we take the bytes and their frequency and graph it out.

166
00:12:03,180 --> 00:12:06,480
So this is just a fancy way of, you know, snapshotting a file without actually

167
00:12:07,180 --> 00:12:11,020
honoring the sequence of bytes. So this just tells you what kind of

168
00:12:11,020 --> 00:12:14,740
bytes are present and what is the, you know, frequency of each byte.

169
00:12:15,060 --> 00:12:19,160
This will give you an insight of whether the file has more non-printable

170
00:12:19,160 --> 00:12:23,180
binary characters, high ASCII characters, or low ASCII printable characters

171
00:12:23,180 --> 00:12:23,940
or control characters.

172
00:12:26,060 --> 00:12:27,920
And you can also visualize

173
00:12:27,920 --> 00:12:31,980
a file by creating a bitmap. Now this is a really interesting way of visualizing a file.

174
00:12:32,080 --> 00:12:36,060
So this can be done for any type of file, any type of network content, any type of

175
00:12:36,060 --> 00:12:39,860
binary blob. You can do this for executable

176
00:12:39,860 --> 00:12:43,680
formats, you can do this for data files as well, and you will see really interesting

177
00:12:43,680 --> 00:12:47,900
patterns evolving. For a file which has a fixed file structure

178
00:12:47,900 --> 00:12:50,860
you will always see patterns. So if you take

179
00:12:51,800 --> 00:12:54,680
let's say for example mp3 file format and

180
00:12:54,680 --> 00:12:58,660
you try to visualize all mp3 files, you will always see some part

181
00:12:58,660 --> 00:13:02,680
of the file to be common. So in the visualization output you will always see similarity

182
00:13:02,680 --> 00:13:06,520
because the file is following certain standard and that standard is common across

183
00:13:06,520 --> 00:13:10,420
all the files because they are all following the same standard, right? So in this case

184
00:13:10,420 --> 00:13:14,720
headers for all those files will more or less have a typical structure and that can be

185
00:13:14,720 --> 00:13:18,620
visualized. In this case what you are seeing is this is probably some

186
00:13:18,620 --> 00:13:22,500
binary content captured over network and because

187
00:13:22,500 --> 00:13:26,180
of all the bytes mixed here, so this particular representation is taken

188
00:13:26,180 --> 00:13:30,540
from a blog by Curtis Mattoon. Shout out to him, he

189
00:13:30,540 --> 00:13:34,560
has written a Python script that can actually

190
00:13:34,560 --> 00:13:38,620
take in any type of file and create such graphs for you. So the one

191
00:13:38,620 --> 00:13:42,720
on the left is in grayscale, one on the right takes the byte values and uses

192
00:13:42,720 --> 00:13:46,520
them for the HSV calculation. And this is a bitmap that you can

193
00:13:46,520 --> 00:13:50,540
create for any type of a file format. Similar, there is a tool

194
00:13:50,540 --> 00:13:54,260
called binviz. How many of you have used binviz by Elder

195
00:13:54,260 --> 00:13:58,540
courtesy? No one? Okay. You should probably look

196
00:13:58,540 --> 00:14:02,460
at it. It's a really awesome tool to visualize your content. You can use it

197
00:14:02,460 --> 00:14:05,520
to visualize any type of a file and it also gives you an

198
00:14:07,020 --> 00:14:09,640
interactive ID sort of format.

199
00:14:10,060 --> 00:14:14,600
It's a website. Basically you go there, you upload your file and it gives you an interactive

200
00:14:14,600 --> 00:14:18,600
webpage wherein you can select a certain section of your file

201
00:14:18,600 --> 00:14:22,600
and see whether that particular section of your file, if it has

202
00:14:22,600 --> 00:14:26,740
certain high entropy content or if it looks similar to a certain file format.

203
00:14:26,980 --> 00:14:30,540
You can do that. Shout out again to BS Manjunath.

204
00:14:31,460 --> 00:14:34,600
He is from UCSB. He and his team, they have

205
00:14:34,600 --> 00:14:38,200
created a tool called Sarvam. So the idea with

206
00:14:38,200 --> 00:14:42,360
that particular tool is to use these graphs, these images and

207
00:14:42,360 --> 00:14:46,160
create a model and whenever you have something new, match it against

208
00:14:46,160 --> 00:14:49,560
the models and if it matches, if the images they match, use image similarity

209
00:14:49,560 --> 00:14:54,300
algorithms and if they are similar, more or less they are coming from

210
00:14:54,300 --> 00:14:58,220
the same file or the malware samples or the binary content is more or less same.

211
00:14:58,500 --> 00:15:02,060
That is the idea behind Sarvam. So what I would like to

212
00:15:02,060 --> 00:15:06,260
highlight here is that you can use visualization in combination with static

213
00:15:06,260 --> 00:15:09,920
and dynamic analysis. Visualization is the most important part here.

214
00:15:11,000 --> 00:15:14,100
Visualization actually helps you to quickly look at the content and classify

215
00:15:14,100 --> 00:15:17,800
it. You have your own insight. You have looked at a lot of different

216
00:15:17,800 --> 00:15:21,940
file types and looking at it visually will give you an insight whether the file is good or not.

217
00:15:22,120 --> 00:15:26,080
Because you will see certain patterns and you will expect certain structures to be visible

218
00:15:26,080 --> 00:15:30,180
for a known file format and if they are not, you definitely know that something is wrong here.

219
00:15:30,240 --> 00:15:34,260
The file has been modified in some way or the file has been changed in some way

220
00:15:34,260 --> 00:15:38,280
and this is not expected. So visualization gives you an insight

221
00:15:38,280 --> 00:15:42,080
which is not typically available with static or dynamic analysis alone.

222
00:15:43,760 --> 00:15:46,240
So these are just some of the features that the tool has.

223
00:15:46,500 --> 00:15:49,700
It can not only do visualization

224
00:15:50,160 --> 00:15:54,040
or statistical analysis, it will also understand the type of the file format.

225
00:15:54,140 --> 00:15:58,100
Like I said, right now the tool understands PCAP and Windows

226
00:15:58,100 --> 00:16:02,160
executable PE file formats. So first of all it will try to

227
00:16:02,160 --> 00:16:06,040
identify the type of the file and then invoke the type

228
00:16:06,040 --> 00:16:09,780
specific parser. So if it is a PE file, it will use

229
00:16:09,780 --> 00:16:14,040
PE specific parser and extract certain fields. If it is a PCAP

230
00:16:14,040 --> 00:16:17,060
file, it will use a PCAP specific parser and extract those fields.

231
00:16:18,220 --> 00:16:21,980
For all types of files, there are certain generic attributes or static

232
00:16:21,980 --> 00:16:25,900
properties that are extracted and these are mentioned here. We would like to understand

233
00:16:25,900 --> 00:16:29,920
if a file has certain regular expressions matching upon it.

234
00:16:30,000 --> 00:16:34,120
If you would like to run your other rules upon the binary content,

235
00:16:34,260 --> 00:16:35,820
you can do that. If you would like to

236
00:16:36,040 --> 00:16:39,660
test whether the binary content is actually shellcode, you can do that.

237
00:16:39,780 --> 00:16:43,700
There are tools to do that. You can use those libraries to understand whether

238
00:16:43,700 --> 00:16:47,080
the content that you have is shellcode or if it is just some random

239
00:16:47,600 --> 00:16:49,040
binary content, binary block.

240
00:16:51,440 --> 00:16:55,780
Like I said, for PCAP file format, the tool will do TCP

241
00:16:55,780 --> 00:16:59,640
reassembly and IP defragmentation. It will extract the streams, it will identify

242
00:16:59,640 --> 00:17:03,860
protocols, it will decode protocol fields and then match your rules upon them.

243
00:17:03,860 --> 00:17:07,180
This is very much similar to what typical IDS will do for you.

244
00:17:07,240 --> 00:17:11,660
You can do this with Python alone. You can do this as a standalone tool. You don't need

245
00:17:11,660 --> 00:17:14,800
to deploy any CPU intensive

246
00:17:16,260 --> 00:17:19,720
tools that take a lot of time, that take some time

247
00:17:19,720 --> 00:17:23,160
to process large amount of packets and then create output for you.

248
00:17:23,260 --> 00:17:27,720
If you would like to quickly prototype your own tools for your specific use case,

249
00:17:27,780 --> 00:17:32,180
you can use a Python API rather than recompiling the whole toolset.

250
00:17:36,220 --> 00:17:38,240
If you have PE file formats,

251
00:17:38,520 --> 00:17:41,360
PE specific attributes will be calculated.

252
00:17:41,840 --> 00:17:45,860
These attributes are extracted by a lot of tools so there is nothing

253
00:17:45,860 --> 00:17:49,640
specific here but there is a very special way we use these attributes.

254
00:17:49,920 --> 00:17:53,600
We use them in combination with the dynamic as well as the

255
00:17:53,600 --> 00:17:56,780
statistical output, the visualization output that the tool generates.

256
00:17:57,080 --> 00:18:01,660
These properties when used in combination, they can be used as

257
00:18:01,660 --> 00:18:05,680
heuristics. So you can write your own rules. If a file has

258
00:18:05,680 --> 00:18:09,580
certain sections of certain names, section names

259
00:18:09,580 --> 00:18:13,320
can be changed easily. But when this is used in combination with certain

260
00:18:13,320 --> 00:18:17,660
visual characteristics, now this becomes a strong indicator. Someone can create

261
00:18:17,660 --> 00:18:21,260
a modified UPX packed PE file

262
00:18:21,260 --> 00:18:25,320
which will not have UPX typical section names that have

263
00:18:25,320 --> 00:18:29,480
UPX0 names in them. So whenever you pack a file with UPX, you will

264
00:18:29,480 --> 00:18:33,160
expect section names to be named as UPX0 and UPX1.

265
00:18:33,280 --> 00:18:36,320
But someone can change those names. It can be changed easily.

266
00:18:37,160 --> 00:18:41,240
But the statistical properties of the file which is

267
00:18:41,240 --> 00:18:45,160
packed by UPX will always remain the same. And that property is the file

268
00:18:45,160 --> 00:18:49,120
because it is packed will now have really high entropy. And the sections

269
00:18:49,120 --> 00:18:53,280
will not be very visible. And when you visualize it, you will not see structures.

270
00:18:53,460 --> 00:18:57,280
You will not see boundaries and patterns. You will not see the headers. And this is

271
00:18:57,280 --> 00:19:01,180
something that the authors or the guys who have created

272
00:19:01,180 --> 00:19:05,160
the malware who have packed it, they will not be able to change that as easily.

273
00:19:05,360 --> 00:19:09,180
So this in combination with the static analysis output can

274
00:19:09,180 --> 00:19:11,640
be a really good heuristics for you to classify files.

275
00:19:13,820 --> 00:19:16,240
Alright, so let's quickly move to the demo.

276
00:19:19,740 --> 00:19:21,300
Alright, so first of all

277
00:19:21,300 --> 00:19:24,800
let me show you a quick demo of how to classify

278
00:19:24,800 --> 00:19:28,580
how to understand whether a particular file has shellcode in it.

279
00:19:28,740 --> 00:19:32,420
So what we are trying to do here is we are using one of the tools called flow inspect.

280
00:19:35,060 --> 00:19:36,020
Can you guys see this?

281
00:19:36,100 --> 00:19:37,480
One second.

282
00:19:43,320 --> 00:19:43,740
Okay.

283
00:20:06,880 --> 00:20:07,320
.

284
00:20:09,160 --> 00:20:09,600
.

285
00:20:10,400 --> 00:20:10,840
.

286
00:20:13,800 --> 00:20:18,240
Okay, so first of all let's see. So this particular tool, flow inspect,

287
00:20:18,520 --> 00:20:22,780
it helps you to, it is more like ngrep. You can think of it as a python based

288
00:20:22,780 --> 00:20:26,800
ideas. What you can do is you can pass it a file, a pcap file specifically

289
00:20:26,800 --> 00:20:30,440
and you can ask it to look for certain regular expressions within them,

290
00:20:30,500 --> 00:20:34,720
within the pcap file. You can ask it to look for a fuzzy string. You can ask it

291
00:20:34,720 --> 00:20:38,640
to reassemble the streams and look for shellcode within those streams.

292
00:20:38,900 --> 00:20:42,820
So that is what the tool does. So what I am trying to do here is I am trying to

293
00:20:42,820 --> 00:20:45,800
run flow inspect upon this file.

294
00:20:47,400 --> 00:20:50,640
This file actually has shellcode. Now we will see what

295
00:20:50,640 --> 00:20:53,500
kind of shellcode this is. From the name you can guess this.

296
00:20:55,620 --> 00:20:57,380
So whenever there is a match,

297
00:20:58,780 --> 00:21:02,600
the tool will actually show you the streams that have, you know,

298
00:21:02,620 --> 00:21:06,600
so we are asking the tool from these parameters you can see. So hyphen

299
00:21:06,600 --> 00:21:10,560
m asks the tool to run the shellcode analysis engine. So once you

300
00:21:10,560 --> 00:21:14,400
do that, if it finds a match, it will show you the stream that matches.

301
00:21:16,180 --> 00:21:18,840
This is, so we found shellcode

302
00:21:18,840 --> 00:21:20,620
in the source code.

303
00:21:20,640 --> 00:21:24,500
This is the server to client traffic. So this particular TCP stream

304
00:21:25,220 --> 00:21:28,420
105.73, this is the client and it is talking to

305
00:21:28,420 --> 00:21:32,500
87.78.87 on port 80. This is

306
00:21:32,500 --> 00:21:35,500
HTTP traffic and on this particular stream,

307
00:21:36,160 --> 00:21:40,360
server has sent some shellcode to the client. Now this is what the tool has

308
00:21:40,360 --> 00:21:44,260
understood. We can probably look more into the pcap, extract

309
00:21:44,260 --> 00:21:48,320
the particular stream and then pass it on to some other tools if you would like

310
00:21:48,320 --> 00:21:52,260
and this would give you more insight. But, you know, if you would just like to

311
00:21:52,260 --> 00:21:56,360
quickly look what exactly the shellcode is, the tool can create a profile

312
00:21:56,360 --> 00:22:00,220
for you as well. You can have a look at the command line options. There are lots of

313
00:22:00,220 --> 00:22:00,440
them.

314
00:22:08,500 --> 00:22:12,020
There are a lot of options available. You can use these options

315
00:22:12,020 --> 00:22:15,940
as per your requirement. So if you would like to do shellcode matching, if you would like to do

316
00:22:15,940 --> 00:22:19,280
regular expression matching, or if you would like to, you know, do

317
00:22:20,420 --> 00:22:23,520
fuzzing matches, you can do that. For shellcode matches,

318
00:22:23,740 --> 00:22:27,780
if you would also like to disassemble the shellcode, or if you would also like to create the

319
00:22:27,780 --> 00:22:31,900
profile for the shellcode, what exactly the shellcode would do once it is deployed on the client,

320
00:22:32,060 --> 00:22:35,000
you can do that. And this is the output of the profile.

321
00:22:38,920 --> 00:22:39,360
So

322
00:22:39,940 --> 00:22:43,980
the pcap that we saw, the tool first of all identified that there is

323
00:22:43,980 --> 00:22:45,920
shellcode within it. And it also identified

324
00:22:45,940 --> 00:22:50,380
identified the type of the shellcode. So this is a shellcode which will actually invoke

325
00:22:51,020 --> 00:22:58,180
the following calls. So the load library A API call is invoked with the WS232 parameter.

326
00:22:58,440 --> 00:23:04,260
So this is a typical API call to load the Windows socket library. After that you see

327
00:23:04,260 --> 00:23:13,420
some socket API calls. The host attribute is 53.20 and the port is 4444. So quickly

328
00:23:13,420 --> 00:23:18,740
this, this particular shellcode will actually spawn a reverse TCP shell if it is executed

329
00:23:18,740 --> 00:23:24,200
on a Windows system. That is what the shellcode will try to do. Now this was identified from

330
00:23:24,200 --> 00:23:30,120
a PCAP. This is something that you can do statically. The tool has not run the PCAP

331
00:23:30,120 --> 00:23:36,800
through a sandbox. It has not extracted anything special. What it does is it reassembles the

332
00:23:36,800 --> 00:23:41,540
streams and then identifies the protocols and then runs them through the analysis engine.

333
00:23:41,720 --> 00:23:43,140
In this case it was shellcode.

334
00:23:43,420 --> 00:23:48,300
If the shellcode is found we generate a profile for that and once the profile is found we

335
00:23:48,300 --> 00:23:52,540
try to see whether this particular profile signifies something bad or good.

336
00:23:52,760 --> 00:23:52,920
Okay.

337
00:23:58,540 --> 00:24:01,680
All right. So now let's move on to the second demo.

338
00:24:34,300 --> 00:24:39,920
What we are trying to do here is we will run this particular tool on a Windows executable,

339
00:24:40,020 --> 00:24:43,200
on a Windows malware and see what report it generates.

340
00:24:43,380 --> 00:24:43,400
Okay.

341
00:24:43,420 --> 00:24:51,600
I have a few files that are actually suspicious. I have got them from some source.

342
00:24:53,060 --> 00:24:53,880
Here they are.

343
00:24:55,540 --> 00:24:58,680
Let's run this on F2.

344
00:25:01,480 --> 00:25:06,500
All right. So the tool will quickly analyze a parser file first of all and, you know,

345
00:25:06,540 --> 00:25:11,960
generate some heuristics for you. So here is the parsed output. It will run the other

346
00:25:11,960 --> 00:25:12,660
rules upon it.

347
00:25:12,660 --> 00:25:18,560
it will run them against the adobe malware classifier it will also try to you know look for

348
00:25:18,560 --> 00:25:23,860
look whether this particular file is part of the whitelist bloom filter so what what we have done

349
00:25:23,860 --> 00:25:29,320
here is we have created a whitelist and the whitelist is not just a static list of all the

350
00:25:29,320 --> 00:25:34,640
good hashes what we have done is we have taken hashes from mandiant and from nsrl these are some

351
00:25:34,640 --> 00:25:41,300
sources that give you a list of white hashes or the files which are commonly seen in uncompromised

352
00:25:41,300 --> 00:25:46,160
windows systems file file hashes for windows files and adobe and you know popular vendors

353
00:25:46,160 --> 00:25:51,260
so we have taken hashes from them and create a bloom filter out of it and if a file particularly

354
00:25:51,260 --> 00:25:55,600
matches the bloom filter we say that you know it is whitelisted it is part of the hash set

355
00:25:55,600 --> 00:26:02,020
which is coming from the white whitelist of files and as such the file is most probably good and

356
00:26:02,020 --> 00:26:06,160
there is always a possibility of certain you know false positives when you are using bloom filter

357
00:26:06,160 --> 00:26:11,100
because it is a minimized set of the actual uh the unflattened list of

358
00:26:11,100 --> 00:26:11,280
hashes so we have taken hashes from them and created a bloom filter out of it and if a file

359
00:26:11,280 --> 00:26:16,680
hashes but in this case what we will do is we will quickly classify them as white or black now

360
00:26:16,680 --> 00:26:21,420
this can be used as part of the scoring as well let's have a look at the web reports

361
00:26:23,800 --> 00:26:29,380
the tool actually generates certain reports for you you can use it as a command line uh tool or

362
00:26:29,380 --> 00:26:34,440
you can you know use the api import it as a python module or if you would just like to quickly look

363
00:26:34,440 --> 00:26:40,220
at the reports you can have a look at the reports uh the sdn ml reports that it creates here you can

364
00:26:40,220 --> 00:26:41,020
see that you know the

365
00:26:41,020 --> 00:26:43,960
uh shellcode reverse tcp file that we had

366
00:26:46,400 --> 00:26:51,820
shellcode reverse tcp file that we had uh here is the byte frequency graph and the visualization

367
00:26:52,400 --> 00:26:58,380
the bitmap for it uh some of this uh some of the fields are extracted and they are passed out and

368
00:26:58,380 --> 00:27:04,380
uh here we can see that the stc traffic which which was the http response sent by the server to

369
00:27:04,380 --> 00:27:10,980
the client actually has windows shellcode within it similar thing can be done for another type of file now

370
00:27:10,980 --> 00:27:16,860
this file has uh this file again has shellcode but this shellcode will spawn the calculator.exe

371
00:27:16,860 --> 00:27:25,040
so calc.exe program on the client okay so this is the report for a windows executable

372
00:27:25,040 --> 00:27:31,380
this is called uh this is the cmd.exe command exe clean file with unpacked unpacked uh just

373
00:27:31,380 --> 00:27:35,840
the last report so we will quickly wrap up after this report uh this is how the uh byte

374
00:27:40,980 --> 00:27:44,280
Now, what is happening here is that you can look at the reports.

375
00:27:44,640 --> 00:27:50,520
You know, the colors, they actually indicate the structures and the boundaries between these colors.

376
00:27:50,600 --> 00:27:52,720
So, you know, you see the color blocks.

377
00:27:53,220 --> 00:27:56,360
These blocks indicate that the file format has certain structure.

378
00:27:56,500 --> 00:27:58,640
It has a particular format that it is following.

379
00:28:00,200 --> 00:28:05,540
You can use the static attributes and in combination with visualization, use it to classify the file.

380
00:28:07,000 --> 00:28:07,660
All right.

381
00:28:14,180 --> 00:28:15,340
So, yeah, that's it.

382
00:28:15,440 --> 00:28:21,580
So, basically, the aim of the talk is to highlight the fact that, you know, visualization can be a good heuristics for you.

383
00:28:21,620 --> 00:28:25,040
You can use it in combination with static analysis and dynamic.

384
00:28:25,100 --> 00:28:29,880
It is not a replacement for static or dynamic analysis, but it can definitely be a complement.

385
00:28:30,100 --> 00:28:37,120
If you just use static or dynamic, you will not get as much insight as you can get in addition with the visual analysis.

386
00:28:37,920 --> 00:28:39,820
The tool is already available on GitHub.

387
00:28:39,980 --> 00:28:40,680
You can use it.

388
00:28:40,680 --> 00:28:41,440
You can extend it.

389
00:28:41,500 --> 00:28:43,160
You can submit your patches if you would like.

390
00:28:43,660 --> 00:28:45,480
Both the tools are open source.

391
00:28:45,680 --> 00:28:49,240
If you have any questions or if you have any feedback, feel free to get back.

392
00:28:49,460 --> 00:28:51,000
My handle is 7s3ram.

393
00:28:51,180 --> 00:28:52,680
I'm on Twitter and GitHub as well.

394
00:28:53,520 --> 00:28:54,260
And that's it.

395
00:28:54,360 --> 00:28:54,620
Thanks.

396
00:28:55,500 --> 00:28:55,860
Thanks.

397
00:28:56,060 --> 00:28:56,120
Thanks.

398
00:28:56,280 --> 00:28:56,640
Thanks.