start	end	text
2400	11200	All right. DEF CON 13 is proud to present Marshall Betto with the Protocol Informatics Project.
13400	18100	Saturday morning at DEF CON. God damn, why am I working Saturday morning at DEF CON?
18160	20480	All right. Marshall, take it away.
26520	29760	All right, so this project is called the Protocol Informatics Project.
29760	34860	It's basically a play on words from bioinformatics, which is kind of my main inspiration for this project.
37200	41700	Yeah, it's not that loud. Should I talk closer? Okay, there we go. Thanks.
42420	46700	So, my name is Marshall Betto. I'm a research scientist at McAfee.
46700	51220	However, this is pretty much all independent research. My company's never even seen this stuff.
51420	57140	So, I think that's why it's worthy, as opposed to something that's out there trying to make money or something like that.
58600	59920	Let's start.
60460	64600	Well, the purpose of this project is to basically automate network protocol analysis.
64600	69700	So, I just added this slide the other night just to kind of say, this is what this thing does.
69700	73920	You input a PCAP file, and output is going to be a packet format.
73920	81080	So, this is my little XML markup of this particular protocol format, which is just kind of something that's made up,
81080	86840	which is basically a two-byte link field followed by an ASCII string or something like that.
88660	92800	My application, what it does is it goes through, does a series of comparisons and checks,
92800	98200	and uses sequence alignment algorithms to understand the general structure of the format.
98200	105120	So, if you were to analyze this markup, basically what it's doing is it's describing this protocol,
105120	110100	saying that it operates on UDP 435364, something I just pretty much made up.
110640	116980	It's a block-based protocol, so you open up a block, you insert the link to that particular block,
116980	122240	which is going to be a short, big in the end, followed by a variable type,
122240	124260	which is going to be a plain text string.
124260	126680	Right now, the default value for that is going to be hello.
127040	128240	And then I close it all out.
128240	132920	So, the general idea is, you take a protocol you've never seen before in your life,
132920	136300	you're trying to reverse engineer a World of Warcraft, for all I know.
136480	140360	You input the PCAP file, and out comes the structure.
140360	142020	So, that's the general idea behind it.
143180	150200	So, the objective of protocol analysis is, first, to understand exactly the format of every single packet.
150200	156820	I mean, that's a pretty simple concept in general, but it does get more complicated as you have to understand
156820	160540	the actual state and the actual handshake and things of that sort.
160540	165060	So, there's really no good way of doing this at this current time.
165060	166700	I mean, it's pretty much a manual process.
166700	169860	People sit down with E3O or something like that and break out a piece of paper
169860	173180	and just start writing down offsets and trying to use their human intuition
173180	179320	to be able to understand what this particular field does and where it's located in the packet.
180000	184080	So, this information is generally important to understand proprietary protocols,
184080	187520	which are actually kind of becoming a little bit more rare, in my opinion.
187520	190160	Of all the security products that I've been looking at and things like that,
190160	192600	everybody seems to be using SOAP over SSL nowadays.
192720	197240	So, maybe this stuff isn't as applicable in those situations.
197240	202200	But, if you're going back and looking at legacy protocols, like some SCADA networks and things like that,
202200	206180	this stuff is going to be a lot more useful because there's really not that much documentation
206180	207440	for that sort of stuff.
207440	211260	So, it's also important for extending the functionality of software.
211260	217500	If you guys want to write an open source client for a particular service,
217500	221040	this project can help out in that regard.
221400	226060	And lastly, and my favorite, is finding vulnerabilities in unknown or badly documented protocols
226060	226980	using fuzzing.
226980	232200	So, what I basically do is I feed it my PCAP file, it analyzes the structure,
232200	237400	pops out the XML markup, then I feed that into my fuzzer and it starts going to work.
238160	241860	So, what protocol informatics does, which is just the name of my program basically,
241860	244260	it heuristically identifies the fields in data format.
244260	249640	So, it's never going to be perfect, but it's as close to automated as you're ever going to get.
251180	256800	So, nowadays the problems with protocol analysis in general is it's a manual process.
257840	261840	Like I was mentioning earlier, you just pretty much do it by hand using ethereal
261840	265200	and fields are identified largely using human intuition.
265200	269080	So, I guess I was talking to Spoon about this a little earlier
269080	273100	and he was basically explaining to me that, well, all these protocols are written by humans
273100	277780	so you can basically use your human intuition to be able to understand how they were thinking
277780	279020	and things like that.
279020	283720	However, computers have never really been able to mimic human behavior that well.
283720	286360	I mean, if you're looking at pattern recognition and things like that,
286360	292020	people are moving over neural networks and other hidden markup models and other things of that sort
292020	298200	in order to try to mimic that intuition in a computer program.
299880	303960	I guess the main challenges with protocol analysis these days is
303960	309040	kind of just understanding general structure when it comes to different types of protocols.
309400	314800	For example, a protocol that's all binary, usually quite hard to analyze
314800	318000	because there's really no points of reference for us.
318000	320320	When you're going through it, you're just kind of looking at the bytes
320320	322740	and trying to find patterns and things like that.
323260	329520	Typically, it doesn't seem like that's going to be pretty much a tedious, painstaking process.
330080	332280	So, this is really an attempt to automate that.
333960	338640	I think the crux of protocol analysis in general is variable length fields,
339820	342340	long strings in the middle of packets and stuff like that.
342340	347280	You don't necessarily know, if it's not plain text data, where it starts and where it ends
347280	349980	and what each byte represents and things of that sort.
352700	355220	Typically, this is all going to be differential analysis.
355220	358480	You're just going to be going through, doing comparison, comparison, comparison,
358480	361420	trying to kind of build a profile as you go along.
362440	366400	Fortunately, however, biologists already fixed this problem back in the 1970s
366400	371220	when they were trying to analyze genetic structure, DNA, amino acids, and all that sort of stuff.
371260	375620	This is all pretty much... I'm not taking credit for any of these algorithms I'm presenting today.
375720	379160	All I'm basically doing is reapplying it to our realm,
379160	381960	which is computer security, computer science, and things like that.
383020	385460	So, bioinformatics. What is bioinformatics?
385860	388600	It's basically information theory applied to biology.
388600	393840	So, it's a lot of probability, Shannon entropy, all sorts of things like that.
393840	401080	And it's mainly used to identify genes in the genetic structure of organisms.
401080	407440	So, Human Genome Project, the majority of the algorithms that are used are mainly sequence alignment algorithms.
407440	412220	What they're going out there and trying to find is where the beginning of these genes start and where they end,
412220	415440	what the function of these genes are, and what the structure of these genes are,
415440	417400	how they fold, and all sorts of things like that.
417400	426500	So, the workflow for how biologists do this, they have these large databases just full of genetic data.
427080	433060	The human genome's in there, they have mice and every other species that they've sequenced.
433320	436320	And they put this all in a database and then they can start running queries on it.
436320	441320	So, saying, like, I know that this gene exists in, you know, human.
441520	446360	It will go through and it can understand common ancestors and things like that,
446360	449640	based on mutation rates and all sorts of stuff like that.
449640	453760	So, if you want to put it in simple computational terms,
453760	457160	it basically is just processing a large amount of structured but complex data.
457160	460060	It's just too much for the human eye to just recognize.
460060	463160	So, you have to put it in a database, you have to run these algorithms on it and things like that,
463160	465460	if you really want to understand the structure.
468100	472100	So, yeah, the objective is to identify genes, determine structure, and the function.
473280	478640	Easier said than done, naturally, but there's a lot of analogies to protocol analysis in this case.
478640	485220	So, one application of bioinformatics is allowing biologists to map phenotypes to genotypes.
485220	489180	Being able to say, blue eyes maps of this particular gene,
489180	492320	attached earlobes maps of this particular gene, and things like that.
496560	499860	Protocol analysis and bioinformatics have a lot of things in common.
499860	502120	They're both operating in large sequences of data.
502360	508160	And my favorite is just the analogy of biology's phenotype-to-genotype mapping
508160	511660	is equivalent to protocol analysis' mapping of offsets to field locations.
511660	517860	We're basically just trying to understand the function of a particular field in a byte stream.
517860	521520	We don't know where it exists, but it's between this offset and this offset.
521520	528920	Fortunately for us, what actually makes this easier is, when it comes to genes and things like that,
528920	533500	it's not all... it doesn't have a defined start and a defined end.
533720	538080	You can have... I think eye color, for instance, isn't just like one single gene.
538080	540120	It's spread out throughout the genome.
540120	545100	So, you have to do a bit more complicated analysis to understand,
545100	549740	here's a block where it starts, then it skips this many symbols, and then it restarts over here.
549740	551860	So, fortunately for us, it's really not like that.
551860	555800	We have fields. Here's your start offset, here's your end offset.
556660	563860	So, what I did was I created an application that was meant to understand the structured, complex data found in our protocols.
565660	570920	So, the core algorithms that are used in bioinformatics in general are mainly sequence alignment algorithms.
570920	577100	There's two of the most classic algorithms, the Niederman-Wunsch algorithm and the Smith-Waterman.
577320	581020	I'll get into each particular detail as we go on, but basically,
581020	584080	the Niederman-Wunsch algorithm is a global sequence alignment algorithm.
584080	592780	It takes into account every single symbol in the genetic sequence and tries to align it to a different sequence.
592780	594540	It's just a pairwise alignment.
594780	598340	Smith-Waterman, on the other hand, is a local sequence alignment algorithm.
598340	604320	It's going to find the most common substring within the two sequences that are most similar.
606500	610840	There's ways to improve sequence alignment, and that's basically by using similarity matrices.
610840	612120	I'll get into that a little later.
612420	617180	All these algorithms are operations on matrices.
618260	621660	There's a lot you can do with different weighting and things like that,
621660	629300	and assigning scores, and how the matrix is traversed is all dependent on the scores in the cells.
629360	634040	There's actually a lot of really neat things that they figured out that they can do using this.
635360	637660	Then what we get into is phylogenetic tree building.
637660	644020	This is kind of cool, because this is how biologists are able to build a tree of life,
644020	648580	not based on fossil record, but instead based on genetic information.
648580	653220	How similar are these two sequences, and being able to go all the way up through the evolutionary tree
653220	655560	to find common ancestors and things like that.
656060	660940	I go over the two tree building algorithms, Upkema and the neighbor joining tree,
660940	664080	which I'm sure many people have heard of that.
664080	666520	And lastly, I get into multiple sequence alignment,
666520	671120	and that's where there's a lot of challenges involved with aligning more than one sequence,
671120	673200	just due to computational problems.
673300	676940	I go over the progressive and tree-based multiple sequence alignment algorithms.
685630	691010	And this is a lot of information, so if I am going too fast, just put your hands up or something like that.
692870	696590	The fundamental purpose of sequence alignment is really simple.
696590	699310	It's take two sequences, regardless of their length,
699310	702450	and insert gaps into either sequence to make them the same length.
703370	705730	So, right here I have an example of a DNA alignment.
706270	708070	There's four base amino acids in DNA.
708070	710450	There's thymine, guanine, cytosine, and adenine.
711030	715650	This is kind of just a really simple example of inserting gaps into the first sequence
715650	717530	to align it to the second sequence.
717530	726350	So, as you can see, there's three gaps that were actually inserted into the first sequence,
726350	731350	which would represent a deletion, evolutionary-wise.
732030	734970	The Niedermann-Wunsch algorithm, which I mentioned earlier,
734970	738890	is a dynamic programming algorithm, meaning that it operates on matrices.
741070	747010	I think it was the first sequence alignment algorithm ever written.
747010	750270	I think it was 1973 when it first came out.
751210	754210	Basically, it performs a global alignment on a pair of sequences,
754210	759270	and what that means is that every symbol in the sequence that goes into the algorithm
759790	761330	comes out of the algorithm.
761330	762290	Gaps are inserted.
762290	764130	There's no substring searching or anything like that.
764130	767470	It's going to try to insert gaps anywhere it can
767470	770390	to make the sequences line up to each other as best as possible
770390	774430	to maximize the alignment, maximize the score of the alignment.
775110	777810	And it's mainly used for analyzing closely related structures,
777810	781790	and the reason for that is if you were trying to analyze the human genome and the mouse genome
781790	785490	and using a global sequence alignment algorithm,
785490	790030	there's going to be so many gaps that are inserted that it's not even going to be relevant at that point.
790570	795570	It's mainly used for maybe comparing two different types of humans' DNA
795570	798010	to understand a brown-eyed person versus a blue-eyed person
798010	801410	versus an attached earlobes person versus a dangling.
803790	807570	Dynamic programming is a common concept in computer science.
807570	809850	It's not coding, per se.
809850	815670	It's just a terminology that's used to solve problems by using matrices.
815670	818170	So the idea is to break the problem up into subproblems.
818170	820290	So you have a matrix.
820290	824810	You perform a computation on each cell in the matrix,
824810	827910	and as the algorithm progresses,
827910	830710	it uses information that was previously gained in the other cells
830710	834590	to progress until the end of the process.
834590	840850	So the results of the previous computations are saved and used by the remaining subproblems.
840850	842630	And Niedermann-Wunsch is a dynamic programming algorithm,
842630	845610	as is most of the sequence alignment algorithms out there.
856230	861290	So how Niedermann-Wunsch works, it's actually a pretty simple little operation that it does.
861770	864490	The first sequence is placed in the topmost row,
864490	867190	and the second sequence is placed in the leftmost column.
867950	870250	And basically there's this little equation right here
870250	875430	that basically is used to assess each cell in the matrix,
875430	877230	starting at position 1, 1.
878230	881850	There's three steps for these sequence alignment algorithms.
881850	883790	The first step is to assign a similarity score.
883790	885370	So basically what you do is,
885370	888430	you go through and you score every single similarity
888430	892410	between the character in sequence 1 and the character in sequence 2.
893130	897590	The second step is you assess all possible pathways through the matrix,
897590	900790	left, up, and diagonal from the particular cell that you're in,
900790	904050	assigning the current cell with the value of the maximum scoring pathway
904050	905810	using this particular equation,
905810	910270	which I have a matrix on the next slide that will make it a bit easier to understand.
910450	914110	The third step is constructing a pathway from the highest scoring cell
914110	916490	back to the beginning of the matrix.
916850	921710	And what this represents is the insertion of gaps in each sequence.
923510	924550	And lastly...
924550	926770	Okay, actually, let me get back to the equation.
928030	931010	What you're going to do is, when you're in cell IJ,
931010	933010	you're going to look at your diagonal cell,
933010	936150	which is I-1, J-1, plus S-IJ,
936150	938370	which S-IJ is going to be the similarity score
940290	943770	between the offset I and offset J in each sequence.
944550	946810	It's also going to look at the left...
947790	952050	It's going to look at the left cell from the current position,
952050	954010	and it's going to add a gap penalty
954010	956530	because if you're going left or if you're going up,
956530	958750	you're actually inserting a gap into the sequence.
958750	960090	And that's not necessarily a good thing
960090	963230	because it kind of makes it a bit irrelevant if there's too many gaps.
963230	964890	So there's something called the gap penalty
964890	970890	that will basically dock the alignment score at the end.
970890	974010	So you'll be able to understand what the relative...
976690	979850	You'll be able to really understand how important this alignment is
979850	980890	and what it really represents.
980890	982050	If it has a really low score,
982050	984370	then you know that it's probably really not that significant.
986590	989270	So this is the visualization of the concept.
989270	992170	I use HTTP just because everybody understands it.
992170	994670	It's a text-based protocol. It's really simple.
994670	997010	You actually probably wouldn't ever have to use this program
997010	1000710	to do text-based protocols just because they're so easy.
1000930	1004190	But for example's sake,
1004190	1006950	I use it just because it's small enough
1006950	1008690	to fit into a PowerPoint presentation.
1008690	1014570	So the first sequence is placed in the topmost row,
1014570	1017510	which is git slash index.html HTTP 1.0.
1017510	1020970	And the second sequence is git slash HTTP 1.0.
1021350	1023210	So as mentioned earlier,
1023210	1024890	you just kind of iterate through the matrix
1024890	1031350	and you perform the scoring based on the similarity between each character.
1031350	1034330	So right here, the Gs match,
1034330	1037790	so you score with one and so on.
1037790	1040390	So you go out and you build up the matrix.
1040410	1041590	One thing to be mentioned, though, is
1041590	1044910	Needleman-Wunsch, kind of like the default
1044910	1046330	runtime environment for it,
1046330	1050090	is to have a match score of 1,
1050930	1052690	a gap penalty of 0,
1052690	1055770	and a mismatch score of 0.
1055770	1057930	So in some cases,
1057930	1060410	you can actually make this a bit more complex
1060410	1065530	by having different scoring mechanisms.
1065530	1067010	G might have a stronger alignment,
1067010	1068750	so you have a higher score in that case,
1068750	1069550	and things like that.
1069550	1072530	And when I talk about similarity matrices a little later on,
1072530	1074750	you'll kind of understand what I'm talking about.
1074750	1077270	It's probably a little too early for that.
1079130	1080410	So what you do is
1080410	1084450	you start in cell 1-1,
1084450	1086410	and you look at your left cell,
1086410	1087310	your diagonal cell,
1087310	1088490	and your upper cell.
1088830	1090670	And you apply this little equation right here,
1090670	1092330	which is going to take the maximum value
1092330	1094250	of the diagonal,
1094250	1095730	or the left,
1095730	1097190	or the upper.
1098590	1099750	For the diagonal cell,
1099750	1103430	you actually add the score of that position.
1103430	1105330	So in this case,
1105870	1106730	it's going to look at diagonal
1107250	1108570	and add 1,
1108570	1110570	because the G's match.
1111210	1113010	There's no match here, there's no match here,
1113010	1115330	so the maximum is going to be the diagonal plus 1.
1115330	1117690	So it's going to leave it a 1.
1118090	1120030	And the way that these matrices are traversed,
1120030	1121410	they're traversed left to right,
1121410	1123130	and then they move downward.
1124590	1127610	The next step, move over one position.
1128250	1130050	This time, it's going to look left,
1130050	1130810	it's going to look diagonal,
1130810	1131870	it's going to look up.
1132450	1133810	It's going to take the maximum.
1135230	1137330	In this case, before, the cell was 0,
1137330	1140330	so this S-I-J in this case is going to be 0.
1143070	1145850	It realizes that the leftmost cell in this case
1145850	1147050	is going to be the maximum,
1147050	1152330	so it places that particular value in this cell.
1153130	1154810	It's a really simple operation, actually.
1154810	1157930	It just goes cell by cell and it builds up.
1158210	1159850	This is kind of like the completed matrix.
1160030	1161750	I do this all in hex down here
1161750	1164110	just to make it all line up and look pretty on the screen.
1165090	1166630	Basically, this is kind of how it looks
1166630	1170650	after you perform all the scoring of the matrix.
1170650	1171670	The next step at this point
1171670	1175210	is to actually traverse back to the beginning of the matrix
1175210	1178410	and understand what your alignment is going to be.
1179370	1182630	In this case, typically,
1182630	1186030	the maximum cell is always going to be your bottom right.
1187090	1189810	What we do is we start in the bottom right cell in this case
1189810	1191370	because it had the highest value.
1192370	1193670	We look at the neighboring cells
1193670	1195270	and we choose the cell with the maximum value
1195270	1196610	and traverse there.
1197630	1198990	We start here at E
1199490	1201970	and we notice that our neighbors are all Ds.
1201970	1203610	Basically, that means that we can go anywhere
1203610	1204730	because they're all equal.
1204730	1206710	We can go left, we can go up, we can go diagonal.
1207190	1209470	I personally like to just go diagonal
1209470	1211530	just because it just makes the most sense.
1211530	1214570	Basically, it signifies that there's not going to be any gap
1214570	1217090	that's inserted into any of the sequences,
1217720	1220550	which may mean that there's actual match there.
1223870	1228930	This is kind of the evolution at this current point.
1229470	1234170	Starting down here, we look, we go diagonal because it's D.
1234170	1235790	At this point in D, we look around.
1235790	1236870	All of our neighbors are C.
1236870	1238890	We want to go diagonal again by default.
1239550	1240650	My mouse is disappearing.
1240870	1242330	The same deal here.
1242330	1244290	All B, so we go diagonal.
1244290	1245150	All the neighbors are A.
1245150	1246150	Go back to A.
1246170	1247850	Neighbors are 9, etc., etc.
1247970	1251610	All until we reach this point up here at this 5.
1251810	1253590	And we look at our neighboring cells again
1253590	1255450	and we realize that there's a 5 to our left,
1255510	1257510	a 4 to our diagonal, and a 4 to our up.
1257510	1259390	We want to have the maximized alignment
1259390	1261330	so we're going to take the path
1261330	1263830	that will result in the maximum score.
1264330	1266370	So at this point, we're going to start going left.
1268170	1269550	This is the completed traverse
1269550	1271530	all the way back to the beginning of the matrix.
1271650	1272790	So at this point...
1273550	1274270	Goddamn.
1274810	1277090	So at this point, we're looking around.
1277090	1279150	We notice that 5 to our left is the maximum,
1279150	1279850	so we're going to go there.
1279850	1280530	Same thing.
1280710	1283190	We're surrounded by two 4s and a 5 on the left,
1283190	1285530	so we go left, etc., etc.,
1285530	1287730	all the way until we hit back here.
1289470	1291030	And all of them are equal again,
1291030	1292710	so we decide that we're going to go diagonal.
1292750	1295270	And then it goes all the way back to the root.
1296430	1298250	So what this means at this point
1298250	1300190	is there are going to be gaps
1300190	1301830	inserted into one of these sequences.
1303150	1304670	So what did this do?
1304750	1306610	We computed a path through the matrix,
1306610	1308390	and now we can apply the rules of Needham and Woons
1308390	1310250	to obtain the two aligned sequences.
1310630	1311910	Basically, here's the rule.
1312270	1315070	Any time that you traverse left on the matrix,
1315470	1316750	a gap is inserted into a sequence.
1316750	1317570	So when you go left,
1317570	1320010	it's going to insert a gap into sequence 1
1320010	1321390	at that particular offset.
1321390	1322950	If you ever go up,
1322950	1324290	you're going to insert a gap at that offset
1324290	1325670	in the first sequence.
1325690	1328410	So, here's the results of that particular alignment.
1330310	1332030	On the bottom down here,
1332030	1334110	kind of a lame example with HTTP, I know,
1334110	1335990	but it illustrates the point.
1336930	1339470	There's gaps that are inserted right here
1339470	1340490	under index.html
1340490	1341790	because that particular sequence
1341790	1344410	did not have that file in there,
1344410	1346610	and everything else matches up.
1347130	1348970	So what this allows us to do, really,
1348970	1351510	is to understand the beginning and ending
1351510	1353610	of the boundaries in the packet.
1356030	1359190	So, if you wanted to analyze this really simple example,
1359190	1360850	you can say that, okay,
1360850	1364030	these two sequences had git slash in common.
1364310	1366030	We can consider that a keyword.
1366650	1368850	Index.html was a file name.
1368910	1370690	It's a variable length field.
1370870	1372810	You can put any file in there that you want.
1373470	1376310	And it was subject to deletion in the second sequence.
1376310	1377870	So what we can assume at this point
1377870	1380010	is that that is a variable length field
1380010	1382090	because it was subject to deletion,
1382090	1384750	followed by the keyword HTTP 1.0.
1386550	1389950	So, if you wanted to actually describe this protocol
1389950	1391010	in a fuzzer, you could say,
1391010	1392950	okay, we're going to send a git space slash
1392950	1394970	and we're going to send a whole bunch
1394970	1396210	of random data in between,
1396210	1397690	followed by HTTP 1.0.
1397690	1398590	Pretty simple.
1400850	1402870	This leads us into explaining
1402870	1404990	the local sequence alignment algorithm,
1404990	1406750	which is used kind of for a different purpose.
1406750	1409890	So, this is called Smith-Waterman.
1409890	1412730	I think it was written in 1979, I think.
1412730	1414010	I might be wrong about that.
1414010	1416050	It's basically a derivative of Needleman-Wunsch.
1416050	1418750	It just has some different rules, basically.
1420530	1422970	What the point that it's trying to do
1422970	1424710	is find the maximum local alignment
1424710	1425910	between the two sequences,
1425910	1428710	which is going to be a subsequence between the two.
1429390	1434310	It's going to find the subset in each sequence
1434310	1437630	that makes the most sense.
1437950	1439670	And that may be subject to gaps
1439670	1441290	and it might not necessarily.
1441830	1443390	So, this is especially useful
1443390	1445630	in determining similarity between distance sequences
1445630	1448670	because you could look at a chimpanzee genome
1448670	1449950	and a human genome
1449950	1451550	and use local sequence alignment algorithms
1451550	1454330	and you'll actually get a lot more data out of it
1454330	1458390	because the alignments will actually mean a lot more in the end.
1458530	1461470	So, local alignment has a strong scoring mechanism,
1461470	1462370	whereas global is weak
1462370	1464050	because of the number of gaps
1464050	1466930	that may be inserted into the global alignment.
1467670	1470230	So, local alignment algorithms are used a lot
1470230	1473610	to kind of search the database
1473610	1477490	to understand which species are the most similar to others
1477490	1478730	and things like that.
1482440	1484340	So, how Smith-Waterman works,
1484340	1486200	it's a derivative of Needleman-Wunsch.
1486200	1487220	The only difference is
1487220	1490360	you kind of start it with some different initial values.
1492200	1494380	Also, this is kind of like a default run.
1494380	1497560	You can also use similarity matrices and things like that
1497560	1499140	to modify how this works.
1499140	1503120	But by default, it has a match score of 2.
1503360	1506520	So, if G and G match, we assign that a 2.
1506520	1508040	It has a gap penalty of negative 2.
1508040	1510480	Any time that we move up or any time that we move left,
1510480	1514060	we're going to subtract negative 2 from the alignment score.
1514960	1518220	The lower the alignment score, the less significant it is.
1518420	1521100	Any mismatch, so G and E don't match,
1521100	1523560	it's going to have a mismatch score of negative 1.
1524440	1528140	The difference is, when you're in the matrix,
1528140	1532540	you start in the cell that has the highest score,
1532540	1534480	and you're going to start traversing back.
1534480	1537680	The difference is, when the score hits 0 in the matrix,
1537680	1539340	you're going to return immediately.
1539500	1542540	And at that point, those particular offsets in the matrix
1542540	1544640	are the ones that are saved and used
1544640	1546580	as a representative local alignment sequence.
1558100	1562020	I actually didn't do a matrix and stuff like that.
1562020	1564400	It was kind of short notice and things like that.
1564400	1567260	But basically, if you want to align these two sequences,
1567260	1569040	same thing as before,
1569040	1571260	the score's going to end up being 18,
1571260	1575680	and it's going to find the substring space, HTTP slash 1.0,
1575680	1578560	as being the most common local alignment
1578560	1580380	in those particular sequences.
1580380	1584520	So that's going to be the most similar difference between the two.
1586360	1588080	Smith-Waterman is actually used a lot
1588080	1590260	to determine the distance between two sequences.
1590260	1594080	So if you want to find the distance between sequence 1 and sequence 2,
1594080	1597020	you're going to take the Smith-Waterman score of sequence 1
1597020	1598480	running against itself,
1598480	1600640	minus the Smith-Waterman score of sequence 1
1600640	1602360	aligned with sequence 2.
1602360	1605200	So in this case, it's 48 minus 18, which is 30.
1605200	1607100	So the distance is going to be 30.
1607100	1609340	Just pretty simple.
1610780	1612580	Basically, here's how it's used.
1612580	1615360	In my program, it's not necessarily used
1615360	1616720	to determine the protocol fields.
1616720	1619200	I use it just as a classifier
1619200	1621080	to allow me to build
1621080	1623320	kind of like a relationship matrix
1623320	1625520	and understand which packets
1625520	1626700	in the tcpdump file
1626700	1628380	are the most similar to each other.
1628380	1631620	Because if you're looking at a complex protocol
1631620	1633300	like isocamp or smb
1633300	1634040	or something like that,
1634040	1636340	they have multiple different message types.
1636640	1637980	This is actually going to be really helpful
1637980	1639440	because it's going to allow us to cluster
1640020	1642280	the different types of messages
1642280	1643920	within that particular protocol.
1645560	1647240	It's used to calculate the relative distance
1647240	1648700	between sequences for clustering purposes
1648700	1650460	and tree building, which we'll get into later.
1650460	1653320	In bioinformatics,
1653320	1655060	Smith-Waterman is used for multiple sequence alignment
1655060	1657040	because DNA has no start and end
1657040	1658600	whereas a packet does.
1658600	1660740	That's also another big advantage that we have.
1660740	1662980	We know where the packet starts, we know where the packet ends.
1662980	1665020	We also have fields that start here and end there.
1665020	1668540	There's no fields that are going to span
1668540	1670440	over multiple blocks
1670440	1671660	and jump in over each other
1671660	1672360	and stuff like that,
1672360	1674280	like it does in DNA.
1677380	1679740	To improve these alignment algorithms,
1679740	1681800	we use what's called a similarity matrix.
1681840	1685220	Basically, we replace our scoring mechanism.
1685920	1687100	Before, I was showing you
1687100	1688940	that a match score would equal 1
1688940	1690700	in Niedermann-Wunsch, a match score in
1690700	1693520	Smith-Waterman would equal 2, and things like that.
1693620	1695180	Similarity matrices, on the other hand,
1695180	1696740	are tables
1697400	1698740	based on the probability
1699360	1700840	that this sequence is allowed
1700840	1702860	to mutate into another sequence.
1703200	1705100	In evolution, it is possible
1705100	1707820	for different amino acids
1707820	1708760	to mutate
1708760	1710840	into others, but not
1710840	1711620	other ones.
1711720	1714700	It basically allows
1715240	1717040	biologists to say,
1717040	1718680	adenine can mutate into thymine
1718680	1720260	but not cytosine, and things like that.
1720260	1722600	By the way, I'm not a biologist, I don't actually know
1722600	1724680	if that is true, but it's just an example.
1726080	1726800	Similarity matrices
1728280	1736340	are pretty different, I guess, because
1737500	1738440	it's kind of
1738440	1739800	based on preference.
1740440	1742400	There's similarity matrices out there that are
1742400	1744080	used only when
1744620	1746440	comparing really distant sequences,
1746440	1748200	and there's similarity matrices that are used when comparing
1748200	1750520	really close sequences.
1751620	1752540	The applications
1752540	1754420	to protocol analysis is really good, though, because
1754420	1756400	we have data types. We have
1756400	1758360	plaintext data, we have binary data,
1758360	1759800	and things like that.
1760240	1762080	We can assume that binary data
1762080	1764200	is acceptable for binary data to mutate
1764200	1766120	into other binary data, and plaintext data
1766120	1768040	to mutate into other plaintext data.
1768040	1770480	Especially if you're using a
1770480	1772460	protocol that usually uses English words,
1772460	1774360	you can actually have a
1774360	1776280	similarity matrix that accepts
1777220	1778400	I before E
1778400	1780660	and other kind of common
1780660	1784460	linguistic rules like that.
1784460	1786180	So,
1786180	1788520	the protocol informatics similarity matrices,
1788520	1790200	our alphabet is 256
1790920	1792020	bytes long, so we have a
1792020	1793880	256 by 256 matrix,
1793880	1796040	and each mutation,
1796040	1797660	each cell contains
1797660	1799720	mutation probabilities between every character, so
1799720	1801620	we might say that it's acceptable for
1801620	1803680	A to mutate into E, but not into
1803680	1806500	hex FF and things like that.
1807040	1807620	So,
1808560	1809800	this is actually kind of some
1809800	1812500	outdated stuff. There's actually a lot easier ways of doing this.
1812520	1813440	So,
1813440	1815040	what this allows us to do is
1815040	1818000	obtain a bit more optimized alignments.
1818160	1819700	This way, if we use these
1819700	1821520	similarity matrices that are specially tailored
1821520	1823620	for different types of network protocols,
1823620	1825480	if you have one that's mixed,
1825480	1827440	has plaintext in it and binary data in it,
1827440	1829900	you could use a particular similarity matrix in there.
1830000	1831700	And basically what this does is it allows
1831700	1833700	when gaps are inserted into the sequences,
1834520	1835680	the alignment is still
1835680	1836860	going to hold true where
1837860	1839600	plaintext data is only mutating into other
1839600	1841520	plaintext data, binary data is only mutating into
1841520	1844320	other binary data. You're not going to have any mismatches in between.
1844320	1845620	So, I guess
1845620	1847680	the kind of role of similarity matrices though
1847680	1849180	is that they're
1849680	1851800	going to really have to be tweaked.
1852000	1853160	And you could have
1854380	1855700	probabilistic models, you can do some
1855700	1857420	Markov chains and stuff like that to
1857420	1860380	initially build your stuff, but it's also kind of subjective.
1861100	1861960	I can say
1861960	1863620	that I know that
1867400	1867880	a plaintext
1867880	1869600	character is commonly followed
1869600	1871680	by a null, so it's able
1871680	1873940	to mutate into a null character and stuff like that.
1874000	1875560	That's very acceptable.
1875560	1877880	So, bioinformatics
1877880	1879800	scientists spend years perfecting their version of similarity
1879800	1881780	matrices. I don't know how many revisions
1881780	1883560	there are of Blozeman and the PAM matrices,
1884080	1885700	but there's quite a lot, and they're each
1885700	1887340	used for kind of different purposes.
1894780	1895620	I kind of actually had
1896440	1896880	a problem
1897620	1899540	trying to represent this stuff and kind of
1900640	1901700	allow others to
1901700	1903600	understand kind of what I'm talking about.
1903860	1905780	And this isn't actually that great, but
1905780	1907780	this is just a kind of stupid example.
1907780	1909540	This superscript B just stands for binary.
1909540	1911440	So let's just assume it's like 0x01,
1911440	1913920	0x02, 0x03, etc.
1913920	1916300	So if you wanted to align these protocols,
1916300	1917600	alignment without proper scoring
1917600	1918740	might actually
1919720	1921420	insert gaps into incorrect
1921420	1923540	places and
1923540	1925180	assume that
1925180	1927560	F might be able to mutate
1927560	1929820	into 0x01 and 0x04
1929820	1931280	and stuff like that, which isn't necessarily
1931620	1932880	a correct alignment.
1933620	1935440	If you use a similarity matrix
1935440	1937400	with decent data
1937400	1939380	that keeps track of
1939380	1942160	mutations between different types of data types,
1942160	1943440	you'll start getting better
1943440	1945360	alignments where all the ASCII
1945360	1947340	data is aligned to each other, all the binary data is
1947340	1948780	aligned to each other, and things like that.
1948780	1951300	So it's basically just a tool to help us
1952180	1952600	signify
1953500	1955160	our alignments a bit better.
1956980	1957400	So
1958240	1959580	I kind of explained so far
1959580	1961320	how to do a pairwise sequence alignment, which
1961320	1963860	isn't that useful. Just have a
1963860	1965380	look at two HTTP requests and say,
1965380	1967660	okay, here's the variable link field.
1967660	1969340	This is where you put file names.
1969340	1970740	It's really not that big of a deal.
1972460	1973460	But it's really not that
1973460	1975560	pairwise sequence alignment isn't really that useful.
1975560	1977480	You really have to start getting the multiple sequence
1977480	1979180	alignment if you really want to
1979180	1981240	get significant results.
1984960	1986900	Multiple sequence alignment is basically the
1986900	1989040	act of aligning more than two sequences.
1990240	1990660	It uses
1990660	1992660	Niedermann-Munch as the alignment algorithm and uses
1992660	1994700	Smith-Waterman as a scoring algorithm.
1994700	1996760	The problem with this is there's computation
1996760	1997640	issues.
2001920	2003300	So Niedermann-Munch, as I
2003300	2005220	showed before, it operates
2005220	2007220	on a n by n, or
2007220	2009160	excuse me, n by m matrix where n is
2009160	2011160	the length of sequence one and m is the length of
2011160	2013320	sequence two. If you want
2013320	2015180	to start aligning three sequences, you can
2015180	2017380	start building hypercubes and things like that and traversing
2017380	2018540	through hypercubes.
2018660	2020800	The problem is the operation is
2020800	2023080	n to the nth power where n is the length of sequences
2023080	2024660	and n is the number of sequences.
2024660	2026640	This isn't really going to be feasible
2026640	2028540	for us. So I have my little
2028540	2030900	illustration of my hypercube being traversed.
2030900	2032780	It can be done,
2032780	2034700	but I've been trying to align
2034700	2036740	sequences that are up to 1500 bytes
2036740	2038520	in length and it takes
2038840	2040660	a really, really, really long time.
2044740	2046540	What we use is a heuristic
2046540	2048060	sequence alignment algorithm.
2048620	2051440	I'm basically sacrificing accuracy for time.
2051660	2052600	The objective is to align
2052600	2054580	every sequence to each other in a reasonable
2054580	2056640	amount of time, but the results are never, ever going to be
2056640	2057380	perfect.
2059440	2060760	The way that this is done
2060760	2062760	is by building these polygenetic trees.
2063440	2064820	Polygenetic trees are basically
2065660	2066620	what you guys would think
2066620	2068360	of a tree of life. You start with
2069040	2070680	common ancestors and you kind of move
2070680	2072660	down and you can start understanding the
2072660	2074640	similarities between certain species and things
2074640	2076400	like that. It's the same thing with packets.
2076880	2078340	We know that in SMB
2078340	2080600	all the SMB authentication packets are going to be
2080600	2082880	more similar than a data transfer packet
2082880	2083760	and things like that.
2084060	2085820	Typically, these are just binary trees
2086540	2088960	which makes it quite simple, actually.
2090140	2092600	It does have an interesting parallel in protocol
2092600	2094640	analysis because protocols mimic
2094640	2096560	evolution by changing their fields.
2096960	2098600	The kind of goofy thing
2098600	2099920	that I have here is what came first,
2099920	2102440	git slash index.html or git slash?
2102440	2104360	Obviously, git slash is going to be the
2104360	2105920	common ancestor.
2109050	2110050	Here's my little
2110050	2112810	rip-off of a tree of life of lizards.
2112910	2113790	As you can see,
2113790	2115710	on the right-hand side,
2115710	2117970	these guys look a lot more similar
2117970	2119930	in structure than to this
2119930	2122010	C. tigris guy over here with the
2122010	2123990	long tail. So, basically, similar
2123990	2126070	situation, apply it to packets.
2138290	2140010	So, to create
2140010	2142230	polygenetic trees, there's an algorithm called Upcoma
2142230	2144330	which is the unweighted paired group method using
2144330	2146530	arithmetic averages. It's pretty simple,
2146530	2148290	actually. This little
2148290	2150130	equation right here just signifies
2150130	2152890	how the distance between two clusters is taken.
2153250	2154050	Basically,
2154050	2155750	the dpq
2156290	2158170	function is going to determine
2158170	2159990	the distance between each
2160650	2162270	sequence in cluster i and
2162270	2164170	cluster j. And it's basically just
2164170	2165970	taking the mean of that and returning that back as
2165970	2167470	the distance of i and j.
2168250	2170370	In this case, the dpq
2170370	2172490	is going to be the Smith-Waterman score and the d
2172910	2174550	ij is the distance between two clusters
2174550	2176270	i and j. So, this is just
2176270	2178290	the operation that's used each step of the
2178290	2179970	way when building the tree.
2181050	2182330	So, to build
2182330	2184110	the tree, each sequence is put
2184110	2186250	into its individual cluster cell, one packet
2186250	2188430	per cluster, basically. Use the Upcoma
2188430	2190130	algorithm to iterate through every single cluster
2190130	2191910	doing comparisons between every single
2191910	2194150	sequence. You take the two with
2194150	2196350	the smallest distance between each other.
2196370	2198250	You create a new cluster k by uniting
2198250	2200010	those two sequences.
2200810	2202310	You define a new node k with child
2202310	2204370	nodes i and j to build the tree.
2204370	2206410	You add the cluster k
2206410	2207790	to the universal set and you remove
2208290	2210290	the clusters i and j. So, basically
2210290	2212570	at the very end, you should only have one cluster.
2213370	2214390	And you should have a tree build
2214390	2215350	at that point.
2216610	2218410	Here is a phylogenetic tree
2218410	2220310	of the SMB protocol. This is just
2220310	2222830	doing null sessions and stuff like that.
2222850	2224450	As you can see, there's
2224450	2226490	actually a pretty large disparity between
2226490	2228090	the differences between
2228550	2230130	different types of packets.
2230630	2232350	The edge length signifies
2232910	2233990	the distance.
2234630	2236570	So, it's kind of
2236570	2237750	more than a pretty picture.
2238590	2239690	I'm sorry I can't actually
2239690	2242370	I don't have any cool interface where I can zoom in and tell you
2242370	2244430	exactly what those packets were and stuff like that at this current
2244430	2246610	time, but from the analysis that I did
2246610	2248310	each one of these clusters up here
2248930	2250270	represented one of the SMB
2250890	2252490	message types in the protocol.
2252530	2254350	So, this is actually going to help us
2255390	2256510	say we want to analyze
2256510	2258130	the SMB authentication
2258570	2260190	packet. Okay, it's
2260190	2262290	this cluster over here, for instance.
2262290	2264170	So, we start at this root node, and then we're going to
2264170	2266810	start doing our progressive alignment over here.
2266950	2268270	So,
2270690	2272650	the tree is your guide. Basically,
2272650	2274610	what this tree is going to be used for is to align
2274610	2276470	the sequences to each other.
2277250	2278550	As I was talking about the heuristic
2278550	2280730	alignment earlier,
2280730	2282650	what we're basically going to have to do is what's
2282650	2284710	called progressive sequence alignment, and what that means
2284710	2286550	is we're going to align two sequences to
2286550	2288530	each other, and then
2288530	2289750	we're going to
2290750	2291990	pick one of the
2291990	2293770	alignments and use that as a representative
2293770	2296010	sequence for those two sequences.
2296350	2297910	Then we're going to align another sequence
2297910	2299930	to that representative sequence. And as
2299930	2301750	gaps are inserted, they are
2302690	2303930	moved down towards
2303930	2305990	the root, excuse me,
2305990	2307810	down towards the leaf nodes
2307810	2309630	on the graph. So,
2309630	2311130	the problem with this is
2313890	2315970	if gaps are inserted into the tree
2315970	2317910	in errant places, that will be propagated
2317910	2320050	all the way down to the leaf nodes, which can result
2320050	2321310	in insignificant alignment.
2322130	2323830	It probably should be a more
2324490	2325250	tree building
2325770	2327310	should probably be a bit more
2327770	2329370	of a manual process where
2330010	2332070	while you're doing the analysis, you kind of look at it
2332070	2333810	and you say, this does look acceptable
2333810	2335910	to me, I'm going to accept this alignment to be
2335910	2337910	the representative sequence, so
2337910	2339950	gaps aren't inserted into errant places
2339950	2340850	basically.
2342590	2344130	Opposed to the exponential hypercube
2344130	2345650	traversal, Upcoma performs
2345650	2347610	in comparisons where n is equal to
2347610	2349370	the depth of the tree minus the root.
2349850	2351590	That's very feasible for us.
2351590	2353550	We can actually align sequences in our lifetime
2353550	2355050	which is very helpful, I think.
2356470	2357790	So, multiple sequence
2357790	2359490	alignment. Basically, the rule
2359490	2361530	as I was explaining earlier about progressive alignment
2361530	2363330	is once a gap, always a gap.
2364230	2365810	Basically, what you're going to do is
2365810	2367570	you're going to align two sequences
2367570	2369510	to each other, place a representative
2369510	2371410	sequence in the root node of that
2371410	2372270	and
2372270	2372390	and
2374290	2376130	as you align sequences
2376130	2378130	to those representative sequences, those
2378130	2380330	gaps are going to translate themselves
2380330	2381830	all the way down to the leaf node.
2382430	2384150	Basically, the rule for
2384150	2386410	doing the tree traversal, you start in the root
2386410	2387910	and if the root is null
2387910	2390150	you start going left and then you go right.
2390610	2392310	If the left is not null and the right is not null
2392310	2394310	which signifies that there are sequences on those
2394310	2396050	two leaf nodes, you align those two sequences
2396530	2398250	you choose a sequence with the least
2398250	2400050	number of gaps inserted and that's kind of just
2401090	2402230	I got the best results
2402230	2404050	out of that. I don't really know if that's kosher
2404050	2406010	in the bioinformatics world or anything like that, but
2406010	2408050	from my experience, that resulted
2408050	2409690	in decent alignment.
2410750	2411970	In this case
2411970	2413770	we aligned sequence one to sequence two
2413770	2415850	sequence one is chosen to be the representative
2415850	2418010	because there were not as many gaps inserted
2418010	2419470	so it kind of lowers the margin of error
2419470	2421490	of error and gaps propagating all the way down
2421490	2424050	through all the alignment. So, what we're going to do is
2424050	2426070	we're going to place sequence
2426070	2428310	one in the root node
2428310	2429730	of these two leaf nodes
2429730	2431570	and we're going to keep track of the edits
2431570	2434470	in the edge. So, the edits in this case would be
2435410	2436410	I don't have
2436410	2438610	the list of the offsets right in front of my face
2438610	2438850	but
2440070	2442330	there's going to be edits that are inserted into sequence two
2442330	2444410	to insert those gaps.
2453700	2454620	Pretty tree,
2454620	2455380	not really.
2455820	2457560	So, you start in the
2458480	2460200	root node, the parent node right up here
2461320	2462300	like I was saying earlier
2462300	2464160	if the root is null, you go left then right
2464160	2466420	the root is null in this case, there's no value
2466420	2468380	being stored there, so we're going to go left.
2469000	2470400	Same thing here, we're going to go
2471280	2472320	our child nodes
2472320	2474100	are not null, so what we're going to do is we're going to
2474100	2476300	align sequence one and we're going to align sequence two.
2478020	2479860	We're going to choose a representative sequence
2479860	2482180	which is going to be one, we call it one prime at this point
2482180	2484360	we keep track of the edits in the edge.
2484720	2486360	So, the edits
2487420	2488320	between sequence one
2488320	2489980	and two are stored in the left
2489980	2492280	edge and the edits for sequence
2492280	2494060	two are stored in the right edge.
2495360	2496420	We just traverse our way
2496420	2497220	back up to the root
2498460	2499660	start moving right now
2501360	2502240	we align
2502240	2504640	the representative sequence which is one prime
2504640	2506380	with three, store those
2506380	2508340	edits in the edge, and now what we need to do
2508340	2509520	is we actually need to
2510100	2512400	propagate those gaps that are inserted via
2512400	2514440	the edits all the way down
2514440	2515860	to the leaf node.
2517220	2518420	If we want to
2518420	2520500	align all the sequences to each other, we basically
2521240	2522820	add up all the edits
2522820	2524240	so sequence one aligned
2524240	2525860	which is over here, we're going to
2526400	2528220	add the edits of one prime three
2528220	2530140	to one two
2530140	2532420	to add the edits
2532420	2534480	the edits in this case are going to be the gaps
2534480	2535900	by the way. If we're going to
2537140	2537660	propagate
2538320	2540640	those gaps down the tree to sequence
2540640	2542500	two, we're going to take one prime three
2542500	2543760	add it to
2544480	2546520	the edits of two one
2546520	2547560	and lastly
2548780	2549820	for sequence
2549820	2551500	three, all we need to do is add the edits of
2551500	2553820	three and the edits of sequence
2553820	2555860	three and one prime in this case.
2557420	2557940	As
2557940	2558840	explained.
2560220	2561840	So, another
2561840	2563740	kind of hokey example of three
2563740	2565100	sequences being aligned this time
2566700	2567180	a bit more
2568600	2569120	verbose
2569120	2571140	usage of HTTP I guess
2572220	2573760	so basically in
2573760	2575860	this case right down here, this is what's
2575860	2577140	called our consensus sequence
2577880	2580000	what we basically do is we start in
2580000	2581600	offset zero and we go down and we say
2581600	2584200	these are all a bunch of G's so we're going to put a G down here
2584200	2584860	etc etc
2585680	2587840	once we start getting over here we start noticing that
2587840	2589540	there's gaps inserted so
2589540	2591800	that might possibly mean that
2591800	2593980	that is a variable link field so in this case
2593980	2595780	I'm just representing that by the question
2595780	2597600	mark down there and
2597600	2599520	all is the same
2599520	2601880	you have a space here, this all
2601880	2604000	matches up just fine, followed by host
2604000	2605820	followed by more unknowns
2605820	2607400	dot three
2607860	2609680	three spaces for .com, .net
2609680	2611940	etc, followed by the
2611940	2613120	user agent block
2613940	2615800	followed by the browser type
2615800	2617840	followed by accept text X amount
2617840	2619880	so, when we're
2619880	2622040	analyzing this consensus sequence, basically what we can
2622040	2623880	say is okay, here's the structure
2623880	2625480	of this particular format
2625960	2627960	we have a get space slash, followed by a variable
2627960	2630460	link string, followed by HTTP 1.0
2630460	2632000	I'm not counting new lines or
2632000	2633880	anything like that at this time, followed by
2633880	2635840	host, followed by another variable link
2635840	2637980	sequence, followed by a dot, followed by another
2637980	2639920	variable link sequence, followed by user agent
2639920	2641720	variable link sequence, accept
2641720	2643340	text X amount. So
2643880	2645960	what we could do at this
2645960	2647340	point is feed this little
2648600	2649900	structure into our fuzzer and
2649900	2651780	just let it start going
2651780	2653580	this is a completely automated process
2653580	2655900	I didn't really have to do anything in this case
2656700	2657840	unfortunately I don't have
2657840	2659820	any good mixed protocol where it's
2659820	2661580	half binary, half plain text
2662720	2664100	for me to show you right now
2664100	2664720	but
2665680	2667780	I'll put up some examples on the website and kind of like
2667780	2669640	walk you through using the tool and stuff like that
2669640	2671340	that might be actually useful
2673360	2674120	one thing
2674120	2675900	that I want to say about this though is
2676620	2677480	if you'll notice
2678260	2679840	with these consensus sequences, I'm just
2679840	2681840	assuming that this is
2681840	2683160	going to be variable length and
2683670	2685500	this is not going to mean anything. So basically
2685500	2687420	what's happening here is I'm losing information
2687420	2689580	there's a C here and two gaps
2689580	2691520	there's got to be a better way to
2691520	2693160	represent these consensus sequences
2693160	2695540	there's something called sequence logos
2695540	2697300	and basically what it is
2697300	2699020	it's kind of just like a stacked histogram
2699020	2701080	of each offset
2701080	2703600	so let me find a good example
2703600	2705700	so right here after the host
2705700	2707800	you have two W's followed by a
2707800	2709780	gap. Basically when you stack that
2709780	2711620	histogram up, usually
2711720	2714720	it's a W but sometimes it
2714720	2715440	contains a gap and things like that
2715440	2717340	so in that case
2717340	2719480	you don't really lose information and you're actually
2719480	2721600	able to use a little more of your human intuition
2721600	2722780	to be able to understand
2723440	2725140	the alignment a bit better than just
2726520	2727320	ruling it out
2727320	2729340	just because there are gaps or because there are mismatches
2729340	2730040	and things like that
2730040	2736620	so basically
2736620	2738960	here's how we're going to analyze
2738960	2740880	the consensus sequence in
2740880	2742700	this case. We could do
2742700	2744940	statistical analysis on each column
2744940	2746900	by building histograms so we build
2746900	2747900	our consensus sequence
2748380	2749880	like I showed you on the previous page
2749880	2751980	and we look at the mutation rates
2751980	2754300	and the offset comparison between
2754700	2756600	each offset. So basically
2756600	2757460	what we can do is
2758180	2759100	we can say
2760640	2762900	this has a
2764180	2764960	mutation rate of
2765440	2766940	let me try to find a good example
2779020	2780160	what we can do is
2780160	2782380	if you look at this block right here of all these question marks
2783700	2785480	almost every single one of these
2785480	2786820	has a similar mutation rate
2786820	2789140	they have a rate of change that is pretty
2789140	2791080	constant. So if we
2791080	2793220	wanted to analyze this like a binary protocol
2793220	2795180	it's a lot more difficult to
2795180	2796840	look at just on the screen
2796840	2798580	we pretty much base our
2799140	2801020	understanding where the fields begin
2801020	2803060	and fields end based on the mutation rate
2803060	2805240	so for instance
2805820	2807080	what we can do is
2807080	2809360	look at the rates of mutation
2809360	2810880	for each offset and
2810880	2812980	using that we can kind of understand
2812980	2814900	what the particular field might be
2814900	2815900	used for.
2816820	2818840	One thing I wanted to mention though is beware
2818840	2820900	of junk data. In the last example if there were any
2820900	2822680	posts in there it would
2822680	2824280	result in kind of a bad alignment
2824880	2826760	and it would actually be a little
2826760	2828580	more difficult to understand
2828580	2830240	what the general structure was.
2830980	2832300	What I was talking about earlier
2832300	2834800	with the clustering algorithms, if you cluster
2834800	2836780	things correctly and you choose
2838040	2839440	the root nodes
2839440	2840300	in each cluster
2843020	2845700	you'll be able to get decent alignments
2845700	2847080	out of it instead of trying to align
2847080	2849320	really distant sequences where lots of gaps
2849320	2850500	are going to be inserted.
2853520	2854680	Classification is your friend
2855160	2857300	the better that you cluster
2857300	2859420	the packets, the better your sequence
2859420	2860980	alignment is going to end up being.
2864320	2865680	This was kind of an experimental
2865680	2867140	phase about a year ago
2867140	2868920	when I was working on this but I kind of
2868920	2871120	already implemented this in code so it's really not that experimental
2871120	2873820	anymore. I just haven't really had a chance to work on the slides.
2874760	2875420	At first
2875420	2877140	what I was basically doing was I was
2877140	2879120	separating the dynamic data, data that
2879120	2881200	was changing versus the static data. So in the case
2881200	2883200	of HTTP I was able to understand
2883200	2884800	just the different blocks and stuff like that
2884800	2886740	and understanding the variable length fields
2886740	2889300	just because it was dynamic data it was changing.
2889980	2891040	To understand integer
2891040	2893800	fields this actually worked out pretty good.
2894200	2894840	You can build
2894840	2896600	just like an engram frequency table
2896600	2898300	for 1, 2, and 4 byte window sizes
2898820	2900220	and you just kind of slide yourself
2900720	2902880	over the consensus sequence at that point
2903740	2905000	and you can use
2905000	2906740	some logic in your program to
2906740	2908620	try to discern what
2909120	2910760	the particular function is of that
2910760	2912160	field. So for example
2912160	2914940	if two consecutive bytes that are integers
2915440	2916620	mutate at the same rate
2916620	2918660	the chances are that they are part of the same field and maybe
2918660	2920740	they are checksum or something like that. They are changing every
2920740	2921760	single packet.
2922300	2923460	The second example
2923460	2925780	if in two consecutive bytes the least significant
2925780	2927860	byte increments faster than the greater significant
2927860	2929820	byte it might be a 16-bit sequence identifier
2929820	2931480	field or something of that sort.
2942100	2942220	So
2944600	2947540	this is all also implemented stuff fortunately.
2947980	2948860	But what I was going
2948860	2950800	to basically do is build a protocol profile based on
2950800	2952900	each consensus and allowing each sequence individually
2952900	2954800	to break down the structure. So
2954800	2956240	for example I would
2957320	2958360	take a
2958360	2959660	SMB login
2960240	2962400	authentication packet consensus sequence
2962400	2964500	I would take a SMB data transfer
2964500	2966520	packet and then I would align
2966520	2968500	those two consensus sequences
2968500	2970500	to understand the general structure
2970500	2972280	of the packet
2972280	2974460	format in general. They are still going to be pretty
2974460	2976460	deviant but you will be able to understand
2976460	2978580	the specific blocks in between.
2978580	2980180	Like on block based protocols like ISACAMP
2980180	2981460	and SMB and things like that
2982280	2983920	that is actually going to be quite helpful
2983920	2985680	I would think.
2986300	2987980	The biggest challenge for me this far
2987980	2989500	is being able to present the data
2989500	2991860	in a readable format where the user can actually
2992520	2994060	understand the results.
2994620	2994900	So
2995660	2997260	I have all this stuff kind of implemented
2997880	2999580	like on a command line tool
2999580	3001640	which kind of made it difficult so I'm just kind of using
3001640	3003720	ANSI color and stuff like that to
3003720	3005660	kind of print all this stuff out.
3006900	3007840	And the last
3007840	3009700	thing I wanted to say about this particular tool
3009700	3011740	is it will never be fully automated
3011740	3013280	if accuracy is in mind but
3013280	3014940	if you are doing protocol fuzzing
3014940	3016600	you don't really have to be accurate necessarily
3016600	3017960	so it's kind of okay to have
3018960	3020620	it's kind of okay I guess to have
3021860	3023280	that kind of entropy
3023280	3025760	in each sequence.
3028960	3030320	Let me get this.
3036920	3037720	Sorry.
3037960	3039940	So basically what this can be used for is
3039940	3042300	understanding unknown protocols, some kind of
3042300	3042460	like
3043760	3045880	worm for all we know communicates with
3045880	3048020	its brothers and sisters all over the internet
3048020	3050800	using some binary format that we've never seen before.
3051100	3052100	This also allows us
3052100	3054040	to fuzz network protocols more efficiently because
3054040	3056700	we don't necessarily have to write protocol specs for them.
3057000	3057960	If it's a common
3057960	3060140	protocol I'd probably say just go
3060140	3062220	read the RFC or something like that and
3062220	3063680	just look at the ethereal
3063680	3066340	samples because there's so many of them.
3066340	3068060	But if you're looking at some black box
3068060	3069940	protocol you've never ever seen before in your life
3069940	3072180	this is going to be a tool that would probably
3072180	3073520	definitely be useful.
3076520	3077180	Also,
3077180	3077660	one thing
3077660	3079400	I kind of forgot to mention but
3079400	3081580	this can also be used for file formats.
3081860	3083500	There's really no difference between
3083640	3085460	a packet format and a file format so if you wanted
3085460	3087480	to understand some proprietary file
3087480	3090020	format, same situation.
3090020	3091560	The only thing that you need to do is just have a lot
3091560	3093420	of samples that are different.
3095760	3097540	And also if any of you guys have any ideas
3097540	3099480	about how this stuff can be used I'm interested in
3099480	3100220	hearing it.
3101440	3102340	My conclusion
3102340	3104320	is this stuff will never be automated
3104320	3106080	100% like I was describing
3106080	3108140	the tree building earlier. It's good
3108140	3109720	if you wanted to
3111960	3112780	be involved
3112780	3114280	in each step of the process so you
3114280	3116280	know which representative sequence is used and which
3116280	3118380	gaps are inserted because as gaps propagate
3118380	3120320	down to the
3120320	3122180	root nodes there can be errant gaps
3122180	3124100	that make the alignment quite useless
3124100	3125340	and things like that.
3126520	3128480	Basically right now I have a Python
3128480	3130580	proof of concept for this. I have some
3130580	3132320	C++ code that I'm working on.
3132500	3134360	It's actually going to be pretty good. I should have it
3134360	3136300	probably finished in about a month or two.
3136620	3138740	The Python code right now, I use Pyrex
3138740	3140240	I don't know if you guys have ever heard that, to do
3140240	3142520	all the hard, computationally
3142520	3144220	expensive algorithms like the Neiman
3144220	3146480	Woons matrix operations and stuff like that.
3147120	3148580	I don't know if you guys have ever seen this
3148580	3150440	tool. It's called Orange. It's pretty cool.
3150780	3152120	It's basically like a widget based
3152120	3154400	visual programming interface that kind of allows you
3154400	3155320	to play with different
3156300	3157000	data mining
3157000	3159760	algorithms. So you can kind of link together everything.
3159800	3161380	In this case, what you'd be able
3161380	3162260	to do with
3163540	3165060	protocol informatics is be able to say
3165060	3167680	okay, I want to use this particular
3169120	3169560	classification
3169560	3171160	algorithm for clustering. I want to use
3171160	3173740	this particular sequence alignment algorithm.
3173740	3175740	I want to use this particular tree building algorithm.
3175740	3176440	Because there's actually
3177220	3179240	four different types of tree building algorithms.
3179240	3181100	There's the upcomma, there's neighbor joining, there's
3181100	3183320	multiple parsimony, things like that.
3183320	3184760	There's a multitude of
3185220	3187280	opportunity to implement all sorts of
3187280	3189360	things and kind of just figure out what works.
3189600	3191400	I guess the whole moral
3191400	3193180	of the story with what I'm trying to say
3193180	3193560	is
3195080	3197340	I know it's kind of difficult for people
3197340	3199340	in this industry to kind of look outside
3199340	3201280	their science that they're working on
3201280	3203380	and kind of understand that people might
3203380	3205060	have already kind of solved everything
3205060	3207320	back in the day. So this is just kind of a good
3207320	3209280	example. This stuff was figured out in the
3209280	3211340	70s and it's 2005
3211340	3213460	now and we found a use for it.
3216460	3217380	I'm taking questions
3217380	3219460	now. Feel free to walk
3219460	3221400	up to get a microphone up here or something
3221400	3222280	like that.
3223260	3224400	Thanks for coming.
3233700	3234500	Questions?
3237540	3238640	I can't hear you.
3239000	3240860	Do you want to come up
3240860	3242520	here and try to use a mic?
3259470	3260390	I don't know.
3262330	3263350	Here you go.
3266180	3266760	Yeah,
3266760	3268660	I'm working on almost exactly the same thing
3268660	3270660	with mutating web things, but I
3270660	3272320	just walked in on the end of your lecture and
3272320	3274600	I was talking to this guy who's also
3274600	3276740	doing tree algorithms, but he's doing
3276740	3277940	it with granularity
3278460	3281000	based on the levels of complexity
3281000	3282320	of the search.
3282780	3284900	Have you looked at any of that stuff
3284900	3286680	for your sorting and tree
3286680	3287640	methods?
3289000	3290300	Yeah, that's basically a maximum
3290300	3292020	parsimony tree building. That's where
3292660	3293220	every
3294020	3296240	step of the tree, you're going to look at
3297100	3298040	the complexity
3298040	3300260	of what it would take for these sequences to mutate
3300260	3302800	to reach this particular result.
3302800	3303920	And usually,
3303920	3306080	evolution chooses the path of
3306080	3307980	least resistance. So, in maximum
3307980	3309760	parsimony trees, you have
3309760	3312020	two different choices. You're going to choose the choice that's the most
3312020	3312860	simple.
3315900	3317200	Anyone else?
3322400	3324440	Okay. Well,
3324440	3326500	this was a lot of information. I'm sorry if I
3326500	3328320	didn't have any other ways to represent it
3328320	3330300	in a kind of clear and concise
3330300	3331240	manner, but
3332340	3334460	this is the website for this project
3334460	3336480	and I encourage you all to
3336480	3338040	visit the site and download the tool and
3338040	3340560	just give me feedback if you got it.
3340560	3341520	So, thanks.
3346250	3347570	Just want to make an announcement
3347570	3349150	that Meet the Fed will be at...