1
00:00:05,410 --> 00:00:07,670
Bonjour, my name is Elie.

2
00:00:07,670 --> 00:00:12,010
I work at Google, where I lead the security and anti-abuse research team.

3
00:00:12,010 --> 00:00:20,070
Today, with Jean-Michel, who is here in spirit, we are going to tell you how we can use deep learning to reduce side-channel attack surface.

4
00:00:20,170 --> 00:00:34,390
Before getting started, I would like to point out that the research you are about to see are part of a larger project that we do in collaboration with many Googlers and external collaborators around hardening hardware cryptography to create more secure devices.

5
00:00:34,390 --> 00:00:51,250
Side-channel attack is one of the most efficient ways to attack secure hardware because instead of targeting the algorithm, which is usually well understood and well scrutinized, let's say AES, instead, it targets the implementation and the interplay with the given hardware.

6
00:00:51,410 --> 00:01:02,830
And this is well less scrutinized because A, there is many of them, and B, it is way more subtle to understand how side effects of code affect the specific hardware and how that can be exploited by attackers.

7
00:01:02,830 --> 00:01:06,570
Here is a concrete example to show you how powerful side-channel attacks are.

8
00:01:06,570 --> 00:01:16,030
Back in 2017, researchers were able to recover Bitcoin private keys out of a Trezor hardware wallet by using side-channel attacks.

9
00:01:16,270 --> 00:01:29,290
They show you that despite the algorithm being well reviewed and the hardware being well understood, the interplay between the two still had some problems that you can exploit through side-channel attacks.

10
00:01:30,590 --> 00:01:36,690
And from the defender side, side-channel attacks are very difficult because they are very hard to debug and fix.

11
00:01:36,690 --> 00:01:43,850
So even if you know you have a side-channel, it's really hard to know where it's coming from and what you can do to fix it.

12
00:01:43,930 --> 00:01:53,450
So if side-channel attacks are both very important and hard to debug, it means that there is room for innovation on how to help with the situation.

13
00:01:53,450 --> 00:01:55,770
And that's where these projects come in.

14
00:01:55,770 --> 00:02:03,750
And our idea was maybe we can try to develop new technology to accurately pinpoint the code which is vulnerable to side-channel attacks.

15
00:02:03,750 --> 00:02:13,930
So developers can quickly isolate it and try to improve it and improve the quality of their implementation and be more resilient to side-channel attacks.

16
00:02:14,350 --> 00:02:16,090
So that was the idea we had.

17
00:02:16,090 --> 00:02:32,950
And then the way we went about that was we proposed to use deep learning and dynamic analysis and combine the two to be able to accurately pinpoint the origin of the leakage which is responsible or which is exploited by a given side-channel attack.

18
00:02:33,190 --> 00:02:36,130
And I know, I know what you're thinking.

19
00:02:36,130 --> 00:02:38,870
You're going to be like, oh my god, one more deep learning talk.

20
00:02:38,870 --> 00:02:42,710
Really, it's all going to be all hype and etc, etc.

21
00:02:42,710 --> 00:02:44,090
Well, actually, no.

22
00:02:44,190 --> 00:02:48,870
This talk, as I promised, is a hacker journey, so we want to make it very concrete.

23
00:02:48,930 --> 00:03:00,250
And what we really want to do today is showcase to you our debugging tool, which we call SCALD, which stands for side-channel attack leak detector, and how that works in practice.

24
00:03:00,250 --> 00:03:17,190
And to make it very concrete, today I'm going to show you how we can use SCALD to debug TinyAS, a very vanilla plain implementation of AES, running on a well-known CPU, which is the SMT32F4.

25
00:03:17,190 --> 00:03:38,390
So we're going to really see in practice today how you can use the technique we developed to isolate a very clear leakage, and hopefully by the end of the talk you will have a good understanding of what the tool is about and why it might be useful if you are working in the field of cryptography and hardware crypto,

26
00:03:38,390 --> 00:03:40,190
or if you are interested in us today.

27
00:03:40,190 --> 00:03:44,050
So, with this, how are we going to go about that?

28
00:03:44,050 --> 00:03:48,030
First, I'm going to briefly detail how side-channel attack works.

29
00:03:48,030 --> 00:03:55,330
Then we're going to discuss how AI-based side-channel attack works, because we need those to be able to pinpoint the leakage.

30
00:03:55,330 --> 00:04:06,330
Then I'm going to deep dive a little bit into what is AI explainability, and how you go about explaining what a machine learning sees, or understands, or how it makes a decision.

31
00:04:06,330 --> 00:04:09,050
And finally, we'll bring everything together.

32
00:04:09,050 --> 00:04:22,330
I'll talk a little bit about the dynamic analysis part of the project, and how we fit it all together to, well, pinpoint the leakage, and I'll show you how it works in practice for the target, our tiny ass on SMT32.

33
00:04:22,330 --> 00:04:29,070
We can follow along by getting the slide, at the very least, and hopefully link the code by going to elit.net slash code.

34
00:04:29,650 --> 00:04:47,610
As I mentioned, the purpose of this talk is to be very practical, down-to-earth on how can you use, and how this type of tool works at a high level, and how can you use it as a practitioner, not necessarily how you can develop such a technique, and how you can research on how to improve it.

35
00:04:47,610 --> 00:04:59,530
If this is something you are interested in, we are working on a paper, a research paper, which has all the technical details about the extension technique we use, the benchmarking, and alternative, and all the good stuff.

36
00:04:59,530 --> 00:05:08,690
And hopefully, the paper will be out shortly, by the time you see this recording, or will have been already put on archive by the time you see it.

37
00:05:08,690 --> 00:05:14,350
With that out of the way, let's start with recapping what side-chain attacks are.

38
00:05:14,350 --> 00:05:22,330
At its core, a side-chain attack is an indirect measurement of a computation result using an auctioneering mechanism.

39
00:05:22,330 --> 00:05:31,960
So basically, instead of observing directly the result, we try to infer what it is using a third-party way, which is what we call an auctioneering mechanism.

40
00:05:31,960 --> 00:05:38,220
Side-chain attacks are used in many, many ways to attack various targets.

41
00:05:38,260 --> 00:05:44,780
Obviously, as discussed in this talk, they are used to recover encryption keys out of hardware secure implementation.

42
00:05:45,180 --> 00:05:53,000
They are also used in web security to perform blind SQL injection, where you cannot see the return of a SQL statement on SQL injection.

43
00:05:53,000 --> 00:05:58,920
And then, they are also used to steal passwords and PINs from secure implementation.

44
00:05:59,200 --> 00:06:07,080
And they are also used, as I mentioned in the early example at the beginning, to recover crypto-wallet private keys.

45
00:06:07,080 --> 00:06:17,920
Those are only four examples of the many ways you can use side-chain attacks, which is basically when you need to verify something you cannot observe, a side-chain attack is usually the way to go.

46
00:06:17,920 --> 00:06:20,020
Let's make that a little bit more concrete for our use case.

47
00:06:20,020 --> 00:06:31,580
When you do a computation for crypto, what you do is you feed a plain text and you feed a secret key, and the algorithm is running on the CPU.

48
00:06:31,780 --> 00:06:34,780
And while it's running, we have some leakage.

49
00:06:34,840 --> 00:06:37,420
The first one is how long the computation takes.

50
00:06:37,420 --> 00:06:45,900
While it's not very relevant to AS when you have hardware acceleration, actually, depending on the computation, the time might change depending on the key.

51
00:06:45,900 --> 00:06:54,320
That was mostly used for RSA a long time ago because RSA uses exponentiation, and when it's not constant time, it can actually help to recover the key.

52
00:06:54,320 --> 00:06:58,820
That's one of the examples, but again, timing is how long the computation takes.

53
00:06:58,860 --> 00:07:01,740
Then we have the one we use in this talk, which is current.

54
00:07:01,740 --> 00:07:13,440
Of course, depending on the operation you do, depending on how many registers you load, how many registers you unload, the amount of power consumption will vary from clock to clock.

55
00:07:13,440 --> 00:07:18,060
And so that's what we can measure, and that's what we can use to infer what is happening.

56
00:07:18,480 --> 00:07:22,800
Then we have a third one, which is less used, but it's still possible.

57
00:07:22,800 --> 00:07:23,840
It is heat.

58
00:07:23,840 --> 00:07:32,740
Of course, depending on which part of the... which type of operation you do, different parts of the CPU will be used, and as a result, some parts will be hotter than others.

59
00:07:33,260 --> 00:07:37,500
A little bit arcane, not most rarely used, I think.

60
00:07:37,500 --> 00:07:41,200
I haven't seen very much concrete examples, but it does exist.

61
00:07:41,200 --> 00:07:46,860
Last but not least, we also have what we call electromagnetic emission, EM.

62
00:07:46,860 --> 00:07:53,100
And EM is also a very powerful channel, widely used to recover public keys.

63
00:07:53,100 --> 00:08:01,680
It is, with current, probably one of the two most used in side-channel attack, timing being also important, as I've mentioned.

64
00:08:01,800 --> 00:08:06,280
But really, current and EM is probably like the two leading technologies these days.

65
00:08:06,920 --> 00:08:08,540
What does it look like in practice?

66
00:08:08,540 --> 00:08:12,720
So here is a power trace of an AES.

67
00:08:12,720 --> 00:08:24,160
And if you look carefully in the middle of the slide, you'll see, well, about 10 times the same pattern, and they do correspond to the 10 rounds of AES.

68
00:08:24,300 --> 00:08:33,420
So you can visually see on a non-protected, or very likely protected AES, that we can observe the round by just looking at the power trace.

69
00:08:33,420 --> 00:08:42,040
If we can observe it as human, it means there are some statistical information there that can be exploited to understand what is happening.

70
00:08:42,040 --> 00:08:49,600
And that's, in a sense, the side channel, the side effect, that we can use to recover an AES key.

71
00:08:49,960 --> 00:08:51,480
In a nutshell, how you go about that?

72
00:08:51,480 --> 00:08:56,700
Well, you get a CPU, that's the target, to do the encryption.

73
00:08:56,700 --> 00:09:03,440
And what you do is, you do record while the encryption is performed, the power trace using an oscilloscope.

74
00:09:03,660 --> 00:09:14,740
And then, in the traditional side channel attack, the state-of-the-art is called a template attack, where you basically combine the traces you observe, and you make statistical estimates on what it could be.

75
00:09:14,740 --> 00:09:18,300
And the statistical estimate will help you to recover your AES key.

76
00:09:18,300 --> 00:09:20,220
This is how it works.

77
00:09:20,680 --> 00:09:25,160
If you wonder what type of hardware we use, and again, this is what we use.

78
00:09:25,160 --> 00:09:27,660
It doesn't mean that's the best, it's just what works for us.

79
00:09:27,700 --> 00:09:30,500
We use an UAE CheapWaveScope Pro.

80
00:09:30,820 --> 00:09:40,380
And for some of our work, however non-distop, we also use a PicoScope 6000, when we need faster sampling, when the target is very fast, and we need a lot of information.

81
00:09:40,380 --> 00:09:52,420
So, for example, last year, when we talked about SCAML, which is a way to do a side-channel attack using machine learning, which we're going to briefly recap in a second, then we do use a PicoScope.

82
00:09:52,420 --> 00:09:55,900
For this work, we just use plain old CheapWave scores.

83
00:09:56,120 --> 00:10:04,460
And again, this is not an app, this just happens to be what we use in practice as we want to make this talk as concrete as we can.

84
00:10:04,880 --> 00:10:13,620
Now that we have an idea of what a side-channel attack are, let's talk about how you would go about using AI to perform such an attack.

85
00:10:13,660 --> 00:10:20,280
Well, this is something we call a side-channel attack automated with machine learning, also known as CAML.

86
00:10:20,280 --> 00:10:24,200
This is what we did present in-depth last year.

87
00:10:24,200 --> 00:10:36,240
But let me briefly recap how they work in practice, because we're going to need the model that we create using a CAML attack to do the explainability and then find out what the leakage is.

88
00:10:37,120 --> 00:10:43,200
If you want to have more detail, by the way, about this type of attack, well, you can check out last year's talk.

89
00:10:43,200 --> 00:10:49,320
It's available on my website at eddy.net slash CAML.

90
00:10:49,320 --> 00:10:53,260
I also probably put a link on to Twitter if you want to follow along.

91
00:10:53,260 --> 00:10:55,220
And again, you'll have way more detail.

92
00:10:55,400 --> 00:11:00,440
I'm going to try to shrink down the explanation as much as I can so you can follow along.

93
00:11:00,440 --> 00:11:03,660
But if you want all the detail, they are in the previous talk.

94
00:11:03,920 --> 00:11:11,400
I also want to say this year is different from last year, so for people who follow both, in two senses.

95
00:11:11,400 --> 00:11:25,760
This time, last year when we talked about using machine learning to attack hardware encryption, we took the worst case, which is we are doing black box attack where the attacker doesn't have any knowledge about the target.

96
00:11:25,940 --> 00:11:36,360
And, for example, cannot have access to the clock because the clock is usually not accessible after when the key is in or the hardware implementation is in production mode.

97
00:11:36,960 --> 00:11:48,480
And so as a result, we were collecting trace in an asynchronous manner, which means that the clock of the target and the clock of the oscilloscope were different, which is why we had to use very high sampling rate.

98
00:11:48,480 --> 00:11:57,000
In this specific case, this year we are changing the model because this time this code is for people who are developing implementation.

99
00:11:57,100 --> 00:12:03,160
So we do assume you have the code, you have the hardware target, you can put it in debug mode.

100
00:12:03,160 --> 00:12:07,260
And so we don't need to create asynchronous traces.

101
00:12:07,260 --> 00:12:14,200
We also have a good idea, if you're a developer, at what time AES starts and what time it ends.

102
00:12:14,200 --> 00:12:17,060
So you can also create shorter traces.

103
00:12:17,420 --> 00:12:30,460
The reason why to do shorter captures is because machine learning has an easier time if you don't capture the whole thing, because that means it spends less time when it trains to eliminate part of the trace which is useless.

104
00:12:30,460 --> 00:12:33,380
So again, this is a white box attacker model.

105
00:12:33,380 --> 00:12:35,440
It makes more sense for that work.

106
00:12:35,440 --> 00:12:45,240
However, do not try to compare the model you use in this talk, which are easier and smaller, than the one in the previous talk where we had a way harder task for the machine learning.

107
00:12:45,240 --> 00:12:52,600
So when you're in a black box, the machine learning works harder, which means you have to train more and use more complicated and more deeper architectures.

108
00:12:52,600 --> 00:13:01,520
In the case of a white box, when you are really laser focusing on one part of the implementation, here will be the first round, we don't need that.

109
00:13:01,860 --> 00:13:08,680
The way sectional attack assisted by machine learning works is very similar to the traditional sectional attacks.

110
00:13:08,680 --> 00:13:15,340
As in the previous case, you have the encryption which is running, and then you capture the trace.

111
00:13:15,340 --> 00:13:26,820
Don't forget as you capture the trace and use two of them to normalize them between 1 and minus 1 because machine learning actually works in that range, which is not what your traditional oscilloscope would output.

112
00:13:26,820 --> 00:13:39,360
And then we feed those traces to a deep neural network and we train to make predictions on what he thinks are the values which can be used to recover the key.

113
00:13:39,360 --> 00:13:46,540
And then we combine those to do a statistical estimate and hopefully you get back your IS key.

114
00:13:46,920 --> 00:13:53,800
One of the advantages of that, as illustrated last year, is that you do not need to do any kind of pre-processing.

115
00:13:53,800 --> 00:14:01,280
You can just feed the trace directly and there is less expert knowledge on it, so it's open to place to do that almost automatically.

116
00:14:01,280 --> 00:14:12,120
And so that makes it a little bit easier and also it's more powerful in a sense than the traditional template attack because of the reason mentioned.

117
00:14:12,120 --> 00:14:18,280
When you do a sectional attack, you do not necessarily directly target to recover the key.

118
00:14:18,280 --> 00:14:20,940
You target what we call attack point.

119
00:14:20,940 --> 00:14:30,160
In the case of TinyAIS, there are two points which works really well, which is a sub-byte in, which is when you XOR the key with a plain text.

120
00:14:30,160 --> 00:14:35,540
And there is also the sub-byte out, which is when you look at the output of the else box.

121
00:14:36,540 --> 00:14:44,640
In this talk, we're going to focus on one of them, which is a sub-byte in, which we know works really well based on our experience and this is something which is a point.

122
00:14:44,640 --> 00:14:47,420
So basically, the machine learning will now predict the key.

123
00:14:47,420 --> 00:14:50,560
That really doesn't work when you try that.

124
00:14:50,560 --> 00:15:03,640
But instead, it's going to predict the sub-byte in and will basically, if you want to do the real attack, you take the sub-byte in and then you have to invert it using the plain text or do another XOR and then you get the key and then it can be a prediction.

125
00:15:04,020 --> 00:15:06,660
So the target point today is sub-byte in.

126
00:15:07,500 --> 00:15:14,260
When the machine learning will predict the sub-byte in, what it does is you get a trace and then it tells you, okay, here is a softmax.

127
00:15:14,260 --> 00:15:24,660
So a most probabilistic output of 256 values will tell you which is the most important value, but also what you think is the second-best value, third-best value, and so forth.

128
00:15:24,740 --> 00:15:26,760
And that's what the softmax does.

129
00:15:26,840 --> 00:15:36,740
So what you do on how to combine those things, that's why we call it a probabilistic attack, is you basically sum them up using log10 because of finding errors.

130
00:15:36,740 --> 00:15:46,280
And then you combine them and hopefully by combining them, you get the most likely value and machine learning is correct most of the time.

131
00:15:46,280 --> 00:15:47,840
So it will quickly converge.

132
00:15:47,840 --> 00:15:53,340
Last year, we showed that for 10 years on a full trace, we only need four traces.

133
00:15:53,340 --> 00:16:00,800
So you can see, you will see that in the specific settings today where it's even easier, well, you need as little as that too.

134
00:16:00,800 --> 00:16:08,120
So basically three, four traces for simple cases, you get the correct value and you recover one byte of the key.

135
00:16:08,760 --> 00:16:12,720
What's important to mention as well here is we have one model per byte.

136
00:16:12,720 --> 00:16:15,580
So we have 16 bytes, which is the normal input.

137
00:16:15,580 --> 00:16:19,620
We have in reality 16 models which are performing one byte at a time.

138
00:16:19,620 --> 00:16:22,200
It is easier for the machine learning to predict one byte at a time.

139
00:16:22,200 --> 00:16:24,620
So, well, you have to train 16 times.

140
00:16:24,980 --> 00:16:28,660
I'm not going to show that on the slide because it's not relevant.

141
00:16:28,660 --> 00:16:30,900
If you can wait for one, you can wait for 16.

142
00:16:30,940 --> 00:16:36,280
However, there are some difference of accuracy between the bytes, but that's not relevant for this talk.

143
00:16:36,520 --> 00:16:37,200
Okay.

144
00:16:37,760 --> 00:16:48,060
For those fractures, this year we use a hyper-tuned residual 1D convolutional neural network.

145
00:16:48,140 --> 00:16:50,280
It's different between this model and last year.

146
00:16:50,280 --> 00:16:52,800
This model is way more efficient.

147
00:16:52,800 --> 00:16:58,620
It's smaller and I think it's 300,000 points or something, 300,000 neurons per meter.

148
00:16:59,340 --> 00:17:02,300
It's way tuned and it works really well out of the box.

149
00:17:02,300 --> 00:17:08,640
This is kind of like our go-to model these days, which is based on our previous work and testing a ton and a ton of models.

150
00:17:09,320 --> 00:17:17,140
The paper I said last year, it's kind of funny, but the paper on all our tests about all the machine learning models will be out at some point.

151
00:17:17,140 --> 00:17:19,400
I said last year it will be soon.

152
00:17:19,400 --> 00:17:24,680
We have some technical difficulty to make everything reproducible and we have a lot of improvement.

153
00:17:24,680 --> 00:17:38,060
But hopefully, I'm really hopeful, knocking on wood, these models and the data that we use to gain our expertise into this will be out and you guys can test it out, hopefully in the near future.

154
00:17:38,060 --> 00:17:47,840
For Scaled, you need the 16 models because you need to know what is the commonality between all the bytes to know exactly what is the main source of leakage.

155
00:17:47,840 --> 00:17:50,400
So you train 16 models.

156
00:17:50,560 --> 00:17:53,200
As I mentioned, the accuracy varies.

157
00:17:53,320 --> 00:17:56,860
As you can see, you reach something that is a validation accuracy, to be clear.

158
00:17:57,200 --> 00:18:09,800
So on data, which has not been seen during the training, you reach something between 63 for the worst one, which I think is byte 4, up to 87% for the best one, which is byte 0.

159
00:18:10,040 --> 00:18:12,080
And again, they're all between that.

160
00:18:12,780 --> 00:18:14,860
For Scaled, it doesn't really matter.

161
00:18:14,860 --> 00:18:23,040
What you need is to be able to isolate enough examples that the machine learning is correct, because those are the ones we're going to use for explainability.

162
00:18:23,220 --> 00:18:27,880
We want to know what the model is using when it's correct.

163
00:18:28,020 --> 00:18:29,460
That's why we don't train more than that.

164
00:18:29,460 --> 00:18:34,600
Try to imagine each of them is about, I would say, 15 to 20 minutes.

165
00:18:34,680 --> 00:18:38,080
So you do 3 per hour and you have 16 to go.

166
00:18:38,080 --> 00:18:42,260
So, you know, that's already about 5 hours of training time.

167
00:18:42,260 --> 00:18:46,280
So we don't want to do 20 epochs, which would be completely overkill.

168
00:18:46,600 --> 00:18:59,260
And as I said, unsurprisingly, because 10 years is not protected against side-channel attacks and our model is really well-optimized, we have high accuracy in 5 epochs for all of them.

169
00:18:59,580 --> 00:19:03,200
Now, the model is good at extracting the key.

170
00:19:03,200 --> 00:19:07,220
So we are able to use it and we can consistently recover keys.

171
00:19:07,220 --> 00:19:13,480
Now the question is, OK, how does it help us to go back and, well, find where the leak is coming from?

172
00:19:13,480 --> 00:19:18,380
Well, this is where you need to add another piece, which is deep learning explainability.

173
00:19:18,680 --> 00:19:20,680
So what is deep learning explainability?

174
00:19:20,920 --> 00:19:25,460
Deep learning explainability was first developed, I believe, for vision.

175
00:19:25,460 --> 00:19:28,440
And the idea was, I have a deep learning network.

176
00:19:28,480 --> 00:19:33,480
And it says, in this picture, we have a boxer and we have a tiger cat.

177
00:19:33,580 --> 00:19:36,320
Now you can ask the question, OK, but why?

178
00:19:36,320 --> 00:19:38,440
Does it really look at the cat?

179
00:19:38,440 --> 00:19:39,560
Does it really look at the dog?

180
00:19:39,560 --> 00:19:41,860
Or does it look at statistics?

181
00:19:41,860 --> 00:19:42,360
I don't know.

182
00:19:42,360 --> 00:19:51,060
Maybe the stripe colors of the tail of the cat or maybe the dog has a leash or something like that.

183
00:19:51,060 --> 00:19:55,640
So what you want to do is you want to have a way to ask the neural network, what do you look at?

184
00:19:55,640 --> 00:19:58,120
And that's what explainability is about.

185
00:19:58,120 --> 00:20:02,980
It's being able to say, for a model, how does it come up with a given prediction?

186
00:20:02,980 --> 00:20:06,740
Why for this specific class, I'll put one output neuron.

187
00:20:07,060 --> 00:20:10,360
What did you use as input and what does the input matter to you?

188
00:20:10,520 --> 00:20:14,260
So it's basically almost inverting the machine learning model, if you will.

189
00:20:15,040 --> 00:20:17,420
And so we call that explainability.

190
00:20:17,500 --> 00:20:20,480
There are many techniques, but the idea is you...

191
00:20:20,480 --> 00:20:25,380
All the techniques have in common that you feed the machine, the model, to the explainer.

192
00:20:25,680 --> 00:20:28,840
Then you feed the input you would like information about.

193
00:20:28,840 --> 00:20:33,420
And then you have to tell him which class you want to have an explanation for.

194
00:20:33,420 --> 00:20:39,880
This is why, as I explained earlier, we need for Skoal to have models which work well.

195
00:20:39,880 --> 00:20:52,060
Not necessarily 99%, but at least very well because we need to have examples of predictions which are successful because we want to know for a given trace and a successful prediction what to think.

196
00:20:52,060 --> 00:20:55,440
So, as I mentioned, you give it to the explainer.

197
00:20:55,440 --> 00:20:59,960
On the other side, you give back the picture we had.

198
00:20:59,960 --> 00:21:04,020
And we say, OK, why did you believe it was a boxer?

199
00:21:04,020 --> 00:21:11,440
And hopefully, it will tell you, well, I look at the face of the puppy and it's a boxer.

200
00:21:11,480 --> 00:21:13,240
OK, that's reasonable.

201
00:21:13,640 --> 00:21:16,820
And you can ask also, OK, how about the cat?

202
00:21:17,600 --> 00:21:25,020
And hopefully, and again, this is a real example out of one of the explanation techniques, it will tell you, well, I look at the cat.

203
00:21:25,020 --> 00:21:31,660
And the reason why I think it's a cat is because, well, there is a face of a cat, but the most important thing is there are stripes.

204
00:21:32,200 --> 00:21:35,040
As you can see, it's the right part of the image.

205
00:21:35,700 --> 00:21:37,360
And you're like, OK, that makes sense.

206
00:21:37,380 --> 00:21:41,240
I guess a cat which has stripes is probably a tiger cat.

207
00:21:41,240 --> 00:21:41,980
Makes sense.

208
00:21:41,980 --> 00:21:45,420
The machine learning is actually looking at what it should.

209
00:21:45,680 --> 00:21:47,980
Was this technique ever useful?

210
00:21:47,980 --> 00:21:49,920
And the answer is yes, absolutely.

211
00:21:49,920 --> 00:21:54,820
There is this very, very famous data set in machine learning for vision.

212
00:21:54,820 --> 00:21:59,180
And it's called the Pascal Vocabulary, Visual Vocabulary.

213
00:21:59,180 --> 00:22:13,400
And what happened was, in the early version of this data set, I think one picture in five horses had a bottom left name of who took it.

214
00:22:13,420 --> 00:22:21,720
And what the machine learning learned was not to recognize the horse, it was to recognize, well, that there was something on the bottom left.

215
00:22:21,760 --> 00:22:35,260
All right, so that's what explainability is for, so that's what we would like, because essentially what we want to ask the machine learning is, given an input and a prediction, can you tell me how you figure out what is leaking, right?

216
00:22:35,360 --> 00:22:41,520
Where do you get information to get to the conclusion that it's the correct key, or the correct attack point?

217
00:22:42,560 --> 00:22:45,160
Well, that seems great in practice.

218
00:22:45,160 --> 00:22:47,740
It seems great in theory, and we can see where it's going.

219
00:22:47,740 --> 00:22:54,460
The thing is, that doesn't tell us how we're going to combine this explainability techniques and dynamic analysis to debug leakage, right?

220
00:22:54,460 --> 00:23:04,680
Because so far, all I explained to you is maybe how we can get some part of the output highlighted, but doesn't tell you how you go back to know where the leakage comes from.

221
00:23:04,960 --> 00:23:14,580
And so that's all the difficulty of code, and that's why it took us about one year, over a year, to actually really know how to get that done.

222
00:23:14,580 --> 00:23:22,760
It's because you need to be very creative around how to combine those things, which in theory should make sense, to actually get the result you want.

223
00:23:22,760 --> 00:23:25,400
So let's deep dive into how you get there.

224
00:23:25,600 --> 00:23:27,680
Our game plan was fairly straightforward.

225
00:23:27,680 --> 00:23:42,220
Again, we start with an explainer, we're going to give it our train models, and then we're going to give it the trace and the prediction, and say, okay, please tell me of the trace, what are the important points for you to make your prediction?

226
00:23:42,220 --> 00:23:43,860
We're going to call that the leakage map.

227
00:23:43,860 --> 00:24:05,300
Then we're going to also run, not necessarily after, but in parallel, a target emulator, which is basically we're going to run our target, which is the given CPU, the specific type, the SMT, F4, 32F4, and the firmware.

228
00:24:05,300 --> 00:24:20,240
So in our case, the firmware we choose is NES, and we're going to emulate it to be able to know at which every instruction cycle in the ARM CPU corresponds to which opcode.

229
00:24:20,240 --> 00:24:30,360
So basically we need to know at what time, precise point in time, a given code instruction was run, and then which code instruction was run.

230
00:24:30,360 --> 00:24:34,160
And so with that, we can combine both of them.

231
00:24:34,240 --> 00:24:49,000
We can combine the leakage map, which tells us, at what time each instruction was run, to be able to annotate the code, which were the leakages, that's the idea.

232
00:24:50,120 --> 00:24:53,560
In practice, this is where it becomes complicated.

233
00:24:53,740 --> 00:24:55,880
There is, of course, a lot of techniques.

234
00:24:56,180 --> 00:25:08,820
Here is a screenshot, here is a figure from one of the recent paper, which is called Static Check for Static Map, which basically we're looking at how efficient is different type of explainability technique.

235
00:25:08,820 --> 00:25:13,440
You can see some of them have more defined opcodes than others.

236
00:25:13,800 --> 00:25:25,040
In this specific, in our research, we tried a bunch of them, including a guided graph cam, which seemed, when we started, to be giving the most defined and precise leakage.

237
00:25:25,720 --> 00:25:28,100
And then we tested a bunch of them.

238
00:25:28,200 --> 00:25:37,980
The idea was, which explainability technique would work best for us, because we wanted to highlight very precisely which part of the trace was the most important one.

239
00:25:37,980 --> 00:25:39,020
How you do that?

240
00:25:39,020 --> 00:25:45,040
Well, you get the explanation, so you run a lot of traces which were successful into the explanation.

241
00:25:45,040 --> 00:25:47,120
Here is the activation map technique.

242
00:25:47,120 --> 00:25:54,020
Then you combine them, and you normalize between 0 and 1 to create some sort of a mask.

243
00:25:54,020 --> 00:26:04,240
And then you eliminate all the noise, and hopefully you get the bottom image, which is this leakage map.

244
00:26:04,240 --> 00:26:12,740
You can see it's strided here, and you can see that some places which are lighter are supposed to be the place where the model is leaking the most.

245
00:26:12,740 --> 00:26:27,920
So according to activation maps, there is, I think it was for byte 0, very, very early there was a leakage, and very, very late, around the 4000 points, so at the end of our traces, some leakage.

246
00:26:28,380 --> 00:26:31,960
Depending on the technique you use, you're going to get different results.

247
00:26:31,960 --> 00:26:41,580
The first one we tested is SNR, which is not a deep learning technique, but is the standard technique used in tight channel attacks to detect whether or not there is a leak.

248
00:26:41,660 --> 00:26:47,240
So it's a very robust way, statistical, well-proved way, kind of like our baseline.

249
00:26:47,240 --> 00:26:57,240
So the signal-to-noise ratio tells you that, as you can see, there is a main leak, and there is a secondary leak somewhere in the middle of the trace.

250
00:26:57,460 --> 00:27:05,140
And then if we look at GRADCAM++, which is one of the latest techniques, the results are not so clear.

251
00:27:05,440 --> 00:27:13,580
We have a bunch of different points, which doesn't seem to align very well with the SNR, or at least they are less defined.

252
00:27:13,580 --> 00:27:16,160
There are zigzags in place, but not exactly the same.

253
00:27:16,520 --> 00:27:22,620
And then the activation map looks almost equivalent to the GRADCAM.

254
00:27:24,220 --> 00:27:29,740
Even the activation map looks at the output of the layer, the lowest layer.

255
00:27:30,100 --> 00:27:31,940
This seems very, very much the same.

256
00:27:31,960 --> 00:27:34,320
So how do you benchmark how good these experiments are?

257
00:27:34,320 --> 00:27:37,200
So the idea is, well, we have a leak map, right?

258
00:27:37,200 --> 00:27:51,820
So what we can do is we can take our test traces that we know the machine learning is successful at predicting, and then we can decide to use the leak map and, let's say, remove the four points, or the eight points, which are supposed to be the most important,

259
00:27:51,820 --> 00:27:53,560
according to the leakage map, out of the trace.

260
00:27:53,560 --> 00:27:55,060
We can just literally blank them.

261
00:27:55,060 --> 00:27:58,300
Blanking them means put them to zero, or to minus one.

262
00:27:58,300 --> 00:28:00,600
Put them to zero, but some idea.

263
00:28:01,400 --> 00:28:14,400
You basically remove the information there, and hopefully, if they are the most important part of the prediction, then the accuracy of the model, if you feed it again, should result in aggregate to decrease accuracy, right?

264
00:28:14,400 --> 00:28:21,320
The idea is that if you blank out the points, which are used by the machine learning to make the prediction, the accuracy should decrease.

265
00:28:21,320 --> 00:28:25,500
So mechanically, the best technique should yield the best decrease.

266
00:28:25,780 --> 00:28:31,240
Baseline, as I said, 100%, because we only used traces, you remove four points.

267
00:28:31,240 --> 00:28:32,020
Why four points?

268
00:28:32,020 --> 00:28:39,080
It's because an instruction text, when we capture them with the oscilloscope, is four points, right?

269
00:28:39,080 --> 00:28:42,700
So each cycle of the CPU is supposed to be four points.

270
00:28:42,700 --> 00:28:47,800
So if we know that, we say, okay, let's try to remove the most important cycle.

271
00:28:48,860 --> 00:28:52,100
If we do it with SNR, the technique seems to work, right?

272
00:28:52,100 --> 00:28:54,800
It seems we reduced by 57% and 44%.

273
00:28:55,560 --> 00:29:07,260
If we do the activation map, and it was our first very big disappointment sometime in the project, last year, not last year, but like a few months back, was like, oh, well, it doesn't work that well.

274
00:29:07,260 --> 00:29:08,320
SNR is better.

275
00:29:08,780 --> 00:29:11,040
So activation map doesn't work really well.

276
00:29:11,040 --> 00:29:17,740
And then, if you remember, I showed you that the leakage map are almost the same, and the result are the same for GraphQAM+.

277
00:29:19,040 --> 00:29:23,820
So the idea seems to work in the sense that SNR works.

278
00:29:24,500 --> 00:29:29,000
Our deep learning explainability, which is way more complicated and should be way better, doesn't.

279
00:29:29,060 --> 00:29:33,220
So that was a little bit disappointing.

280
00:29:33,280 --> 00:29:35,240
And so it was like, back to the drawing board.

281
00:29:35,240 --> 00:29:36,460
What can we do?

282
00:29:36,460 --> 00:29:37,860
Let's go back to that.

283
00:29:37,860 --> 00:29:52,860
And so the way we went about that is like, OK, let's write our own technique, because what we want is to only find the top point so we can probably do something with a very old idea, which is occlusion, and try to make it a little bit better.

284
00:29:53,120 --> 00:29:58,900
So I know it seems weird to say, OK, let's invent a new technique for a specific task.

285
00:29:58,900 --> 00:30:03,140
But the thing is, we are trying to do something very, very unique, very, very precise.

286
00:30:03,140 --> 00:30:06,960
We don't want to have like the exact region and have more like a holistic understanding.

287
00:30:06,960 --> 00:30:10,140
We want to have like top 5 or top 20 points.

288
00:30:10,140 --> 00:30:19,520
So because our optimization for our technique is different, and the type of underlying algorithm you want to use is a little bit different, we can use occlusion.

289
00:30:19,520 --> 00:30:22,620
We should literally try to use a window to do that.

290
00:30:22,620 --> 00:30:28,360
And so Scaled Explanation Technique is actually exactly that.

291
00:30:28,360 --> 00:30:48,920
It's a hybrid version of occlusion where we start to eliminate large regions so we don't choose more regions, and then with convolutions, convolutive occlusion, which is something we developed for this, to actually really pinpoint which part of the trace for the region that we think are predictive,

292
00:30:48,920 --> 00:30:51,760
which of the exact points are the most important.

293
00:30:51,980 --> 00:30:58,080
And as you can see, the traces are way cleaner than the one we had before.

294
00:30:58,680 --> 00:31:02,420
And they clearly outline leakage.

295
00:31:02,420 --> 00:31:16,200
Also, as you can see, if we took byte 0 and byte 7, it is clear that byte 0 is before byte 7 for in-terms of leakage, which makes a ton of sense because obviously TinyES processes one after the other.

296
00:31:16,200 --> 00:31:17,740
It's not vectorized code.

297
00:31:17,800 --> 00:31:20,420
So that will be exactly what we expect.

298
00:31:21,420 --> 00:31:26,820
Okay, so we're like, okay, trace-study code should work, right?

299
00:31:26,820 --> 00:31:30,620
So go back to the benchmark, run the thing, and voila!

300
00:31:30,620 --> 00:31:35,840
The numbers are way better, or at least better, for byte 7.

301
00:31:35,940 --> 00:31:40,480
And so, again, so now we have a technique which works better.

302
00:31:40,540 --> 00:31:44,280
Our map is more precise, so we hope that the leakage pinpoint will be more precise.

303
00:31:45,220 --> 00:32:00,920
And just to give you a visual comparison, to finish on that side, it's very clear that, A, the SNR and SCALD basically find roughly the same region, except the region is better defined with SCALD, which is good, because the SNR is not wrong in general,

304
00:32:00,920 --> 00:32:11,360
and B, we have something which is cleaner, which means that we have improved over existing state-of-the-art and also finding something completely out of the Teracle model.

305
00:32:11,360 --> 00:32:15,580
So if the Teracle model is more precise, it's good.

306
00:32:16,220 --> 00:32:24,600
And so, as I said, benchmark shows that in every case, SCALD actually outperformed everything we tested.

307
00:32:24,680 --> 00:32:28,300
It's not to say there is not a better way, but this is working for us.

308
00:32:28,300 --> 00:32:30,820
And this is good enough, as we will see.

309
00:32:30,920 --> 00:32:34,540
So we came to favor this thing because it's also quite fast.

310
00:32:34,740 --> 00:32:39,080
It takes, I don't know, about 10 minutes to explain a network.

311
00:32:39,160 --> 00:32:45,840
Okay, now let's explain in greater detail how you go from model to leakage map.

312
00:32:45,840 --> 00:32:47,900
The question is how you go from leakage map to code.

313
00:32:48,160 --> 00:32:54,820
Well, for that, as I mentioned, we have an emulator, which is based on Unicorn and Rainbow.

314
00:32:54,820 --> 00:32:58,980
And basically, we run it with a firmware on the CPU.

315
00:32:59,280 --> 00:33:06,580
And we basically have a state automaton to try to emulate what's happening during the leakage map.

316
00:33:06,580 --> 00:33:15,120
But the idea is you want to do AS start, run the AS, AS stop, and basically pick up what is in sync with the leakage map.

317
00:33:15,320 --> 00:33:19,600
And then what happens then is you get mapped ASM.

318
00:33:19,600 --> 00:33:31,640
So what you get is you get, okay, this cycle maps to this time, this point in the track map to that cycle, that cycle map to this specific CPU instruction.

319
00:33:32,020 --> 00:33:37,020
And then what we do is when we have a CPU instruction, we build a tree.

320
00:33:37,020 --> 00:33:45,820
And the tree, we bubble up the instruction to a given code line using the debug symbol of our firmware.

321
00:33:45,820 --> 00:33:51,100
The firmware we run, the firmware we have is in debug mode.

322
00:33:51,100 --> 00:33:55,160
Again, we're back to this idea that this is a tool for developers.

323
00:33:55,160 --> 00:34:00,540
So when you test for leakage, you can compile with the debug symbol because you don't try to harden it.

324
00:34:00,540 --> 00:34:01,840
We just want to debug it.

325
00:34:01,840 --> 00:34:07,160
So we have the debug symbol which helps us to go back from the instruction back to the line of code.

326
00:34:07,160 --> 00:34:11,100
And hopefully, that gives us an idea of where to map the code.

327
00:34:12,440 --> 00:34:18,200
So there's a theory, but we haven't told you or haven't shown you if it really works in practice.

328
00:34:18,200 --> 00:34:24,480
And the thing is, before I do that, I want to be on phase three of this project.

329
00:34:24,480 --> 00:34:43,980
The first one is, we spend a lot of time talking about explainability because if we don't have extremely precise explainability technique, which exactly pinpoint what point in the trace is responsible for the leakage, we're going to fail.

330
00:34:43,980 --> 00:34:53,080
We're going to fail because most instructions on ARM are two, maybe three, maybe four seconds long at most.

331
00:34:53,080 --> 00:34:57,480
Not at most, but one to two instructions for an additional standard.

332
00:34:57,700 --> 00:35:00,420
So it's literally four to eight points.

333
00:35:00,660 --> 00:35:04,660
Like our margin of error is maybe one point, but that's about all we get.

334
00:35:04,660 --> 00:35:13,520
So if we don't have precise mapping, then if we map, let's say, 10 points ahead or if we have a Windows of 10 points, it's meaningless for us because then it might be three instructions.

335
00:35:13,520 --> 00:35:16,340
And those instructions might belong to a different line of code.

336
00:35:16,340 --> 00:35:19,560
So basically, your analysis is completely botched.

337
00:35:19,880 --> 00:35:38,000
At the same time, we need to have an emulator, which is also single cycle precision, because what happens is, if you do not take into account, let's say, pipeline flush, if you don't take into account the CPU pipeline size and things like that, you get it wrong.

338
00:35:38,040 --> 00:35:46,580
You get it wrong because you're going to shift everything by, let's say, by one cycle for each addition, then your whole traces is completely shifted.

339
00:35:46,580 --> 00:35:51,300
And then even if your leakage mapping is good, you get somewhere in the code which is not relevant.

340
00:35:51,300 --> 00:35:57,380
So you need cycle precision emulator, cycle precision, single point precision explanation.

341
00:35:57,520 --> 00:35:59,580
So you need something extremely precise.

342
00:35:59,580 --> 00:36:02,980
This is very, very much a precise work.

343
00:36:02,980 --> 00:36:06,140
And then on top of that, you need a bit of computation.

344
00:36:06,140 --> 00:36:12,400
Again, as I said, you need to generate about a... we use, I think, a one million data point, something like that.

345
00:36:12,620 --> 00:36:16,500
And then even for the explanation time, we use 60,000 traces.

346
00:36:17,080 --> 00:36:18,320
You need 16 models.

347
00:36:18,320 --> 00:36:23,200
As I mentioned, you can shorten the time, of course, by using good models who convert very quickly.

348
00:36:23,680 --> 00:36:29,040
You need to generate 16 explanations for all those models in your trace, and then you need to map everything.

349
00:36:29,460 --> 00:36:35,100
So with our optimization, that will take you a day or so of work.

350
00:36:35,100 --> 00:36:43,420
And, of course, as I said, most of it is parallelizable because the 16 models and 16 explanations can be run on distinct CPU.

351
00:36:43,420 --> 00:36:47,260
So you can make it 16 times faster.

352
00:36:47,340 --> 00:36:52,660
So that's really good because we want to use that as a fast iterative tool for developers.

353
00:36:53,520 --> 00:36:54,400
All right.

354
00:36:54,720 --> 00:36:58,600
So at this point, you're like, OK, he gave me a long speech of why it's really hard.

355
00:36:58,600 --> 00:36:59,920
So probably they failed.

356
00:36:59,920 --> 00:37:03,080
And it's really, really hard because, yeah, emulating a CPU is hard.

357
00:37:03,080 --> 00:37:04,700
And I'm telling you it's true.

358
00:37:04,700 --> 00:37:07,180
I don't think we have perfect emulation of CPU.

359
00:37:07,180 --> 00:37:15,200
I think we have very precise emulation of the instruction we need for AES.

360
00:37:15,200 --> 00:37:23,080
In particular, one thing we don't have, and disclaimer, is we don't have implemented mapping for divide.

361
00:37:23,260 --> 00:37:24,280
Why?

362
00:37:24,280 --> 00:37:32,320
It's because division actually takes a lot of cycle and a very different branch of cycle on ARM CPU.

363
00:37:32,320 --> 00:37:34,160
So we don't really know what to do there.

364
00:37:34,160 --> 00:37:39,240
So that's only working for now for AES and for transparency.

365
00:37:39,440 --> 00:37:57,380
That being said, if we try to apply what I said to our model, what we expect to see in the theoretical sense is we're supposed to... if the model is targeting sub-byte in, then it must exploit something in the add-on key because add-on key is where you do...

366
00:37:57,380 --> 00:37:59,180
the key exploits the plaintext.

367
00:37:59,180 --> 00:38:02,420
So it must have most of the... mainly cache should be there.

368
00:38:02,420 --> 00:38:04,160
So it's a theoretical thing.

369
00:38:04,260 --> 00:38:09,260
And so what we do to verify that what we said works is to actually try to verify that.

370
00:38:09,520 --> 00:38:12,160
And so that's what the schedule output looks like.

371
00:38:12,160 --> 00:38:13,420
It's a terminal thing.

372
00:38:13,420 --> 00:38:25,400
You basically run it and at the end spit out this, as I said, tree, which maps cycle instruction, which is not displayed for visibility, to code line, which is basically numbers.

373
00:38:25,400 --> 00:38:29,140
And then I filter everything which is not leaking.

374
00:38:29,140 --> 00:38:36,880
And then what this mapping tells you is, yes, the main leakage is on line 2.13 of add-on key.

375
00:38:36,880 --> 00:38:37,860
So that's promising.

376
00:38:37,860 --> 00:38:44,500
It also has, interestingly enough, a second leakage, which is later on into the cipher functions.

377
00:38:46,660 --> 00:38:49,500
So what is the line 2.13, right?

378
00:38:49,980 --> 00:38:53,700
Well, the good news is it's exactly what we predicted.

379
00:38:53,700 --> 00:39:05,040
This is exactly the line in the whole code that you can find on GitHub, which is exactly where the key is XORed with the plaintext.

380
00:39:05,240 --> 00:39:16,760
So the model, and Scaled in general, really clearly is using what a theorem would predict, which is there is a leak in that specific line because they are doing...

381
00:39:17,340 --> 00:39:22,340
they are showing the value of the register with some assignment.

382
00:39:22,540 --> 00:39:24,320
And so this is what it is.

383
00:39:24,320 --> 00:39:25,700
So that's a success.

384
00:39:25,700 --> 00:39:28,880
That's what gives us confidence that what we do works in practice.

385
00:39:29,360 --> 00:39:32,480
It's really giving us interesting results.

386
00:39:32,740 --> 00:39:34,580
I will not claim it works in every case.

387
00:39:34,580 --> 00:39:43,600
I'm sure that with more complex implementation, like masked AES or more complicated security, the result might be drastically different.

388
00:39:43,600 --> 00:39:49,560
As I said, our emulator did not fully emulate all the operations it can do on ARM.

389
00:39:49,640 --> 00:39:54,180
There is still a lot of uncertainty, but however, it works.

390
00:39:54,380 --> 00:40:02,860
I think it's a very promising step towards the right direction of having tools who go back from leakage to exact line of code.

391
00:40:02,860 --> 00:40:06,020
As far as we can tell, it's the first time it has ever been done.

392
00:40:06,500 --> 00:40:10,120
Back to the secondary leakage, I don't have a good explanation for it.

393
00:40:10,120 --> 00:40:17,120
The best thing I can come up with today is that it probably went when some of the registers are unloaded.

394
00:40:17,120 --> 00:40:22,760
And so this is where it started with a mixed column, if you look at line 371.

395
00:40:23,200 --> 00:40:28,480
So maybe that's what happened, it's unloading some of the registers and then going back.

396
00:40:28,480 --> 00:40:33,780
We need a little bit more analysis for that, but basically it will be interesting to know why there is another leakage.

397
00:40:34,040 --> 00:40:45,480
So hopefully today I showed you how we use Scalar to automatically isolate the number of codes and show you how it works in practice and show you concretely what is the benefit of the tool.

398
00:40:45,780 --> 00:41:07,220
And really our hope is this type of tool, we keep building it and we get feedback from the community to build something which will empower people who develop secure hardware to quickly figure out and patch where the leakage are coming from so we can develop stronger crypto in an easier way and a faster iteration so we all benefit from stronger,

399
00:41:07,220 --> 00:41:08,300
more secure devices.

400
00:41:09,480 --> 00:41:22,300
And to take over the talk, as I said, as we discussed last year, machine learning is a way to automate such an attack and you reach state-of-the-art with it so it's really one of the most...

401
00:41:22,880 --> 00:41:44,780
it's at the forefront of such an attack and this year we flipped the use case on its head which is really what was the intent of the project from the onset two years ago which is to try to use all those knowledge to actually help developers building better tooling for them to reduce the cost of developing secure implementation and make them better.

402
00:41:45,180 --> 00:42:03,440
So that being said, this is a very, very new field, a very exciting field with a lot of ideas, a lot of energy around it and it can really use more people interested in it so if you have some interest into crypto, into machine learning, it's a great time to get in and work with the community on this type of ideas.

403
00:42:04,420 --> 00:42:06,720
Thank you so much for attending this virtual talk.

404
00:42:06,720 --> 00:42:08,080
I wish it would be in person.

405
00:42:08,080 --> 00:42:11,140
I'm going to miss DEF CON as you probably do.

406
00:42:11,180 --> 00:42:26,480
I hope you're all well and if you would like to follow up and keep up with what we're doing, we try to publish as fast as we can modulo some delay to provide you information about what we do on such an attack on the website.

407
00:42:26,480 --> 00:42:30,220
Hopefully we'll have an official project website in the future as well.

408
00:42:30,220 --> 00:42:39,020
Thank you so much for listening to this talk and then do not hesitate to reach out on Twitter or by email or any other means.

409
00:42:39,100 --> 00:42:40,600
Happy to answer questions.

410
00:42:40,600 --> 00:42:41,720
Thank you so much.

411
00:42:41,980 --> 00:42:42,460
Bye.