Welcome, everyone, to the Crypto and Privacy Village. This is our seventh year and DEF CON 28 here in safe mode. I am Haim Cohen. I'm one of the hosts here. Let's just get some stuff out of the way. You can join us at Discord. You can find us there. We have a couple channels. The gold bug just opened up, so if you're interested in the gold bug puzzle, we did that very well. And let's just introduce our first speaker, Hanno Berg, and he's going to go. Yeah, hi. Let's switch these slides. Okay, you can go full screen and you can go. Okay, do you see the slides? I see them. Yeah, okay, great. Yeah, hello. So I'm Hanno and I'm going to talk about StartTLS today. This is some research that I've been doing together with Damian Hodebniak, Fabian Ising and Sebastian Schinzel. And we are soon going to publish a paper on this. Yeah, we kind of figured out that StartTLS seems to not be very well researched and it has some issues that are not very well known and we looked a bit deeper into this. So, from the general idea, I guess most people are roughly aware what StartTLS is, but if we talk about TLS connections, there's two ways to do a TLS connection. One way is that we have an extra port for the encrypted version of protocol where we directly start a TLS connection on that port. And this is called implicit TLS because we kind of implicitly are using TLS based on the port number. We already know that this is an encrypted connection. And then there's this other way which is StartTLS where we start with some protocol that is not encrypted and then we have some special command which is usually StartTLS command that upgrades this connection to be encrypted. And in the web we are only using implicit TLS basically. So what we usually do is that we have HTTPS which is kind of its own protocol on its own port which is HTTP protocol with a TLS wrapped around it. But StartTLS, the most common usage is in the context of emails. And in this talk I am mostly going to talk about the connection between an email client and the server. I will talk a bit about server to server connections later but also we haven't focused so much on this because this is kind of a different issue because the considerations for server to server are quite different than from client to server. So all email protocols we use, I guess you are aware of that, we have SMTP, POP3 and IMAP. We use SMTP to send emails, POP3 to fetch emails or IMAP also kind of to fetch emails but also to manage a mailbox on the server. All of these protocols support StartTLS. And for the client to server communication these protocols also all support implicit TLS. So we kind of, as a user we can choose which one we use. So if we can choose between the two then the obvious question is which one is better and which one is more secure. So how does this StartTLS work? So these email protocols, they are all kind of relatively simple text protocols. So it's a line-based protocol where a client sends a command to a server and gets some kind of answer. So this is an SMTP session here. The S and C at the beginning that is server and client. So if you connect to an SMTP server, the first thing is the server sends this line which is basically a hello message which contains its host name and a special code number. And then you send this elo command that is basically from the client, always the first command which is kind of introducing the client. And then the server usually gets back with a list of features that the server supports. And here you can see in this list that it ends with this 250 star TLS which means okay, the server just told us that it supports star TLS. And then the client can use this star TLS. This is simply a text command. It gives them this command star TLS. Then the server answers with a 220 code which means everything is all right. And the okay behind that is basically meaningless. That's more or less like a comment. And then we do a handshake. And for the colors here, the gray thing is the plain text part of the protocol. And then we have a TLS handshake. And the green thing here, that is then encrypted and authenticated with all the properties that we usually expect from a TLS connection. So we expect that there's some sense of security here while the thing at the beginning, it's plain text. So an attacker can read it, an attacker can manipulate it. And then basically we start over because kind of all the information we got before the TLS handshake, we don't really trust it. So we send this elo command again. And then maybe we can do a login and send an email. And you don't need to understand the details of how the protocol works. But the basic idea just that should stick here is we have a plain text protocol. We have a command that starts a TLS connection. Then we do a handshake and then we kind of move this protocol into TLS. Now, first, something which is obvious, that if you use start TLS in an opportunistic way, this is insecure. So there is some email clients implement start TLS in a way where they will say, okay, if the server offers start TLS, then I will use it. And if the server does not offer it, then I will just stay in plain text mode. So we will use start TLS if we can, but if not, then we will just do plain text. And a network attacker can attack this. We figured out that this is actually not that common anymore. This used to be the case maybe 10 years ago, but today most clients don't really do this anymore. And it's obvious that an attacker can in such a scenario, an attacker can just they can pretend to be the server and then they can say, yeah, I don't support start TLS and then the client logs in and then the attacker knows the username and password. So that is obviously insecure, but that's not really what our research was about because that's, yeah. Yeah. But this turns out to be not the only security problem with start TLS. So you could think, okay, this is a problem, but if we implement a client in a way that it always forces this start TLS connection and will just refuse to connect if it cannot do that, then this is secure. But it turns out that this is not the case. So if you look at what start TLS is, then it's kind of a form of state transition where you have the protocol, which can be in two states. It can be in a plain text state and in an encrypted state after the TLS handshake. And this raises quite a few questions. So at first you can ask, okay, we have the state transition, but which protocol features can we use before we turn the encryption on? What can happen in the plain text part? And also how should we treat the data that we got before the handshake, for example, server features or if there's any login, because we cannot really trust that data as it's obviously not secured. And also you could ask, do the implementations, do they have a clear understanding what is encrypted and not encrypted? And do they clearly separate that? So just one example, which is a relatively obvious problem, the IMAP protocol has a feature which is called alerts, which is basically the server sends a message to the client and the client is supposed to show this to the user. And these alerts, it turns out, they can be sent at any time. So they can be sent in the plain text part, which is interesting because this is a screenshot from Outlook where the IMAP server sends a message to the user and this message is not secure. It is sent in plain text, even though the mail client is configured to use a secure connection. And here you can even see, you can even include a link which will be highlighted. So we have here, your IMAP server wants to alert you to the following, please download my post something something and then the attacker can put in a link where he provides a malicious file. So we have an alert sent in plain text but to the user it looks like this is a legitimate message from his mail server. So he may want to trust it. So yeah, this is just a design problem of the protocol, how it is specified. But then we came upon the issue of buffering bugs. And this is something that has been known for a while. And the question here is really, is something part of the TLS session or not? And do clients and servers have a good idea about this? So there was a vulnerability in 2011 already, so this is nine years ago, which was named plain text command ejection in multiple implementations of star TLS. This was discovered by the author of Postfix. And what happened here is, like if we look again at this star TLS connection and this is like only the, I've shortened this a bit, I've cut away the stuff that currently doesn't matter, like we have this client sends the star TLS command, gets an answer from the server and then there's a handshake happening. Now, what can happen if we do something like this, where a client sends the star TLS command and then sends something else in plain text and then gets the answer from the server that he can now start the TLS handshake and then they do the TLS handshake. So it turns out that there are quite a few implementations of servers that will then process both of these commands, the star TLS and the whatever other command you sent together with the star TLS in a single TCP packet. That it will put this in some kind of buffer and process every command after another and then it will answer to this command that was appended to the star TLS in the TLS session. So we kind of have something here that traverses from the plain text part of the protocol to the TLS part of the protocol. Like we send something in plain text but the server kind of thinks that this is part of the TLS connection. Yeah, so that's basically what I just said. So this bug was originally found in Postfix but it turned out it affected multiple mail servers including Cyrus, Courier, Qmail and various proprietary mail stacks. So there's an advisory by US CERT from that time which contains a long list of affected servers. So this turned out to be a very widespread problem. Now you may wonder how you can attack this and this is kind of, I think this is maybe part of the reason why this bug is not very well known because the Postfix authors, they have described very well what the problem is but they haven't really published any exploit or explained how you could actually attack this. So it was kind of seen by and there was some email discussions from mail server developers where people were like, yeah, this is kind of a weird behavior but it's not really a security problem. But you can actually attack this pretty easily and this takes a bit of thinking what's going on here but so this is the first part of the attack which is we start a connection that is in plain text. So we have again this start message that's not really the interesting part. The interesting part begins where we have the start TLS and then this red stuff, this is injected by an attacker. So the attacker is injecting here another elo command which is, that is just always the first command that comes after start TLS. That's why we have to send that. And then the attacker locks in with its own credentials. So our assumption here is the attacker also has an account on this mail server but it could be something like a public email provider address. So that is a plausible assumption. And then the attacker is starting to send an email like he's setting a from address and a to address and the to address particularly is his own address and then it sends this data command which is basically now the email content starts. And then we get an answer from the server. This is now the answer to the start TLS command. So now we're doing a TLS handshake the client and the server but we have these injected commands. And then the server will answer to these injected commands. So it will send this normal greeting that always comes after the elo command and then it will authenticate the user and then it will say okay now the email starts. And now what happens is that the client doesn't know about any of this. It doesn't read from the server because it doesn't expect it to send anything. It sends its own elo command, it authenticates and then it starts sending a mail. Now what happens is that all this stuff, all this blue stuff which the client sends, this ends up being part of this email that we started sending to ourselves as an attacker. So the attacker now gets an email and that contains the authentication data from the victim, so this is a pretty severe attack. Like if you're a network attacker and we have a server vulnerable to this, we can force the client to send his credentials to us. There's a similar attack with IMAP that is a bit more tricky because like you can, in IMAP you can kind of store a mail into a mailbox and then it's the same idea. One problem with that was that in order to do that you need to know the exact size of the message that's going to come. That means you kind of need to guess how long the password of the user is, but you may be able to see that with a side channel attack and also it turns out that many mail servers on failed login attempts, they will just try to reconnect. So you have kind of as an attacker, you have multiple tries to do this so it's still a feasible attack. But it is a bit more difficult than the SMTP attack. Okay, so this bug was discovered in 2011. So you would hope that people have fixed this, but as probably no one is surprised, it's actually not fixed everywhere. So we did some scans and we found out that, yeah, in SMTP servers 1.5% are vulnerable. In IMAP and POP3 it's more, it's around 2.5%. I guess the reason why this is different is because this was originally described in an SMTP server, so that's probably where people had it on the radar. But the very same class of bug can happen in basically every protocol that supports StarTLS. Although I have to say for POP3, we didn't really find an exploit to attack this, but it's still kind of something that should be fixed. Now there's another question you can ask and that is, can we have the same kind of bug, but the other way around? That is, on the client side. So can we confuse the client to get data that was sent in plain text and the client thinks it's TLS data? And in turns out, yes, we can do that. So the idea is pretty much the same. The client sends a StarTLS command, then the server sends this 220 code, which is kind of the just approving that, yeah, I got the StarTLS, now we can do TLS. And then we can append additional data that's coming from the server and then the client will interpret that as part of the TLS session. So yes, this attack also, this vulnerability also exists on the client side. So clients can be vulnerable to the same class of bug and we have called this a response injection because the other thing was called a command injection and here we call it a response injection. This is not that severe, but for example it can be used by an attacker to spoof mailbox content. You kind of have to guess what commands the client will send, but if you know which mail client it is, you can usually know that. So for example, you can show the user an IMAP inbox that is not his own and that contains other data. So it's still an issue. And the really surprising thing was that more than half of the mail clients that we tested were vulnerable to this bug. So this is a very very widespread issue. So to sum that up a bit, this DartTLS command injection, it was discovered in 2011 and it affected multiple implementations and even though many implementations were found back then, they are still widely used vulnerable implementations. Also, the same bug class exists on the client and the majority of implementations are vulnerable or were until we have reported it to them. And just very generally, if you're doing an implementation of DartTLS and you're not aware of this bug and you're doing a naive implementation, then you will have some kind of command buffer where you read the commands line by line and then you will usually have this bug too. So it's kind of one of these things where you have to actively be aware of this bug in order to prevent it. And this, in my opinion, is a clear sign that this is not something where we should say this is a flaw in the mail client, but this is a systemic problem in the standard. Like if you have a situation where the same bug appears again and again and again and in multiple different implementations, then you should wonder if this is something where we can just... if we can do better on the level of the standard and not say it's the flaw of the implementations. Yeah. There's more issues with DartTLS. One, the IMAP protocol has a feature which is called pre-auth. The idea here is that an IMAP server can, send a line which basically tells the client you're already logged in, you don't have to authenticate. So the idea here is, for example, if you have a controlled network where you do authentication based on IP addresses, maybe then you can say, okay, if a connection comes from this IP address, then we just assume that this is already authenticated and the user is already logged in. So now, DartTLS, the standard clearly says that it must not be performed in an authenticated state. So if we take these two things together, like we have a feature, DartTLS, that you must not use when in an authenticated state, but the server answers with this pre-auth line which says you are already authenticated. Now, what should a client do? Like we have a client that connects to a server, it gets this response, it's configured to use DartTLS, then we kind of have a contradiction here. Like this is kind of a logical inconsistency in the standard. The standard doesn't really say anything about this. This is just where the standard has an internal conflict. And after all, the only reasonable way for a client to react to this is to just terminate the connection, because it cannot do DartTLS in this situation. So there's no way to secure this connection and if it's configured to have a secure connection, it should just not accept this. Now, not surprisingly, some clients will allow pre-auth connection and then just don't do DartTLS because they can't. And this is also a security flaw. You also obviously can do mailbox content spoofing, like you're a network attacker person in the middle and then client connects, you send this pre-auth response and then you're kind of the upstream server for that client. So you cannot steal credentials because the client is already authenticated, but you can forge the content of the inbox. And also, you can sometimes steal mail, for example, if you're using IMAP then usually a sent mail is stored in the send folder, so that you can steal. And some clients also, when they connect to an IMAP server, they start synchronizing folders, so there you may be able to steal mailbox content. Then there's an IMAP feature which is called mailbox referrals. And the idea here is that the server can tell a client that it should use another server. So it's something like a forward feature, where a server can say, yeah, like I'm forwarding you to this other server if you want to access this mailbox. And now the interesting thing is if you combine these two things, you have this pre-auth, which means you can prevent the client from starting a TLS connection. And then you can do this referral, which refers the client to another server. And this can be our attacker server. So we prevent the TLS connection from happening as an attacker. Then we forward the user to a server and we control that server. So this can have TLS and the client logs in. And then we have the client's credentials. So this is also a pretty severe attack. We have to say here that this mailbox referrals feature, and there's a similar feature called login referrals, these are not widely supported. So we only found one mail client where we could perform this attack. This was a mail client called Alpine. So most mail clients, they just don't support these features, so they are not affected by this. But still it shows kind of that there are features in IMAP where it's not really thought through what that means in combination with StartTLS. So yeah, to summarize that, so StartTLS has a systemic problem with buffering bugs. They can appear both on the server and the client sites. There are several features in IMAP that are problematic or insecure if you combine them with StartTLS. And also StartTLS does not really provide any security advantage over normal implicit TLS connection. So our conclusion is you should not use StartTLS for mail connections, for email client connections, I guess. So if you are using email, which I guess most of you do, and if you're using a mail client, then you should change its configuration to use the implicit TLS pods and not to use StartTLS. And then you will not be affected by most of these attacks. Some can still happen, but because some it's enough when the server is supporting StartTLS, but most of these attacks will be prevented. If you run a mail server, you should definitely make sure that you support implicit TLS. I mean, there's an RFC that says that you should use the implicit TLS pods, but there are, for example, the Microsoft SMTP servers, they don't support implicit TLS for SMTP. And if you can, then disable StartTLS. Now we are aware that this is not an easy thing, particularly if you have existing customers. Like if you have an existing user base, then telling them, yeah, sorry, half of your mail configurations will break next week, that's a tough thing to do. So I'll say it would be ideal to just disable it on the server, but this is probably not feasible for people who have existing user bases. But if you start like a new mail server setting, then you can start right away without StartTLS. So then I want to talk a bit about the special case, the server-to-server or MTA-to-MTA connection. In case you don't know, MTA stands for Mail Transfer Agent, which is another word for mail server. So this situation there is a bit different. For one, there's currently no way to use implicit TLS on MTA-to-MTA connections. That's just not part of the standards. There's no RFC how to do this. And it would not be easy to implement that. And also, traditionally, the server-to-server connections are purely opportunistic. So usually mail server doesn't validate a certificate because also usually the certificate host is not the host of the email address. So even validating the set doesn't make much sense. And you usually don't have an expectation against active attackers. Now, this is changing a bit. So there are two standards that one is based on DNSSEC and called DAIN and the other is called MTA-STS, which is kind of a trust on first use kind of mechanism that are trying to secure these server-to-server connections. It would be a different talk to talk about the advantages and disadvantages of those. My personal opinion is that MTA-STS is the more promising mechanism, but that's a controversial discussion. Let's just put it like this. But so yeah, it would be challenging to introduce an implicit TLS mode into MTA-to-MTA connections. Like right now, we are just using star TLS. Even if we use this advanced mechanism, we are still using star TLS. And the problem here is like that we have a lot of compatibility issues here. Like in email, it's a very old protocol and things are very established and probably a lot of mail servers have a firewall that they don't allow other ports than port 25. So it would be very challenging to change the protocol to support an implicit TLS mode. So I don't see that happening anytime soon. So if we want to secure these MTA-to-MTA connections, like if we talk about something like MTA-STS, then we should really need to make sure that we fix these buffering bugs, like these command injection and response injection that I've talked about earlier. If you think that you want to secure these server-to-server connections, then you also need to think about this. And you need to basically audit the implementation so they don't have these bugs. Then just quickly also start TLS is used in many other protocols like XMPP, FTP, then managed sieve, which is a mail filtering thing, then LDAP, MySQL, NNTP. And I guess this is obvious, but the kind of bugs that we found, they can be present in implementations of all protocols that support start TLS or something like start TLS. So there's lots of potential for future research here. Like this is something that should be tested in more settings. And also our general recommendation that if you have a choice between start TLS and implicit TLS, then you should prefer implicit TLS. And for future protocols, maybe you should just only support implicit TLS. And then something that is kind of not on the security side, but we can also ask what about performance? What about speed? Because I don't know. I come from Germany and Germany is known for having pretty bad mobile Internet. Not sure if you're aware of that, but if you ever go to Germany, that's what you will experience. So if you try to send an email with a bad Internet connection, it's pretty annoying. Like you have these things that you may get a connection error or the email may be sent, but it's not stored in your send folder because these are two different connections. So having a better performance here is good and also more reliability. If we need fewer connection roundtrips, then it's more reliable. This I also find interesting. There's been a lot of effort to reduce roundtrips in TLS. In TLS 1.3, by design, if you make a new connection, it has one roundtrip less than older TLS versions. And there's also a zero RTT mode, which is kind of... there are some security concerns about this, but the TLS community has decided that they still want to do this despite these security concerns. And I find this interesting that so much effort has been put into reducing TLS roundtrips, but it seems nobody has tried to optimize email protocols the same way. And if we look at startTLS, then all the information that is exchanged before we do startTLS is kind of meaningless, because we cannot trust it anyway, so we need to retransmit everything after the TLS handshake anyway. So for example, the server capabilities, you're always getting them again after the handshake. So you can think about this, that everything that is happening before we do this handshake, it is just wasted. It doesn't have any meaning. So this kind of directly leads to that if you use startTLS, you need more roundtrips. It depends on the protocol and also a bit depends on the implementation, but it's usually two or three roundtrips more than if you directly use implicit TLS. So avoiding startTLS is not just more secure, it's also just faster. Which is, I guess, good. And there are some other ways. This is kind of now going a bit into other topics, but there are other easy ways to improve email protocol roundtrips. So email authentication uses something called Zazzle. Zazzle is an authentication framework that is used in all three email protocols. And it's kind of an abstraction of different authentication methods. Although most of the time everyone uses username and password, but there are different authentication methods. And if you look at this, how this looks like in IMAP, we can have something like the above, where we have an authenticate plane and then we have some Base64. And this Base64, it is basically, it's a zero byte, then a username, then another zero byte and a password. And that is our login. So this is one roundtrip to login. And we can have this thing below where we have authenticate login, then we have a new line. So this is kind of optional. We can, in this authenticate line, we can already send data or we can just not. And then the server asks for a username. This is also Base64 encoded and asks for a password. So this is three roundtrips. So there's quite some difference here and it's quite varying what will happen depending on what client and server configuration you use. So the first thing is that IMAP has a feature called SASL IR, which basically means you can initiate an authentication right away with data. So this is the thing that you have this authenticate plane and send something in the same line where... or you can also not do that. And of course, this is faster. This saves you a roundtrip if you can send data right away with the authentication command. And also, Zazzle has two different modes for username password authentication. They are called plain and login. To make matters even more complicated, both POP3 and IMAP, they have a built-in authentication that is not part of Zazzle, which is kind of part of their protocol itself. In IMAP, this has one roundtrip. In POP3, it has two roundtrips. So this login method, this is not standardized and it is also declared obsolete by the IANA. So the IANA has a registry of Zazzle authentication methods and there it's declared as obsolete. And what it basically does is it sends the username and the password base64 encoded in two roundtrips or just two lines in other words. And then there's this plain authentication method, which is a standard and it sends username and password encoded in one roundtrip. So you save another roundtrip here. So the plain method is standardized and it's faster than login. So it's clearly the better method. And from the security, it doesn't really matter. It's a normal username and password authentication. There's nothing fancy about it. So almost everyone supports both, like both servers and clients, but there's not really a reason to support login. So if you want to have faster email, then you should use implicit TLS and not start TLS. You should support the sessil-ir, which is specifically in IMAP. And you should disable the sessil-login method. Like for example, I figured out my own mail server, it supported both. And my client, if it supported both, it used login because that was the first in the client. It's in class mail. It's the first method that it supports. So it's slower for no gain. And I disabled it on the server and now I'm saving. So yeah. Yeah, that was it. Yeah, don't use TLS. Thanks for listening and I guess we have some time for Q&A now. Hold on. Welcome back, everybody. Can you, Hanno, can you unshare your screen maybe so we can just see you? I just want to put my background on. Okay. So we have, I think we have three questions for you. Can you hear me? Okay. So is the implication that the clients in the SMTP case for ease just ignore responses from the server? I expect them to be a state machine that says, wait, what? I haven't started sending them yet. What do you mean? No, that's not the implication. So the client is not ignoring these answers, but it is like it is processing these line by line. So it is sending a command to the server and then it's reading a line from the server and processing that. And that is why if you do this attack, you kind of have to inject all the commands that the client will send so it will get the answers that it expects. Is this understandable? I hope so. We are monitoring the discord. So if they reply back, I can do it. Yeah. Actually, give me one second. I think we are going to try and share the slides with you so everyone can see it. Give me one second. Share this. I did that. Sorry, everybody. Let's see. So they want me to... let's see. I know we have dead air. Let's see if I can share the screen. I am now also in the discord. That's okay. I will be able to see questions later if they are. I see your screen now. Okay. The second question I have, in the context of SMTP, do you see MTA STS having positive, negative, no change to the start TLS risks? It doesn't... I think the question is framing it wrong. The thing is, MTA STS is introducing an expectation that we didn't have previously. That is that these connections are secure against active attackers. And these attacks can compromise this expectation. So before, we could say these attacks don't matter for MTA to MTA because we don't expect it to be secure anyway. And now we want to introduce connection security for that. So now we have to care about these attacks. But it doesn't really change the risk. It's just that what we try to achieve with MTA STS that is compromised if we have these vulnerabilities. Does that make sense? I mean... Yes, I understand it. Again, we can be in the Slack Discord and maybe later, if people have more questions, you can join in there. Last one that I have. In what kind of issues does the speaker see with implicit TLS? So I think if you compare start TLS and implicit TLS, there are no added issues by implicit TLS. Because it's just the simpler thing. Like with start TLS, we start in plain text and then we do a state transition and that introduces potential for errors. And with implicit TLS, we just don't have that. So I don't see any risk in implicit TLS that you wouldn't have with start TLS. Of course, it has all the risks that you can have otherwise with TLS. If there's a bad implementation, if there's bad randomness or vulnerable algorithms, then you have all the same risks. But you would have the same risks with start TLS as well. We're getting another question. Any thoughts on Telnet start TLS? I have no idea, sorry. I have seen that it exists, but I'm not sure if anyone is using that. I would be curious if anyone is using that. I feel that people use SSH there usually. And then our final question, just thanking you for the talk. Is anyone working on email POP3 replacements with something more secure? I'm not sure. There's not really a problem with POP3. This is not the issue of the talk, but I feel POP3 is a really simple protocol and I kind of like that. I'm also one of these weird people that I'm still using POP3. I like to have the mails on my laptop locally. If you use it with TLS, then it's a fine protocol. Well, again, we would like to thank you for your talk. If anybody has any more questions, you can hang out in the Discord. We'll try and relate them to you, anything like that. I want to thank everyone who's watching the live stream for sitting with us through the first ever DEF CON Safe Mode and the Glitch Crypto and Privacy Village. I want to remind everyone again that we have the Gold Bug Challenge that's still there. And I would like to, again, thank you for your talk and I hope you enjoyed yourself. I guess we wait. Okay, I can leave here now?